Generally, the harder you "push" a driftboat, it just raises the bow higher out of the water. DB's reach their theoretical maximum speed with a very small motor. A 4HP will give you about the same top speed as a 8HP... the difference being that the 8 will perform much better against a strong current.
On my bud's boat, we run a 80# electric, which works great... the downside being the weight of the batteries (often run 4 in tidewater/bays).
You can easily troll, but in the wind, or with a strong cross-current, not so much.
It's been said -- A motorized driftboat is about the worst powerboat ever concocted. It's also one of the best fishing platforms ever concocted.
Anywhere from a 2.5 to a 10 should hook you up. Or an electric (I'd think you'd want at least a 50# thrust).