You'll always see the headlights dim when pulling heavy loads. You need some massive power out of the alternator to never "dip into the battery." And that is with your RPM's up to at least around 2500 or higher to actually see the rated output, if you're just letting it idle while winching then the alternator is barely putting anything out.
The M8000 (like most winches that size) can pull over 430 amps at full load. Even at half load it's nearly 300 amps. The factory alternator on the 3.5L is rated at 140 amps if I recall but that would only be that much at probably 2,500 to 3,000 RPM engine speed. You'd need at least a dual 200-250 amp alternator setup to not need the battery for supplemental power if you plan on winching at full (heavy) loads. Those alternators are stupid expensive though (starting around $350 for the cheap ones and up from there) and I don't think it'd be very easy to add a dual alternator setup to the 3.5L since you'd need to make custom brackets most likely, and the 5.3L which you can find actual kits for leaves little room to actually fit them in with our tiny engine bay (the 5.3L barely fits by itself!).
The whole purpose the battery serves is two fold. One is obviously to start the engine. The other though is to act like a reserve of power for when the power requirements are higher than the alternator can provide. Mainly when sitting at idle at stop lights and drive through's when the people inside are taking for bloody ever. But then you get moving again and the RPM's go up and the alternator can go back to providing the power and recharge the battery. And if you're like us it's to provide reserve power to get your sorry butt out of a sticky situation while winching out.
Your most cost effective option is a 2 battery setup. Or even cheaper (and better in my opinion due to versatility and simplicity) these days is to just keep a lithium jump pack in the truck (basically a 2 battery setup but the 2nd battery is MUCH smaller and it's portable!). And to reduce the load on the winch you can run a pulley block to half the load on the winch/double your pulling power for the same amp draw.
How much winching are you actually planning on doing? I run a group 31M AGM deep cycle (though I hear it's better to use a regular starting battery for winching due to the high amp draws) and have never drained it dry while winching, though I don't really do much winching at all since I just don't get myself into situations it's necessary. It's an insurance policy, not a ticket to be careless. More often you're better off doing a strap than a winch, unless you're by yourself but then you have to deal with finding something to actually hook to. Might be easier up in Canada with all them trees, but I bet as most of our luck tends to be, you'll end up stuck in a spot with nothing to hook to. Maxtrax are a better investment with that regard.
Doing a course every now and then is not something I feel warrants sinking $500 at a minimum for a single 250 amp alternator (because don't forget you'll need to update the wiring to handle that amount of draw without melting anything) and if you want to go dual 250's to have sufficient power to not use the battery then you're looking at well over $1000 once you get done with brackets, belts, alternators, wiring, etc. And that's just for the parts, labor is going to cost you a good amount more if you have to have someone else do it for you.
If you're going to winch competitively, which is the only time I could see this level of investment being justified, then you need to be looking at a different rig entirely for that in my opinion. I don't think the H3's frame rails could handle repeated heavy winching on a frequent basis. Something will definitely tear out after a short while without reinforcing the frame horns substantially.
All that being said, it's your rig so if you want to go that route and have the coin for it, by all means do. Just make sure you post it all here for everyone to see! It's always neat to see what ideas people come up with and how they make them a reality.
