I down-load all my mags by one round, mainly to make it easy to quickly seat a spare. Compare inserting a fully-loaded mag with inserting one that has one fewer round sometime. You'll notice a huge difference. When using a rifle, I down-load by two.
(It should be noted that I'm referring only to double-stack mags here. I don't have much experience with single-stack mags, so this might not apply to them.)