As far as I can see the penalties only apply when crafting, harvesting and using magic - and for the two former, it seems like most crafters/harvesters run around pretty much naked any way. I will assume that even medium armor is too much for magic users.
So melee fighting - why light/medium? Any dodge bonuses or similar, or are they just there for variation (which is no shame of course

Not asking from a powergamer's perspective, just curious.