If you think your riding does not affect the horse...

Imagine that you have to give someone a piggy-back ride every day. They are difficult to carry around: they sway from side to side/have no balance, or they lean to one side more than the other. If you carried them consistently, you would have to compensate for this lack of balance and/or asymmetry by using one side of your body more/differently to carry the weight. Each day, this would change the way you move. Changing the way you move would affect you physically: you would start to feel sore, your knees might hurt, your back would ache. Not to mention how frustrating this must be. 

This is what your horse is dealing with...and he can’t yell at you verbally.