Is my pony too small for me?

Not at all! You guys look great together! And no offense, but he's a pony. And anyways, someone already said this, but the way he's stocky makes it better how you don't look as tall on him.