Does height matter in a relationship?

I'm alot taller than my boyfriend and people always tell me that its weird. Is it that big of a deal or does it not really matter?