Should I let my boyfriend tell me what shoes to wear?

So I'm 5'2, and my boyfriend is kinda short, like 5'5 at most. I personally love his height, though I know for a guy its not the best ego booster. But whenever I wear high heels or wedge heels, he gets really snarky and tells me that if I cared about his opinion or respected him, I wouldn't wear high shoes around him. He kind of makes me feel bad about it, but I also don't like anyone trying to make me dress a certain way. Am I being insensitive? Should I just not wear my higher shoes around him? I told him I would dress my way regardless, but I'm not sure if I'm in the wrong or right here.