Why don't prisoners have the right to vote in the U.S.?

I just found out not too long ago that prisoners don't have the right to vote here in the U.S. and that just seems unbelievable. I'm not necessarily saying I want them to vote, but I think they should at least have the right to if they want to. If it effects them while they're in prison, or once they're released, shouldn't they have at least some say in it?