Zelda Ocarina of Time help!?

Okay, so I was playing Zelda Ocarina of Time and when I was an Adult in the Fire Temple, I went ahead and found all of the prisoners in the Gerudo Valley. This was about 2 years ago and I picked the game up today beat a lot of it and just finished the Shadow Temple. Now when I have to do the prisoner thing, I cant do anything except walk around all these open gates andguardss. Im really confused, can someone help me to know where to go or how to beat this?