Do College/University rankings matter?

No, college/university ranking does not matter. What matters is that if the colleges you are applying to are the right fit for you. "Right fit" means that the resources and educational and extra-curricular opportunities offered by the college matches your interests. Enrolling at a university/college where you would thrive and excel is important.

I completely agree with Srihari. Although college rankings might bear some significance (however little) while applying for jobs, college is what you make of it. If you are enjoying what you are studying, the people around you, and the campus atmosphere, rankings don't mean much.