Someone I know once fell into idolatry. Not the kind of idolatry you may be thinking of, no, he fell into idolatry of the Pope. Then, when the Pope said something that sounded contrary to what we know of the faith, this man ended up falling away from the Church. Yet, although we do not want the Pope to say something incorrect, there is nothing in our faith that says that the Pope is always correct in everything he says. If, however, we idolize the Pope, then when we realize he is human and can fall into sin just as much as the rest of us, we are crushed by it. We as Catholics believe the Pope is infallible only when He speaks Ex Cathedra; we do not believe that the Pope is sinless.
The reason idolatry is wrong is because only God--Father, Son and Holy Ghost--is worthy of our total devotion and worship; all else is created and not divine. Thus, to give devotion or worship to something other than God is idolatrous and evil (and it has vast numbers of consequences that we do not even realize).
This means that we need to seek to determine what level of devotion we have to those other things. Should we be honorable and respectful towards the Pope, even if he says something that is wrong? Yes, of course we should. Should we, furthermore, appreciate the Church, the Mass, the rosary, and numerous other holy things? Yes, we should, but they are not God Himself, and thus as much devotion as we may have to them, we must never idolize them and grant them the same reverence that we accord only to our Divine Creator and Judge.
Do not idolize any part of our faith; whether it be the Church, the building we have Mass in, the Pope, the Bishop, the Mass itself, the Blessed Virgin, the Angels, they are all subject to Christ exactly as the rest of us are, and we need to grant them the respect and honor that is their due, but never grant the respect and honor that is God's alone. Check yourself; be certain of what your commitments really are, and where your heart is truly at. Idolatry is always a grave sin.