I've had similar conversations with co-workers who use "retarded" in an off-hand way. What people don't seem to realize is that mental retardation is a very serious topic for those of us who deal with it everyday. The colloquial use of retarded is not just to describe something as bad, but as bad in a funny way. Many people object to the use of retarded to mean bad, but I also object to it meaning trivializable.
Objecting to this usage can be a tricky line to walk, since most people don't use the word intentionally to hurt people, and especially since it is used humourously, it can be awkward to make a point about it. The people who say "retarded" don't mean to trivialize the disabled, but that is what they are doing. It hurts to hear people dismiss an entire population so lightly.
If you find yourself using the word retarded to make people laugh, would you do me a favor and think twice? Thanks.