While I was searching on how to replace the turn signal lens in the passenger side view mirror for my 2007 S60 2.5t, I found this:
Swedespeed Forums - Side View Mirror Lense
There is a diagram and instructions on how to remove the plastic mirror housing.
I followed the instructions and it come off fairly easily. That being said, I always cringe when I'm popping off some plastic piece on a vehicle for the first time. Good luck.