2002 base model S60.

The other day I noticed that the button on the front of the shifter was cracked and broken. It still works and I can't see it but it bothers me.

I can't find my Haynes manual so I went online and found some Youtube videos. According to the videos, I should be able to just grab the shifter ****, pull up, and it will come out. I'm not a small guy and I'm pulling with all I can muster given the angles and nothing budges.

Anyone have any advice?