"hijack" the XM audio signal for an extra line-in on car stereo

Hi all, first post here, hope someone can help. Ok, I'm attempting to hijack the audio signal from my factory XM unit (don't use it) to the Car stereo so I can plug in my PNP Sirius unit. I've identified the 4 wires I need: left and right positive and negative wires. I have wired in this jack using the positives as left and right, and I have both negative wires connected to the common on the jack. It works, which is awesome, but the sound is not perfect. Something isn't quite right. Kinda sounds like headphones when you pull the plug half-way out....has that weird mono sound (hard to describe). The sound is WAY better with my Sirius plugged in the main AUX in the front of the radio. I'm thinking there's something I'm not taking into account. Maybe I need a "Floating Ground Adaptor" or something? Any ideas would be terrific! Thanks

The XM unit may have sent an amplified signal to the radio, or some other non line out compatible signal, without specs no way to really know.