The only time it matters is if things are opposite phases, where low frequencies then cancel out. The classic example of this is wiring loudspeakers.
Mixers sometimes have phase reversal switches on microphones to resolve any possible problems - but it's by no means a common facility.
If you want to test phase reversal, try reversing the polarity of your speakers (you must do both of them, assuming stereo?), see if you can hear any difference?. Then try just reversing ONE of them, and try and hear a difference!. Lastly, with one speaker still reversed, move the two speakers close together - oops!, where did the bass go?.
An opamp doesn't have enough output current to drive a speaker directly. Its output current is limited to about 20mA peak.
You need a power amplifier to drive a speaker with Amps of peak current.
You can't hear the difference between an explosion or an implosion. But if you are worried about having the "wrong" true phase, reverse the wires on both stereo speakers or use a power amplifier that is non-inverting. :lol: