The output impedance of the amplifier needs to be as low as possible, for a good damping factor.
Damping factor is well important, if the damping factor is naff, you can "hear the sound of the speaker instead of the music", if you know what I mean.
You'll find the Zout = milliohms. Less than an ohm I'd expect for a 386. I understand Zout is a compromise in commercial HiFi amps, it mustn't get too low for some reason.
The amplifier also needs to be able to drive the minimum impedance you plan on connected to it.
<..> the minimum specified impedance on the datasheet is 4Ω.
A "4 ohm rated" amplifier will drive a lower ohmage, provided the output level is lower than normal. Provided the package power dissipation of the IC is not exceeded, it'll be OK. After all the amplifer is just a power supply with a variable output voltage. In practice you can have a shorted speaker to an amplifier and the amplifier won't blow up until the user turns up the volume high to compensate for the low sound.
Of course using lower than standard load, the IC is not going to perform as standard in all areas, that's the art of engineering isn't it
rant mode on: If it was a commercial product, you have to remain inside absmax at all times, for legal reasons, even if one of those absmax ratings is plain absurd. Even if there is no risk whatsoever of stress to the component, no sane design engineer is willing to risk his reputation by programming himself to be a witch-hunt scapegoat, should there ever be a failure totally unrelated to it.
If everybody followed what it says in datasheets to the letter and no one took responsibility for breaking a few rules here and there, there would be no progress and improvements. and we'd probably still have today a man walking in front of every car with a red flag to ensure 'health and safety' is maintained. And if there wasn't acceptable risk managed by a responsible crew, commercial aircraft would be too heavy to get off the ground!!