A resistor in series with the battery will cause the supply voltage to the Jfet preamp and amplifier IC to jump up and down as you play, causing severe distortion on the loudest sounds.
The TDA7052A amplifier was designed to use a 6V power supply and drive an 8 ohm speaker but you are overloading it with a 9V battery and a 4 ohm speaker. The overloading will cause it to overheat and it will protect itself by reducing the gain as you play music. Then the volume will suddenly drop then slowly increase over and over.
Try adding an 8.2 ohm 1W resistor in series with the 4 ohm speaker and use a new 9V battery. The resistor will cause the speaker to sound boomy and muddy on low frequencies and reduce the maximum output power to about only 0.3W.