ebay wants you to buy 5 of those cheap Chinese amplifiers, then maybe you will find one that works properly.
Are the battery cells all fully charged? Measure the loaded voltage of each one, maybe the (cheap Chinese) protection circuit is bad or is connected wrong.
The battery holder might have poor contacts.
EDIT: If the IC in the amplifier is a genuine Texas Instruments TPA3123 then its minimum voltage is 10V and your 4 cells produce only 4 x 3.6V= 14.4V. Maybe your power adapter has a higher voltage, measure it. Edited again.
EDIT again: It might be a Chinese copy of a real TPA3123.