That's good too. That's the bahavior I expect.
You can try to look at the voltage across R24 on power up. Blanket replace C12 and C10. Particularly C12. Q12 may be missing some protections, so check Q12 out of circuit.
Another thing, you can try is to lift R20 at R18 and take a 1.5V battery and apply 1.5 V to the lifted side of R20(+battery) and speaker + (-battery). The output should mute.
Suspect components are Q12, Q13, Q14, C10, C12, R24, R23, D4, D3, R21 with the higher probablility going for C12, Q12 and C10 in that order.
If you can force the circuit to turn on with the battery trick, then you can check for diode drops at the transistors and diodes. The power on delay/mute looks like it comes from R24 and C12.