The circuits in the linked schematic are common-source amplifiers (Q1-Q2 and Q3-Q4) with active (current source) loads. They have very high unloaded gain. Their output impedances are high, so they need to be lightly loaded to maintain high gain. I simulated Q1-Q2, and the capacitor values are all wrong for an audio amplifier. I'm not sure what the original intent was, but if you make C3=100uF and C1=10uF, the simulation shows approximately 55dB unloaded gain, with -3dB points at ≈20Hz and 90kHz. Harmonic distortion for ≈1V p-p output was extremely low, <-60dB.
As Audioguru pointed out, the FETs need to be fairly well matched. If the bias current of Q1 exceeds the Idss of Q2, the gate-source of Q2 will be forward-biased, and the circuit won't work properly.