If V1 and C3 in that circuit represent a transformer, rectifier, and filter capacitor, then C3 is
much, much to0 small to be a proper filter capacitor. Hence the regulator drops out because its input voltage sags below its Drop-Out Voltage (See Vdo on the data sheet).
To figure the correct value for the filter capacitor, do this simple calculation: (You have to make certain assumptions):
Assume a load current for your power supply is I=5A
Assume that the voltage in C3 can drop 3V (peak to peak Ripple) between current pulses from the rectifier so Δ=3V.
Assume you are using 50Hz AC and full-wave rectification, so time between pulses is t=10ms
∫I*t = q = C*Δ
So: C3=∫I*t/Δ = 5*0.01/3 = 0.017F = 16,667uF
So here is a sim to confirm the above. Pay attention to the peak currents required of the simulated transformer and rectifiers. The ripple in the sim is ~2V (not 3 as assumed) because the time that the filter capacitor has to supply the entire 5A load current is only ~7ms (not 10ms, as assumed) so the calculation yields a filter capacitor value which is a bit bigger than necessary.