You have a low dropout 5V regulator. It is completely different to a normal 7805 regulator because to achieve low dropout its series pass transistor is a PNP that has some voltage gain, instead of an NPN emitter-follower without any voltage gain used in a 7805 regulator. Because the PNP has voltage gain then a pretty big output capacitor is needed, and tantalum is spec'd because it has much lower inductance than an electrolytic.
National Semi (not the Taiwan copy-cat company) invented the LM2940 low dropout regulator a long time ago, and it has problems. It needs an expensive tantalum output capacitor and it needs a load current.
For all my 9V battery operated 5V regulators I use the National Semi (they have been purchased by Texas Instruments) LM2931 low dropout 5V regulator. It uses an inexpensive 100uF electrolytic output capacitor and does not have a minimum load current.
The input and output capacitors should be mounted VERY CLOSE to the pins of the regulator IC. Yours are too far away.