"Socket testers" are a common, standard item, eg.
To run an LED from 120V or 240V AC, you can use a capacitive dropper. They use the AC impedance of a capacitor:
1 / (2 x PI x F x C)
in place of a high value, high power resistor
A 0.22uF capacitor has an impedance of around 12000 Ohm at 60 Hz.
That would give about 9mA at 110V, or 19mA at 230V
With a bridge rectifier that's the current the LED would get; with just a parallel reverse diode, it would average half of that.
You also add a relatively low value resistor to limit surge current for when the circuit is powered on.
It's a very common circuit principle, in things that can be totally insulated so there is no possibility of anyone touching the parts - they are live at lethal voltages!
The image below is an example I found on google, though I think they have calculated the current incorrectly.
You would need that twice, one for each LED.
Connect the resistor input to the live or neutral pin, and the lower connection to the ground pin.
Preferably use X or Y type capacitors, or one rated at least twice the maximum AC voltage, for reliability.