Hi mgdaviso,
What you are describing is known as a bi-stable.
In particular that group of bistables known as flip-flops,
which in addition to having two stable conditions,
will flip from one condition to the other,
from identical pulses to the same input.
How simple would you like your components to be ?
Relays ?
Solid state ?
Integrated circuits ?
I vaguely recall an arrangement having two small neon lamps
and a few resistors and caps, but i don't recall the circuit.
That was about the simplest i can think of.
You could make a small assembly using two transistors and
a few resistors and caps to do as you describe using a press
button, there are many such circuits about.
Or you could get a ready made chip with a few flip-flops
on it, and just use one of them.
That would be the easiest.
Best of luck with it, John