I think you can get a NMOS-type MOSFET transistor to do this too. It is a 3 terminal solid-state device (it doesn't physically move to do what it does). The source(S) and drain(D) terminals go in series with whatever you want to switch on and off, but the voltage across them has to be above a certain level (it shouldn't be much of a problem, it depends on the transistor but its usually just 5V I think). The gate(G) is where you apply the on/off signal. If the G-to-S voltage is below a certain level (say 0V), it "shuts off" and if the voltage is above a certain level (say 5V) it turns out. It works more cleanly and quietly than a relay, but there is a very small leakage current when it is shut off. It also uses much less power than a relay to keep it turned on.
To get the switch behavior you want, I think you can use a pull-down resistor in the gate.
I am unsure how it will behave if power is connected to S & D but G is left floating...
D
|.....................................Power switch with other end connected to +V
|........................................|
|........................................|
-------G ---------------------|
|........................................|
|.....................................Pull down resistor with other end connected to 0V (or -V)
|
S