If you still want to do what you describe in your original post (i.e., a fixed amount of movement for each press of a directional button), you need feedback from your motor to tell your arduino the servo's position. How do you propose to use a single transistor to do that? Have you drawn a circuit for it?
Servos without power do not return to their original position. True, power-off holding ability can vary greatly between different servo types, as resistance to movement is due mainly to the gear train and the motor. "Coreless" servos have very little holding ability without power. When you apply power, your servo will assume the position the programs tells it to assume. Maybe your arduino reboots in that "zero position" state. That is a programming issue, not a mechanical one.
John