I had the negative of the servo connected to the negative of the +5v battery cell instead of ground. (First thing mentioned by Sceadwian and again by others later) So thanks for that one !
That is what you should have, the ground, the servo negative, the 5v battery cell's negative, they should all be tied together as a common ground. the negative of the 5v battery cell = ground(or at least it should).