Yeah, you sure your battery isn't the problem? The thing's going to have some resistance, and if your battery is sucking down a lot of juice there will be a voltage drop.
Also, if the contact resistance is too low because there isn't enough FORCE on the contacts, that will cause it. You see that in a relay sometimes. If you get a low-ohms meter and measure contact resistance you can see it drop an order of magnitude or more when you really mash down on a set of electrical contacts like that.