Hi,
You are saying that when you draw 150ma from the USB port that the voltage of the USB port drops to 4.1 volts? That doesnt sound typical.
There is a spec in USB that calls for the device to 'ask' the computer for 'permission' to use more than 100ma in increments of 100ma up to 500ma (USB 2.0) but usually the port puts out more current even if there is no negotiation. It could be that your particular port doesnt work like that though, so it thinks your device only needs 100ma.
Is there some way you can build this thing using only 100ma?