If you have a power supply set to 4V, and you use a clip lead to connect the stopped motor to the power supply, the supply will happily deliver the initial inrush current required to start the motor. As the motor comes up to speed, the current will reduce to your measured 30mA, but during the inrush the current is likely to be 10 times that for ~1/4sec. The solar panel wont deliver the inrush current.
Connect a 1000uF capacitor across the panel (observe polarity). Put it in sunlight, and then connect the motor after the capacitor has charged. The capacitor will deliver the starting surge, and then the panel will keep the motor running.