USB devices are hot swappable. You can plug them in and unplug them all day. Ejection is usually used for devices (USB, 1394, PCMCIA) that have mappable drives, so the OS doesn't get confused or lose data that could be in cache.
In fact I don't know of any component that you could actually physically damage the hardware by unplugging it while the computer is on. At most you would lose data.
If the game controllers don't work any more you should warranty them. If that's not possible I don't know. These electronics aren't really all that home serviceable anymore unless you really know what you're doing, and I don't.