If both computers are on together, then, as you said, there is no need to switch the 5V supply.
The 4066 IC could be supplied by this 5V.
The control pins of two switches (belonging to D+ and D- of one computer) could be connected together and made at ground level (for off state) by a common resistor (say of 10K). By using 1P2T switch, the 5V could be applied on the switch control pin (10K) of one of the two pairs.
Edited:
But if you can use 2P2T switch, the 5V could be switched too.