This can happen if the input pin is floating briefly during power-up before the pull-up resistor fully stabilizes the logic level. Even with a 10k pull-up, there's still a short window during boot where noise or voltage instability might register as a press.
You might want to try a stronger pull-up (e.g., 4.7k or even 1k), or better yet, add a small capacitor (like 100nF) from the input pin to ground for some basic debounce and filtering. Also make sure your switch isn't bouncing or picking up stray signals during the USB plug-in.