Arduino picking up button presses on power up of the board

So I am starting to get into smd soldering. I thought I'll build myself a smd ovwn
I have a button on it that allows you to switch between reflow soldering and just preheating the board.

Occasionally I find when I plug usb pwr in, Arduino picks up button presses. I do have a 10k pull up on the switch input. So why am I experiencing this strange behaviour