Per shaneshane1 recommendation I added a 10k pulldown resister and this dramatically improved things. I also added a 100 uf [what I had at hand] bypass capacitor near the IC. This also helped.
I noticed with the switch I would still occasionally get a little occasional 'twitch' on the led.
I wired up a 555 in astable mode and hooked that to the input. I also put a bypass cap on the 555 as well since I've read that is necessary in these circumstances. When this was all connected the little twitch went away.
So I will conclude that the switch was not properly debounced causing problems,
Thanks for everyone's help. Such a simple experiment really helped me learn.