1. By pressing the button, the circuit randomly selects an active peg with no indication of which one it is.
2. The player starts removing pegs one at a time.
3. There is no indication when an inactive peg is removed.
4. When the active peg is removed, the relay closes to do something.
5. When all pegs are returned, the relay opens.
6. If all pegs are returned AND the button is pressed, the game restarts.
Yes ___
No ___
If yes, you need a 5-input AND or NAND function to detect when all pegs are in, five individual 2-input AND functions followed by a 5-input OR function to detect when the active peg is removed, an oscillator, and the ever-popular 4017. Note that I'm talking about functions, not actual logic gates. Except for the 4017, everything can be done with a small number of discrete transistors. For example, the 5-in NAND can be an 8-input gate in a 14 pin package, or five 1N4148 signal diodes and one 2N4401 transistor, or five sections of a ULN2003. Not automatically the right way to go, but neither is a fist full of CMOS.
All of those Karnaugh-state-boolean thingies notwithstanding, you have only two things to resolve with logic: a) that the active peg has been removed; and b) that all pegs are back.
There are a lot of ways to detect the pegs. With optical you have to make sure the light from adjacent pegs doesn't confuse the sensors. One way to do this is reflective. There are single-piece reflective sensors with an LED and phototransistor that have a very short detection range. Put a little aluminum foil on the bottom of a flat peg and you can sense when it is within 1/2 inch of the sensor. Or have the peg push one springy piece of metal into another as a leaf switch. Or buy a $1 limit switch with a leaf arm, mount it vertically, and have the peg actuate it as it slides by. That's probably the most reliable if you can control the diameters of the peg and its hole so there isn't much axial wiggle. It's like plugging headphones into a jack, but without the high insertion and extraction forces.
ak