I had a quick look at your schematic, but it was too complex for me to attempt to try to trace out. So, I'll just offer a few general suggestions.
1. You're using 555 & 556 timers which are notorious for putting horrendous glitches on the power bus. Do you have sufficient bypass caps across the 555 and 556 power pins?
2. You mention that you're using some micro-switches. Is it possible that contact bounce is causing a problem?
3. The clock inputs to your D flipflops have large value capacitors to ground that will limit the signal rise/fall time. You may be exceeding the maximum allowable rise/fall time of clock input.
Sorry! I find it hard to follow what's going on myself!
1. Ok I'll try bigger de-coupling caps. I thought 1uF ceramic would be enough.
2. Unlikely. The micro-switches are purely there as end-stops. When it's working properly, apart from the first cycle when the top one gets hit for calibration, they should never be touched. The up/down signals come from the op-amps (here used as comparators, though I think I might re-jig it to use an LM393)
3. The 74hc74 has Schmidt trigger clock inputs, so it should be able to handle the slow rise. Seems to work ok.
U7b clock input should have a resistor to ground so that it doesn't float when both diode connected inputs are low. (I'm assuming 74x74 is CMOS logic.) Ditto on the large capacitors at clock inputs -- max rise/fall time is definitely exceeded. That is no way to create a delay for contact bounce, or otherwise, if that is the idea there.
Oh, I thought I'd changed that, it should say 74HC74. But all the 7474 series AFAICT have Schmidt trigger clock inputs. Only one of the caps is for debouncing, and that is C3, combined with R6 and R4. The other two are for timing: C17 creates a delay when the head gets to the bottom of it's travel, to allow for over-shoot, (whilst Q is over-ridden by the op-amp's output (U2b)) and C1 is to create a delay whilst the digi-pot, U3, gets some pulses to wind it up to the top, though I made a mistake there and didn't allow for it's direction to be set (corrected now with yet another timing cap...)
I'll put a pull-down on U7b clock input anyway, there could be some false triggering going on.
BUT the whole flip-flop/timing thing works perfectly, the problem definitely seems to be with how the digi-pot is being set at the point when U7a gets it's clock pulse (or ramp in this case). What is supposed to happen there is, the monostable, U5(1) gets triggered when U7b changes state, provides a brief pulse to U3 c/s, so it will get the correct direction from whether u/d is low or high. The pulse also blocks input from the opto-interrupter, so that Q3 will reflect the state of U7a Q output on it's collector (since it's input comes from !Q). C22 ensures this signal is delayed long enough for C/S to get the falling edge of the pulse. So I'm really thinking the failure is occurring here somewhere, I just can't work out where.