Right, so this is The Ridiculously Complicated PCB Drill. You set upper and lower limits using potentiometers, press a footswitch, it moves the spindle motor down, pauses, and returns. When you first turn it on it goes through a self calibration routine to correlate the digial pot with the physical position of the head. It's based on the chassis from an old CD-ROM drive with geared DC motor. Except for the digital pot and a couple flip-flops, the whole thing is analogue. Mike (one of them) called it "Mickey Mouse logic". I don't care - apart from this "wandering" which looks like mechanical slippage (it isn't) it works very well.
It's a single count from a photo-interrupter, and the gear in question is small and made of plastic. I did try sticking an optical disk on it but it didn't make any difference. Maybe it wasn't contrastey enough. The interrupter is mounted with an adjusting screw to get the best signal. The geartooth wave goes into a p-mos (bss84) which turns it into a slowish rise/fall pulse. An extra signal gets switched in when the direction needs to change, which overrides the pulse input and tells the digital pot that it's feeding into to change direction. The direction signal is open circuit when receiving pulse input. A couple of small caps set up the necessary delay for this control pulse. Then it all goes into an NPN as the output device to drive the pot.
The digital pot is.... (checks data sheet) oooohhh hang on, rising edge makes it go up, falling edge makes it go down, which is exactly what I want to happen anyway. I thought it was falling only. I designed this thing seven years ago and then shelved it after the initial build highlighted this exact problem. I'm still re-learning how it works!
So I'm chasing a red herring with this phase reversal. It was the difference in up/down pulse counts that got me onto it but it's just plain the wrong idea. Would explain why doing it actually made the problem much worse!
I think the extra signal in this experiment was just due to flakey breadboard assembly. I got it clean since writing but the pulse counts are still off.
I found part of the problem was due to vibration, also there's a little bounce (hard to spot) at the end of travel. But it doesn't explain everything.
Anyway, it did occur to me to add a second photo-interrupter as a kind of compensation,, but I'm really not sure what I'd be doing with it... I'm almost at the point of replacing the optical system with a mechanical pot, but that would make me sad