Alec_t briefly mentioned lack of hysteresis in the circuit, and that was also the first thing that I noticed. Since you don't show details of what is connected to the + and - inputs of the comparator, it's impossible to tell exactly how the circuit was intended to handle situations when the input signal(s) vary when close to the trip point. Since you infer that the original circuit was tweaked until it worked, then it may have always been borderline, and some component value has now drifted far enough to cause problems. In my opinion, in this kind of circuit, there needs to be some hysteresis to prevent relay chattering. I would connect a large value resistor between the comparator output and the + input. As a starting value I'd try a resistor that's about 20 times the impedance of the source that's feeding the + input.