Well, of course it does. Think about it. Load current slowly increasing. Eventually, it gets to the point where it gets greater than the sensor's threshold, relay pulls-in, disconnecting the load, making the current go toward zero, below the sensor's threshold, so it drops out the relay, which reconnects the load, but the load current is still over the limit, so the whole cycle repeats forever.
You need to implement a latching circuit, where the relay stays latched the first time you get over current, requiring human intervention to clear the latch by clearing the fault, and then resetting the latch...
Only other alternative is a true constant-voltage, current-limited power supply. If the current limit of such a supply is exceeded, the output voltage drops below its normal voltage.