I assume the 3 phase breaker will trip if the phase and neutral currents don't add up to zero. I have 3 phase breakers in my house, I can arbitrarily turn on single phase appliances and therefore have very out of balance phase currents, the overall current, including neutral, is however still zero. My recollection is that an out of balance current of 35mA is required to trip an RCD.
I assume there are also breakers that detect out of balance phase currents to protect motors etc.
Mike.