We basically don't have any real information. We seem to be playing "20 questions"
a) 12 V motor? ratings
b) Some sort of H-bridge with relays
c Relay Ratings?
If the following are true then you can implement a few structures. Anyway coast/brake/fwd/rev are possible states,
For the simple cases, Inputs can be direction and not enable or fwd/rev
You must wait some amount of time when reversing the direction and between stop/starting. You CAN arrange one relay to short out the motor, thus braking it. You can use a DPDT relay for direction and an SPDT relay for ON/BRAKE (No coast mode)
4 relays in an H bridge will give you a coast mode. A ceramic cap at the motor will help with brush noise.
Snubbers (RC networks) may have to go across the relay coils.