I = E/R, so if the motor winding resistance is 3Ω, and the supply voltage is 12V, then I=12V/3Ω = 4A per winding. Is the motor rated for 12V to be applied to the winding? I'm guessing that the motor is looking for ~5V per winding. I am not surprised that things seem to be getting too hot...
Is the motor heating up, or is the L298n heating uP, or are both heating too much? The L298n will not stand 2 x 4A without using PWM to reduce the effective winding current.
Before you go building an H-bridge, you need to get complete specs on the motor, and then match the power supply voltage to the motor rating.
If you need to reduce 12V to some lower voltage to match the motor, then you can get a separate switch-mode buck converter to do that, and then run the H-bridge on the lower voltage.
Alternatively, you can use PWM in the H-bridge to reduce the effective "holding" current to the motor windings while retaining the 12V supply voltage. Even the L298 might work with the proper PWM input applied to it.