... suggestion ...
If you have a working, open loop prototype of your physical plant, that is, a magnetorquer pair, with a rotational mass, and a magnetometer to provide angular position data, including the h-bridges and power supply, then take a look at the Matlab System Identification Toolbox.
The idea is to use the SI Toolbox to get a Laplace Transform version of the open loop, i.e. non-controlled, plant model.
As I understand it, the basic idea is to use a swept frequency sinusoidal signal as an input to the plant, and obtain the resulting output variable characteristics. If you could find a way to get appropriate data files for the output angle magnitude and phase, that might be sufficient for input to the software. As part of this step, you would have to have a practical means of transforming the sinusoidal signal into a PWM form that could be used by the h-bridge.
A separate approach to input data to SI Toolbox is to use first principles, which consist of equations of motion and mathematical relations. The problem here is that you have certain elements, such as the h-bridge driver, which are not be easily modeled.
If you can obtain the experimental data files resulting from the sinusoid input, then everything would be encompassed, and you don't have to worry about a mathematical model from first principles.
Once you have a Laplace Transform of the open loop plant, you can choose a control law that actually makes sense. It might be some form of PID. It could be a type of pole/zero filter. When you are able to plot the open loop root locus of the actuator and physical plant, you will be able to look at the control situation with insight.
HP used to make a sophisticated machine to accomplish this modeling process.
HP's device used a swept sinusoidal signal.
I think that you can do the same thing with Matlab System Identification Toolbox.