Hi Willen,
What is there to figure out here? The balun can only hook up two different ways. You have a primary and a secondary. Call either end primary then call the other end secondary.
Connect the primary to the FM transmitter and the secondary to the antenna. Try the transmitter out by transmitting something.
Then remove the balun, connect the secondary to the FM transmitter then the primary to the antenna. That's the second way to connect it. If it works better with primary to FM transmitter then keep it that way, or if it works better with secondary connected to FM tranmitter then keep it that way instead.
Do you know the impedance of your transmitter and antenna? Those TV baluns are made for 75 to 300 ohm coupling usually. Since that's a 1 to 4 ratio, if you have a 50 ohm antenna and 200 ohm transmitter it might also work for that.
You could look up the impedance of a dipole antenna, the kind you have, on the web.
There's also a chance you can rewind the balun to match your impedances if 1 to 4 doesnt work, but you'd have to know the impedance of your FM transmitter and of the antenna too.