If the coax does not match the output impedance of the balun (& input impedance of the antenna) it will change the apparent impedance dependant on the coax length.
If the analyser is 50 Ohm unbalanced, it's not going to work well on a balanced signal.
Measure at the balun input. If the balun is good, any antenna mismatch will be transformed through that; a match is a match at both impedances.
The link cable should be a multiple of half a wavelength, allowing for the velocity factor of the cable you are using, and as short (few half waves) as practical.
(An electrical half wavelength will give a 1:1 impedance transfer).