Why is there a square wave displayed on the CRO?
I can only assume that your microwave source is amplitude modulated with a square wave.
Normally the VSWR is Vmax/Vmin, but in this case could it be that the detector has a square law amplitude response rather than a linear amplitude response. (The detector is calibrated or has a response which is linear with incident microwave power?).
Not having any knowledge of your microwave test bench, it is difficult to be more helpfull.
JimB