The RMS output voltage is the effective, equivalent DC voltage that will produce the same power into a load. For a sinusoidal wave-shape, which is usually the case for a transmitter output, the RMS output voltage is the instantaneous peak of the output voltage (voltage between zero volts and the peak) times 0.707
It is not common to require measurement of the unloaded output as the unloaded voltage may be high enough to damage the transmitter.
It may be possible to use an oscilloscope to measure the peak output voltage and calculate the RMS voltage from the above equation (again assuming a sinusoidal waveform) depending if the oscilloscope has enough effective bandwidth to accurately display the frequency of the signal, or a good correction factor is available for frequencies somewhat higher than the bandwidth. Some oscilloscopes have a circuit to measure the RMS voltage directly and, so, the wave-shape may not need to be sinusoidal.
A calibrated diode RF peak detector can also be used to measure the peak voltage from which the RMS voltage can be calculated in like manner. Such a detector is a special, high speed diode that rectifies the transmitter's AC output (or a known attenuated sample of its output) and a capacitor follows the diode, charging to the peak voltage as measured on a voltmeter. Again, assuming a sinusoidal waveform, the RMS voltage is calculated from the peak voltage.
Another method is to measure the output power into a known resistance (usually 50 ohms) using a true RF Power Meter (as compared the pseudo RF Power Meters that indicate power based on diode peak detector voltage into a 50 ohm load). The RMS voltage can be calculated from E= sqrt(P/R) where E is the RMS voltage. This measurement will only be good for determining the voltage into a 50 ohm load, however. The advantage to this method is that the wave-form need not be sinusoidal since power is being measured, not peak voltage.
Too, a calibrated spectrum analyzer can measure the output power, and thus the voltage, again into a 50 ohm resistor only -- similar to the RF power meter method. Most spectrum analyzers can also display the power in terms of voltage on the amplitude scale. Of course, attention must be paid to the maximum power limits of all of these measuring devices with a suitable RF attenuator inserted as appropriate to prevent damage.