If you can produce a 'square wave' at the desired voltage/pwoer requirements, you might be able to use it for low power devices that use switched mode power supplies, such as mobile phone chargers, printers, small LCD tv's, DVD players etc.. since these first rectify the mains intput to DC anyway. And even though they rely on the peak value (~300v DC from 240 AC) most SMPS will accept a wide input voltage range, some from 90-250V - so peak voltage isn't always an issue. Of course, a square wave is significantly noisier than a pure sine wave, so I can't see it not interfering with audio.
There are pseudo-sine-wave inverters (again for LOW power) that produce the output via resonance rather than digitallly controlled wave shaping or PWM. A resonant royer converter generally has a fairly clean output, but using off-the-shelf mains transformers in reverse, the output frequency would be much higher - again, not always a big deal for appliaances which immediately rectify the mains input). They won't have square wave outputs, but not true sine either - just less noisey and easier to filter.
Oh and the purpose of the 2200uF cap is to block the DC offset, so the transformer see's +4V and -4V with reference to ground, as opposed to 2V and 8V. In the first case, the net/average magnetic current in the transformer is zero. In the second case, the magnetic field builds, and never colapses giving rise to saturation. - bit out of my depth on this one though