I've never seen a loudspeaker with higher impedance as 50ohm.
The 100V line amplifier output impedance is 120ohm, but need a matching transformer for each loudspeaker.
The problem with a 2-way, hands free intercom is that the speaker also must act as a microphone without feeding back on itself. Normally the best is between 8 ohms and 120 ohms. Commercial intercoms operate in the 45 ohm range. This seems to allow the speaker to act (somewhat) as a microphone and speaker without too much distortion. If you did a search for "speakers 45 ohm" you will find some suppliers like:
Maybe you give a try to two telephones speakers.
(the old-style type with a mechanical dial)
I think I remember them to have resistances as high as 500 ohms
The phone is good idea, but as i remember the old telephone with dial contain 75ohm speaker with steel-membrane. The Taiwan-junk wall phone contain often 50ohm speaker, and (here in Europe) the door-phone systems working with 50ohm (plastic-membrane) speakers.