Help requested for modifying an intercom

First post here, so please bear with me...
I have a marine intercom, a Raytheon430, which uses a speaker to be used as a 'talkback' system, i.e. it is also used as a microphone, for hands-free use.
I want to convert the remote talkback speaker with a press-to-talk button, so that the input from the remote is blocked until the press to talk switch is pressed, allowing privacy and stopping eavesdropping.
I simplistically though that placing a diode in the circuit would work, but it introduces too much distortion.
Any other ideas?
Thanks!

Assuming it is voice activated..... Is there an adjustment for the level where the speaker becomes the microphone? If so this might be the place to put your switch.