I agree. TDM is less prone to distortion and noise over an IR system then trying to do some form of analog modulation.
With TDM you don't use a separate carrier. The IR transmitter is just modulated on/off with the digital bits, similar to the signal from an IR remote control. You need some form of a sync signal to synchronize the transmitter and receiver, for example: two words of all zero or all ones. You then send the two signals in alternating sequence, such as alternating bytes from the A/D serial outputs.
That is not to say that this approach is easy, just a good approach for running two audio signals over one IR channel. Any technique to broadcast two speech signals over one IR channel will not be simple. They all involve some form of time or frequency division multiplexing.