I just now saw this thread. Subscribed and hoping to see you make progress on this
I'll try to offer some pointers, but most of it is already documented in my thread about how I reverse engineered the communication.
I started by probing each wire with an oscilloscope, and observing.
If there are simple analog audio signals for speaker and microphone, those should be very easy to identify: try doing something on the phone that produces sound, and talking into the microphone, while watching the oscilloscope (of course you'll have to play with the setup of the scope to make sure you are able to see anything that happens). If you see analog waveforms that correspond with the sound, then you've got something.
For digital communications, you first have to figure out what type of communication is used. I got lucky that my phone used very simple UART communication (one wire for TX, one wire for RX; clearly distinct individual bytes of data; can't get any simpler than that!). I know some phones (Motorola?) use some kind of proprietary 3-wire serial data communication.
Once you determine the digital communication type, then you have the big puzzle of figuring out what the data actually means. Again, I got lucky that my handset sends simple 1-byte messages for button presses, etc., which was very easy to capture and identify.
For the transceiver -> handset communication, I just recorded the data sent while performing various operations on the phone, and started looking for patterns of how the data related to what was happening on the handset display.
A UART-USB adapter and RealTerm software was an amazing tool that allowed me to both capture sequences of serial data, and produce sequences of serial data sent to the handset (often one byte at a time) to really see exactly how the handset reacted to each byte of data "in slow motion". Again, I got lucky that the protocol was very simple: nearly all commands to the handset are single-byte commands. And ASCII characters for printing text were very obvious.
Good luck!