Depends on the kind of modem.
Lets just say that it's a POTS (plain old telephone system) modem.
There are a number of standards from the old AFSK Bell tones through to some complex 64QAM systems.
And there are a whole family of protocols for how modems talk and agree with each other.