A toroid, even with no load, still has a high in-rush - presumably as it's builds up the magnetic field?.
The reason transformers of all types exhibit an inrush current is not because the magnetic field is building up-the surge in current occurs when the magnetic field has stopped building up due to saturation of the core. This is also the reason that the surge that occurs is dependent on just where the grid sine wave is when the switch is closed. If the grid sine is just crossing zero, then the surge will be a maximum. If the switch is closed when the grid sine is at a peak, then there won't be any significant surge at all.
The magnitude of the surge will depend to a small extent on where the magnetic flux in the core was when the power was last turned off.
I have a linear power supply with a 400 VA toroidal transformer, and I put a current shunt in series with the primary winding (120 VAC here in the U.S.), and disconnected the secondaries from the rest of the circuitry. I set up an oscilloscope to capture the current in the primary and the applied line voltage. I turned the supply on and off a lot of times until I had captured the current surge when the primary voltage was applied just as the sine was crossing zero volts. The peak surge current was about 180 amps.
I then reconnected the secondaries and captured the surge current when the line was connected just at the peak of the grid sine. This should give the maximum surge due to charging the capacitors in the secondary bridge rectifier circuit. The peak surge was about 20 amps.
So, we can see that the surge due to saturation of the transformer core is much larger than that due to charging the secondary caps.
The reason toroidal transformers have a larger surge than standard E-I lamination type transformers, is that the E-I lams are butt stacked (typically with overlap of the butt joint by the next layer of laminations), and there is a significant air gap in the magnetic path compared to the tape wound toroidal transformer.
I've attached a couple of scope photos showing the surges I described. The green trace is the line voltage and the purple trace is the primary current.
Tcur1 shows the current when the grid is connected just as it crosses zero volts, increasing in the positive direction. The two very tall spikes are due to the sparking at the switch when it closes; this provides a convenient way to know when the switch closes. The current pulse doesn't happen until the sine reaches its peak. The flux in the core is proportional to the integral of the applied voltage (Faraday's law), and the core is only big enough to sustain the volt-seconds of a half sine. Under normal operation, this is just enough that the core won't saturate, since when the grid voltage gives a positive half-sine, the core is normally starting from negative saturation, not from zero flux, such as it does when the supply has been off and is then turned on.
Tcur2 shows the primary current when the secondary rectifiers are re-connected, and the grid voltage is applied at the peak of the sine. This should give the maximum possible surge due to charging the caps. You can see the dip in the line voltage where the line voltage is pulled down by the uncharged caps on the secondary. Note that the current scale is decreased to 50 amps/cm from 100 amps/cm.