It's determined by the power the transformer must handle. From that you pick a core size and then wire size based upon the number of turns needed and resistive losses as Mike stated. Sometimes it's somewhat of a iterative process.
For a Transformer, Many Wire Charts will tell you the Wire Gauge Based on the Current at 700 Mils circular area per Ampere.
(A Mil is 1/1000 of an Inch.)