Found the answer through various test software.
A 48MHz clock / 4 will draw more current than a 12MHz clock despite what the datasheets say. Beware about upgrading the F320/321 for the F381 as a pop in replacement. Avoid the F381 if you want to use an external crystal. An erratum from Silabs buried deep in their website.