I would bet that the major contributors will be, in this order:
- coil inductance drift
- capacitor value drift (even NPO drift can be significant, but even if small, they all add to drift)
- transistor and varactor diode reactance drift
- DC power supply voltage drift
The coil may be spec'd over a broad temperature range, but that doesn't mean it's value is rock stable.
You can compensate to some degree by having a larger lock range for your VCO, but that means increasing the Hz/Volt gain of the VCO which has, as you probably know, many other ramifications.