You need data on just how "unstable" this is (noisy?).
There is nothing wrong with the 16F676 ADC, it should perform as specified. Read the specs and the errata. IIRC, noise is only like one or two codes unless you reduce Vref.
Noise reduction is most effective when your sampling meets the Nyquist criteria of the noise, that is, it is sampled at twice the noise frequency. Then you can write a lowpass FIR filter. If your sampling does not meet Nyquist criteria, lumping a lot of values together and averaging still carries a random component and the results will be inconsistent. After all, pull out a calculator and average sin(x) 16 times, making up 16 totally random numbers for x. The result won't average to 0. It could in theory be very high too, up to 16. If you meet the Nyquist criteria with the sampling period, with enough samples and a digital lowpass filter, it WILL be close to 0.
1. You need a better signal filter. An RC filter of 1K, 0.1uF has a cutoff of 1.6KHz and a pretty gradual rolloff too. In short this cannot filter the 120Hz of rectified/filtered 60Hz ripple at all, you need a far bigger cap. Go for 10uF at least. Don't increase R above 2.5Kohms or the ADC will become less accurate.
2. Your analog reference voltages may be unstable.
3. You've written in a math error somewhere, perhaps causing a Divide by Zero or overflowing an integer if certain ADC values are encountered.
4. You're not meeting the required Sample/Hold times of the PIC ADC. I don't know how much control the 16F676 gives you there.