Ok, so if you connect a 2.5V source with a 1.5Ω source resistance to the ADC pin, it works. (Two 3Ω resistors in parallel)
If you connect a 2.5V source with a 150KΩ source resistance to the ADC pin, it doesn't.
If you connect the opamp with a source/sink current of ~ 25mA to the ADC pin, it doesn't work.
I still say the opamp and the 150K pot is fighting the DRiver inside the PIC. I still think the TRIS bit for that pin is in the wrong state. A 2.5V source with a 1.5Ω source impedance can source almost an amp, so it could overwhelm the PIC's output driver and still cause the ADC to see ~2.5V.