Post the code your really using there no way to help you without it.
Swordfish doesn't like osccon setting in the main code I've had the osc not start up to full speed hours after the chip is powered on.
That is an excellent point Burt and I had forgotten it. Any initialization done in Swordfish modules will be done before any code executes in the main program. If the chip defaults to a slow clock speed, it can take a long time to reach oscillator settings in the main program code.
There's a reason I don't use the internal oscillators – it can be troublesome getting them setup correctly.