PIC32 Oscillator Problem

[Hero.sl]

I'm using a PIC32MX460F256L on a custom board which only includes the oscillator circuit and power supply connections. Must tell you, its not perfectly designed.

The problem I'm facing is that my external oscillator is not working properly. I'm used a 4MHz crystal with 22pf caps on OSC1 & OSC2 pins. When I run the following program just to check whether the timing is correct, the delay routines takes very long than 1s.
Then I replaced the crystal with a 20MHz one and changed the necessary config bits. But still the DelayMs(1000) is taking nearly 10s.

#include <plib.h> 

#define GetSystemClock()       (80000000L) 

// For 20MHz 
#pragma config FPLLODIV = DIV_1         // PLL Output Divider 
#pragma config UPLLEN   = ON       // USB PLL Enabled 
#pragma config UPLLIDIV = DIV_5         // USB PLL Input Divider	* 
#pragma config FPLLMUL  = MUL_20        // PLL Multiplier 
#pragma config FPLLIDIV = DIV_5         // PLL Input Divider	 * 
#pragma config FWDTEN   = OFF           // Watchdog Timer 
#pragma config FPBDIV   = DIV_1         // Peripheral Clock divisor 
#pragma config WDTPS    = PS1           // Watchdog Timer Postscale 
#pragma config FCKSM    = CSDCMD        // Clock Switching & Fail Safe Clock Monitor  
#pragma config OSCIOFNC = OFF           // CLKO Enable 
#pragma config POSCMOD  = HS            // Primary Oscillator	 * 
#pragma config IESO     = ON          	// Internal/External Switch-over 
#pragma config FSOSCEN  = OFF           // Secondary Oscillator Enable (KLO was off) 
#pragma config FNOSC    = PRIPLL        // Oscillator Selection 
#pragma config CP       = OFF           // Code Protect 
#pragma config BWP      = OFF           // Boot Flash Write Protect 
#pragma config PWP      = OFF           // Program Flash Write Protect 
#pragma config ICESEL   = ICS_PGx2      // ICE/ICD Comm Channel Select 
#pragma config DEBUG    = OFF            // Background Debugger Enable 


void DelayMs(WORD delay) 
 { 
     unsigned int int_status; 
     while( delay-- ) 
     { 
         int_status = INTDisableInterrupts(); 
         OpenCoreTimer(GetSystemClock() / 2000); 
         INTRestoreInterrupts(int_status); 
         mCTClearIntFlag(); 
         while( !mCTGetIntFlag() ); 
     } 
     mCTClearIntFlag(); 
 } 

int main() 
{ 
    SYSTEMConfigPerformance(80000000); 

    TRISE = 0xFF0;          // LEDs 

    while(1){ 
         LATEbits.LATE2 = 0; 
         LATEbits.LATE3 = 0; 
         LATEbits.LATE0 = 1; 
         DelayMs(1000); 
         LATEbits.LATE2 = 1; 
         LATEbits.LATE3 = 1; 
         LATEbits.LATE0 = 0; 
         DelayMs(1000); 
    }	

}

When I select the Internal FRC oscillator and change the input divider to 2, the program works perfectly and DelayMs(1000) gives nearly 1s time delay. So I think there is something wrong with my external oscillator circuit, but I have no clue what it is. VDDCORE is connected to 10uF electrolitic & 0.1uF ceramic caps. Further more, I'm using a break out board for the PIC. So there is nearly 2cm distance in path between the crystal and the OSC pins.
Any idea where I have gone wrong? Thanks