You're waveforms are somewhat convoluted - took me a sec to figure it out. The pic you show of the SW pin is really depicting the OFF time of the internal FET, which is about 400nsec. The voltage on the SW pin is about 15V and the input is 5V, so the voltage across the 10uH inductor is about 10V. This yields a peak inductor current of about 400mA - which is in line with what the datasheet says - the switch turns off when it hits 400mA.
The switch turns on when the voltage on the FB pin has fallen below 1.233V. With the input at 5V, the switch will be on for about 900nsec (800nsec plus 100nsec prop delay) for it to reach the limit of 400mA. Looking at the other waveform you show of the voltage on the FB pin, this seems to fit also - the switch turns on when the voltage jits 1.233V. You don't see anything happen to the voltage on the FB pin until the switch releases 900nsec later - which with the rate of fall of the voltage on the FB pin (approx -180mV/usec), the bottom of the FB voltage should be at about 1.07V - which is fairly close to where it is. It appears that the IC is doing what it is supposed to be doing. I guess now we need to figure out how to make it do exactly what you want it to do.
Oh - if you look at the waveform for the voltage on the FB pin, the rate of fall is about 182mV/usec. With a 0.1uF capacitor and the voltage falling fairly linearly, you can assume that the current coming out of the cap is all due to the LEDs and that calculates out to be about 18mA.
I think what you're going to have to keep in mind is that 1.233V divided by the sense resistor is not going to be the average current through the LEDs. I don't really have the time to figure out the average voltage across that resistor, but it is going to be a function of the input voltage, max inductor current (400mA), the output capacitance and the current through the LEDs. When the switch turns on, there will be a delay which will cause the FB voltage to fall below 1.233V before the switch turns off and the inductor dumps current to the output cap. That delay will be: tdelay=400mA*L/Vin. (tdelay is really just the ON time of the FET)
When the FET turns off, the inductor will dump the current into the output capacitor. We know what the peak inductor current is (400mA) but we really don't know what the voltage on the SW pin will be - we can assume 15V from the scope shot you supplied, though. So while not entirely accurate:
Toff=400ma*L/(15V-Vin)
This time can approximate the peak voltage the output capacitor will charge to. We know the inductor current will decrease linearly from 400mA to zero in the time Toff. That current is put directly into the output cap (ignoring the constant current draw of the LEDs). This can be put into equation form with :
iL=400mA-(15-Vin)*t/L
With iL=Cout*dv/dt, solve for dv in terms of dt, then integrate from 0 to Toff and you should get a rough approximation of the ripple voltage on the output capacitor. If I did it right it should be:
Vp-p=Cout*L*(400mA^2)/(15-Vin)-C*((15-Vin)/(2*L))*(400mA*L/(15-Vin))^2
With all that info, you should be able to reconstruct the ripple voltage across the current sense resistor, transform that into an average current calculation and then use that equation to solve for the value of the current sense resistor that you want for a particular average LED current. Keep in mind that as the input voltage varies, so will the output current.
Hope that wasn't too confusing, I wrote in in a stream of consciousness type manner.