Continue to Site

Welcome to our site!

Electro Tech is an online community (with over 170,000 members) who enjoy talking about and building electronic circuits, projects and gadgets. To participate you need to register. Registration is free. Click here to register now.

  • Welcome to our site! Electro Tech is an online community (with over 170,000 members) who enjoy talking about and building electronic circuits, projects and gadgets. To participate you need to register. Registration is free. Click here to register now.

Single Data Type Rounding Error

Status
Not open for further replies.

Mity Eltu

Member
I am using a single data type variable to take a running average of four numbers. I made this really simple so I could verify that it is doing what it is supposed to do, but I find it doesn't do what I think it should.

Here's the code:
Code:
'18F4520
'LCD Configuration - 4-bit lower
Define LCD_LINES = 2
Define LCD_CHARS = 16
Define LCD_BITS = 4
Define LCD_DREG = PORTD
Define LCD_DBIT = 4
Define LCD_RSREG = PORTD
Define LCD_RSBIT = 0
Define LCD_RWREG = PORTD
Define LCD_RWBIT = 1
Define LCD_EREG = PORTD
Define LCD_EBIT = 2

Define CLOCK_FREQUENCY = 8
Define SINGLE_DECIMAL_PLACES = 6
Define SIMULATION_WAITMS_VALUE = 10

'Global Variables
Dim flags As Byte
Dim vals(4) As Single
Dim i As Byte
Dim ii As Byte
Dim avg As Single
Dim num As Single
Dim cnt As Single

main:
    OSCCON = 0x72  'Internal 8MHz osc
    Gosub setup
   
    Lcdinit 0
   
    Lcdout "Ready"
    WaitMs 1000
    Lcdcmdout LcdClear
    Lcdout "Average = "
   
   
loop:
    'running average:
    'array of 4 numbers, [0-3]
    'first four are loaded with index 0-3, summed
    'and divided by 4
    'then  start replacing each number 0-3 where
    'index 0 holds the oldest, and hence, the
    'smallest number
   
    If flags.0 = 0 Then  'new number coming in NOTE: I made this so it would just run instead of waiting for the timer interrupt
        flags.0 = 0
        num = cnt * 1.10000
        vals(i) = num
        avg = 0
        For ii = 0 To 3
            avg = avg + vals(ii)
        Next ii
        avg = avg / 4.00000
        Lcdcmdout LcdLine2Clear
        Lcdout #avg
        i = i + 1
        If i = 4 Then i = 0
        cnt = cnt + 1.00000
    Endif

    Goto loop
End                                              


setup:
    INTCON = 0xd0  'int0 en
    INTCON2 = 0xc0  'pull ups disabled, int0 on rising egde
    PIE2.1 = 1  'tmr3 int enabled
    RCON.7 = 1  'priority ints enabled
    IPR2.1 = 1  'tmr3 int hi priority
   
    CCP1CON = 0x00  'cap / comp off
    CCP2CON = 0x00
   
    SSPCON1 = 0x21  'spi on, fosc/16, clk idel state low
    SSPSTAT = 0x40  'sample in middle, cke=1 - xfer on clk high to low
   
    ADCON0 = 0x00  'A/D off
    ADCON1 = 0x0f  'all digital
   
    CMCON = 0x07  'comp off, digital io
    CVRCON = 0x00
       
    HLVDCON = 0x00  'Low volt detect off
   
    TRISA = 0x00
    TRISB = 0x01
    TRISC = 0x00
    TRISD = 0x00
    TRISE = 0x00
   
    LATA = 0x00
    LATB = 0x00
    LATC = 0x00
    LATD = 0x00
    LATE = 0x01  'porte0 set for MAX6675 chip select

    'set timer3 for interrupt for logging new number
    'for demo use timer0 and 256 prescale
    T3CON = 0x30  'tmr3 1:8 (262mS) prescale, off
   
    flags = 0x00
    i = 0  'array index
    cnt = 1
Return                                           


On High Interrupt
    If INTCON.1 = 1 Then  'int0
        INTCON.1 = 0
        flags.0 = 1  'set val to be added
    Endif
   
    'if tmier ovf, set flag1 to get new number
    If PIR2.1 = 1 Then  'tmr3
        PIR2.1 = 0
        flags.0 = 1
    Endif
Resume

And here is a list of the averages it gives:
0.275000 - correct
0.824999 - incorrect. should be 0.285
1.649999 - incorrect. should be 1.65
2.750000 - correct. weird
3.849999 - incorrect. should be 3.85
the rest are incorrect from here out following the same behavior of x.xx9999.

Is there some setting I need to change or is this just a function of the single precision data type? I don't believe there is a round command in the basic compiler, so is there a simple way to fix this?
 
hi Mity,
This is your program on my PC, I have used the IDE Uart to show a printed record of the 'sums'.
I have noticed these rounding errors when calculating GPS data angles etc.

Eric.
BTW: I have spent ages on the SD card trying to get some of the commands working in a reliable way, looking forward to seeing your latest SD prog.
 

Attachments

  • A36.gif
    A36.gif
    16.5 KB · Views: 292
I am using a single data type variable to take a running average of four numbers. I made this really simple so I could verify that it is doing what it is supposed to do, but I find it doesn't do what I think it should.
.........

And here is a list of the averages it gives:
0.275000 - correct
0.824999 - incorrect. should be 0.285
1.649999 - incorrect. should be 1.65
2.750000 - correct. weird
3.849999 - incorrect. should be 3.85
the rest are incorrect from here out following the same behavior of x.xx9999.

Is there some setting I need to change or is this just a function of the single precision data type? I don't believe there is a round command in the basic compiler, so is there a simple way to fix this?

Mity, what you are seeing is the "single" precision accuracy, which is 6 to 7 digits (mostly 7). That is what causes the "9999" types of answers (I assume the 0.824999 you really mean 0.284999). This is a common error in any single float, it cannot guarantee the last digit, it could be +/- 1 digit. You can trim this with some form or "rounding", but I don't see that function in this compiler...
Using float (single) variables with 7 digit precision will always have +/- 1 digit error for the last digit. IEEE 754 standard clearly states "This gives from 6 to 9 significant digits", meaning you can only be 100% sure of the first 6.
 
I did a complex breakdown of floating point format some time ago!!! It's a very weird way of representing a number but there isn't anything better....

Oshonsoft does have the DECIMAL_PLACES directive but I don't know whether it rounds up or down ( I would have thought down )

In a single there will be 24 mantissa bits and unless the decimal fraction is weighted to a "achievable" bit there will always be re-occurring 3's, 6's and 9's.... In fact any number that doesn't divide into 10 so unless the last digit resolves to a 1,2 or 5... trouble... The though behind it.

2.499999 is only 0.000001 in error so who gives a jot!!

The mantissa fractions divide down bit by bit, each bit is weighted so:-

0.5, 0.25, 0.125, 0.0625, 0.03125... etc... so there WILL be numbers that will be out of reach
 
I understand, from the big picture kind of perspective, how this single precision stuff works, but these numbers are all well within the 7 significant digit representation. The compiler, for whatever reason, rounds everything down, so that 0.284999 comes to be 0.284 when the decimal places is set to 3. That makes no sense to me whatsoever, but it is what it is.

I guess I'll have to deal with that or switch back to sourceboost c++ and try to bang out the sd card interface.

I think I'd rather have mu gums scraped.

Thanks.
 
The seven decimal place can be hard to achieve with a single precision... It has nothing to do with Oshonsoft, but some compilers will cater for the tiny discrepancy... When the number overflows it should technically be rounded up.. Online calculators do... But Oshonsoft is a cheap compiler..
 
Hi,

I'm also having the same problem.

I think I read somewhere that only '1' single can be used in each program. Is this correct?

For what I need: There are 4 numbers with digits after the decimal place. I don't need complete accuracy, so if I multiply them up till they are whole numbers, i,e, (For 100 multiplied by 'say' 0.001) will 100X1000 solve my problem?

Camerart.
 
No, you can have more than one single in a program. The number of singles might be limited due to their size (32 bits), but as long as there is space, you can have as many as you want.

If you are going to scale them up by 1000, you will need to make sure that the data type you have selected for the answer is large enough to handle the largest POSSIBLE result. For instance, lets say you decide t use the following:

Dim num1 As Single
Dim num2 As Word

num1 = 0.001
num2 = num1 * 1000

This will work just fine, but if num1 is ever larger than 65.536, your result will be incorrect as a word data type cannot hold a number larger than 65536.

Hope that helps.
 
I understand, from the big picture kind of perspective, how this single precision stuff works, but these numbers are all well within the 7 significant digit representation. The compiler, for whatever reason, rounds everything down, so that 0.284999 comes to be 0.284 when the decimal places is set to 3. That makes no sense to me whatsoever, but it is what it is.

I guess I'll have to deal with that or switch back to sourceboost c++ and try to bang out the sd card interface.

I think I'd rather have mu gums scraped.

Thanks.
The compiler does not round down, but truncates the number, when displayed.
It can be handled, for example to display three decimal places, add 0.0005 to the number before displaying it.
 
No, you can have more than one single in a program. The number of singles might be limited due to their size (32 bits), but as long as there is space, you can have as many as you want.

If you are going to scale them up by 1000, you will need to make sure that the data type you have selected for the answer is large enough to handle the largest POSSIBLE result. For instance, lets say you decide t use the following:

Dim num1 As Single
Dim num2 As Word

num1 = 0.001
num2 = num1 * 1000

This will work just fine, but if num1 is ever larger than 65.536, your result will be incorrect as a word data type cannot hold a number larger than 65536.

Hope that helps.

Hi M,

Yes it helps, thanks.

If the number reaches larger than the POSSIBLE result, you say it will be incorrect. In simple terms could give two numbers that are the same, result in not being the same?

C.
 
Hi,

I'm also having the same problem.

I think I read somewhere that only '1' single can be used in each program. Is this correct?

For what I need: There are 4 numbers with digits after the decimal place. I don't need complete accuracy, so if I multiply them up till they are whole numbers, i,e, (For 100 multiplied by 'say' 0.001) will 100X1000 solve my problem?

Camerart.

If you don't need decimals, you can simply cast the floating point number to an integer.
dim a as single
dim b as word
b=a ' b will be the integer part of a

What is the range of your variables? If > 65535 then they must be long variables.
 
If you don't need decimals, you can simply cast the floating point number to an integer.
dim a as single
dim b as word
b=a ' b will be the integer part of a

What is the range of your variables? If > 65535 then they must be long variables.
Hi J,

I've been trying to use decimals (SINGLE), but they never rounded up properly, what ever I tried. I made a test program using LONGs and no SINGLES. multiplying the decimal points out of the variables, then dividing them down to what they should be, and this appears to be doing what I want.

EDITED.

Thanks, C.
 
Last edited:
Status
Not open for further replies.

Latest threads

New Articles From Microcontroller Tips

Back
Top