Odd Arithmetic Results

[dkatt]

Hello Everyone,

My curiosity is killing me, so perhaps someone can help me understand why I stumbled across these unexpected results not only in ZBasic, but VB6 as well.

Using the following code, one would expect increasing a single-type value by 100th value increments should result in a consistent increase in likewise results, however when SVal reaches 0.10, I suddenly see 0.0999999(?). I wondered about this so ran this same code in VB6 and got 9.999999E-02(??). See the table for the other odd values too. Whats more, if I change the initial SVal to 0.05, then 0.10 comes out okay, but 0.12 is now 0.1199999(???)

I suppose I can wash it all away using the Fmt function and fancy string/single conversions, but still, why would I need to do this? Is this some feature of the underlying arithmetic operations which I have no control? Is the SVal actually 0.10 and the CStr function changing it to 0.0999999? Can I keep/coax/force the operation to consistently produce a nice and tidy two-place single value?

Any enlightenment would be greatly appreciated.

-dkatt
Newbie ZX40 User [and liking it]

Code: Select all

Dim SVal as Single
Dim LCnt as Integer

Sub Main()
  SVal = 0.00
  Do
    LCnt = LCnt + 1
    SVal = SVal + 0.01
    Debug.Print CStr(LCnt); ":"; CStr(SVal) 
  Loop until LCnt = 25
End Sub

[table]
[mrow]
Expected
[mcol]
ZBasic Output
[mcol]
VB6 Output
[mcol]
SVal = 0.05
[row]
1:0.01
2:0.02
3:0.03
4:0.04
5:0.05
6:0.06
7:0.07
8:0.08
9:0.09
10:0.10
11:0.11
12:0.12
13:0.13
14:0.14
15:0.15
16:0.16
17:0.17
18:0.18
19:0.19
20:0.20
21:0.21
22:0.22
23:0.23
24:0.24
25:0.25
[col]
1:0.01
2:0.02
3:0.03
4:0.04
5:0.05
6:0.06
7:0.07
8:0.08
9:0.09
10:0.0999999
11:0.1099999
12:0.1199999
13:0.1299999
14:0.1399999
15:0.1499999
16:0.1599999
17:0.1699999
18:0.1799999
19:0.1899999
20:0.1999999
21:0.21
22:0.22
23:0.23
24:0.24
25:0.2499999
[col]
1: .01
2: .02
3: .03
4: .04
5: .05
6: .06
7: .07
8: .08
9: .09
10: 9.999999E-02
11: .11
12: .12
13: .13
14: .14
15: .15
16: .16
17: .17
18: .18
19: .19
20: .2
21: .21
22: .22
23: .23
24: .2400001
25: .2500001
[col]
1:0.06
2:0.07
3:0.08
4:0.09
5:0.1
6:0.11
7:0.1199999
8:0.1299999
9:0.14
10:0.15
11:0.16
12:0.17
13:0.18
14:0.19
15:0.2
16:0.21
17:0.22
18:0.23
19:0.24
20:0.25
21:0.26
22:0.2699999
23:0.2799999
24:0.2899999
25:0.2999999
[/table]

[dkinzer]

Is this some feature of the underlying arithmetic operations which I have no control?

ZBasic uses the standard IEEE-754 format for floating point representation. One problem with this format is that not all decimal fractional values can be represented exactly. For example, the value 0.01 has the internal representation &H3C23D70A which is actually 9.9999998e-3. Iteratively adding this value to a sum results in an ever increasing cumulative error. An IEEE 754 Calculator is useful for analyzing such problems.

Compounding this problem, conversion back to a string may also introduce an error depending on the value. Generally speaking, when working with floating point numbers you should be using the Fmt() to produce a result with a precision that is appropriate for the value in question.

Other issues arise when attempting to compare floating point numbers. The article Comparing Floating Point Numbers contains useful information on this topic.

For an in-depth discussion of floating point accuracy, error sources, etc., see What Every Computer Scientist Should Know About Floating-Point Arithmetic.

In cases where you need more accuracy than can be relied upon with IEEE-754, you may want to use fixed-point arithmetic. In addition to possibly higher accuracy, fixed-point is also usually faster.