HOWTO: Work Around Floating-Point Accuracy/Comparison Problems

ID: Q69333

The information in this article applies to:

Microsoft Visual Basic Standard and Professional Editions for MS-DOS, version 1.0
Microsoft QuickBASIC for MS-DOS, versions 1.0, 1.01, 2.0, 2.01, 3.0, 4.0, 4.0b, 4.5
Microsoft BASIC Compiler for MS-DOS, versions 6.0, 6.0b
Microsoft BASIC Professional Development System (PDS) for MS-DOS, versions 7.0, 7.1
Microsoft GW-BASIC, versions 3.2, 3.22, 3.23

SUMMARY

To reliably test whether two floating-point variables or expressions are equal (using IEEE format or MBF), you must subtract the two variables being compared and test whether their difference is less than a value chosen at the limits of significance for single or double precision. NO OTHER TEST FOR EQUALITY WILL BE RELIABLE. The following formulas reliably test whether X and Y are equal:

For single precision, you must test whether the difference of X and Y is less than the value 7 significant digits smaller than X or Y. Divide X or Y by 10^7 to find the comparison value. For example:


      ' To try this example in VBDOS.EXE:
      ' 1. From the File menu, choose New Project.
      ' 2. Copy the code example to the Code window.
      ' 3. Press F5 to run the program.

      IF ABS(X! - Y!) <= (X! / 10^7) THEN
         PRINT "Equal within 7 digits"
      ENDIF

For double precision, you must test whether the difference of X and Y is less than the value 15 significant digits smaller than X or Y. Divide X or Y by 10^15 to find the comparison value. For example:


      ' To try this example in VBDOS.EXE:
      ' 1. From the File menu, choose New Project.
      ' 2. Copy the code example to the Code window.
      ' 3. Press F5 to run the program.

      IF ABS(X# - Y#) <= (X# / 10^15) THEN
         PRINT "Equal within 15 digits"
      ENDIF

The IEEE floating-point format is found in the Standard and Professional Editions of Microsoft Visual Basic for MS-DOS, version 1.0; in Microsoft QuickBasic for MS-DOS (QB87.EXE coprocessor version only), versions 3.0, 4.0, 4.0b, and 4.5; in Microsoft Basic Compiler for MS-DOS and MS OS/2, versions 6.0 and 6.0b; and in Microsoft Basic Professional Development System (PDS) for MS-DOS and MS OS/2, versions 7.0 and 7.1.

MBF (Microsoft Binary Format) is found in Microsoft QuickBasic for MS-DOS (QB.EXE non-coprocessor version only), versions 1.0, 1.01, 2.0, 2.01, and 3.0, and in Microsoft GW-Basic Interpreter for MS-DOS, versions 3.2, 3.22, and 3.23.

The information in this article is also included in the Help file provided with the Standard and Professional Editions of Microsoft Visual Basic for MS-DOS, version 1.0.

MORE INFORMATION

NOTE: Significant digits in a calculated number can be lost due to the following: multiple calculations, especially addition of numbers far apart in value, or subtraction of numbers similar in value. When a number results from multiple calculations, you may need to change your test for equality to use fewer significant digits to reflect the mathematical loss of significant digits. If your test of significance uses too many significant digits, you may fail to discover that numbers compared for equality are actually equal within the possible limit of accuracy.

In the Basic products listed above that use IEEE floating-point format, intermediate calculations are performed in an internal 64-bit temporary register, which has more bits of accuracy than are stored in single-precision or double-precision variables. This often results in an IF statement returning an error which states that the intermediate calculation is not equal to the expression being compared. For example:


   X = 25
   Y = 60.1
   IF 1502.5 = (X * Y) THEN PRINT "equal"

Running the above code will NOT print "equal". In contrast, the following method using a placeholder variable will print "equal", but is still NOT a reliable technique as a test for equality:


   Z = 25 * 60.1
   IF 1502.5 = Z THEN PRINT "equal"

Note that explicit numeric type casts (! for single precision, # for double precision) will affect the precision in which calculations are stored and printed. Whichever type casting you perform, you may still see unexpected rounding results:


   PRINT 69.82! + 1    ' Single precision, prints 70.82.
   PRINT 69.82# + 1    ' Double precision, prints 70.81999999999999.

For an exact decimal (base 10) numeric representation, such as for calculations of dollars and cents, you should use the CURRENCY (@) data type found in Visual Basic for MS-DOS, version 1.0 and Basic PDS for MS-DOS, versions 7.0 and 7.1. The CURRENCY data type exactly stores up to 19 digits, with 4 digits after the decimal place.

Reference:

Both the IEEE and MBF standards attempt to balance accuracy and precision with numeric range and speed. Accuracy measures how many significant bits of precision are not lost in calculations. Precision refers to the number of bits in the mantissa, which determines how many decimal digits can be represented.

Both IEEE format and MBF store numbers of the form 1.x to the power of y (where x and y are base 2 numbers; x is the mantissa, and y is the exponent).

MBF single precision has 24 bits of mantissa, and double precision has 56 bits of mantissa. All MBF calculations are performed within just 24 or 56 bits.

IEEE single precision has 24 bits of mantissa, and double precision has 53 bits of mantissa. However, all single-precision and double-precision IEEE calculations in Visual Basic for MS-DOS, version 1.0; in QuickBasic for MS-DOS, versions 3.0/4.x; in Basic Compiler for MS-DOS, versions 6.0, and 6.0b; and in Basic PDS for MS-DOS, versions 7.0 and 7.1re performed in a 64-bit temporary register for greater accuracy. As a result, IEEE calculations are more accurate than MBF calculations, despite MBF's ability to represent more bits in double precision.

Most numbers in decimal (base 10) notation do NOT have an exact representation in the binary (base 2) floating-point storage format used in single-precision and double-precision data types. Both IEEE format and MBF cannot exactly represent (and must round off) all numbers that are not of the form 1.x to the power of y (where x and y are base 2 numbers). The numbers that can be exactly represented are spread out over a very wide range. A high density of representable numbers is near 1.0 and -1.0, but fewer and fewer representable numbers occur as the numbers go towards 0 or infinity.

The above limitations often cause Basic to return floating-point results different than you might expect. More information on this topic can be found in the Microsoft Knowledge Base by querying on the following words:

floating and point and format and QuickBasic

The IEEE floating point format is documented in the following manuals:

Pages 16-21 of "Microsoft QuickBasic 4.0: Basic Language Reference" for QuickBasic for MS-DOS, versions 4.0 and 4.0b. Note that the 4.0 manuals need a correction on page 19 to the MHex$ code example, as explained in a separate article found in the Microsoft Knowledge Base by querying on the following words:
MHex$ AND 19

Pages 12-17 of "Microsoft QuickBasic 4.5: Basic Language Reference" for QuickBasic for MS-DOS, version 4.5. Note that this optional manual must be ordered separately using an order card provided with version 4.5.

Pages 702-705 of the "Microsoft Basic 7.0: Programmer's Guide" for Microsoft Basic PDS for MS-DOS, versions 7.0 and 7.1.

The MBF floating point format is documented in a separate article which can be found in the Microsoft Knowledge Base by querying on the following words:

MBF AND conversion AND exponent

Additional query words: VBmsdos QuickBas BasicCom 1.00 1.01 2.00 2.01 3.00 3.20 3.22 3.23 4.00 4.00b 4.50 6.00 6.00b 7.00 7.10

Keywords :
Version : MS-DOS:1.0,1.01,2.0,2.01,3.0,4.0,4.0b,4.5; :3.2,3.22,3.23,6.0,6.0b,7.0,7.1
Platform : MS-DOS
Issue type :