HOWTO: Work Around Floating-Point Accuracy/Comparison Problems

The information in this article applies to:

• The Standard and Professional Editions of Microsoft Visual Basic for MS-DOS, version 1.0
• Microsoft QuickBasic for MS-DOS, versions 1.0, 1.01, 2.0, 2.01, 3.0, 4.0, 4.0b, and 4.5
• Microsoft Basic Compiler for MS-DOS, versions 6.0 and 6.0b
• Microsoft Basic Professional Development System (PDS) for MS-DOS, versions 7.0 and 7.1
• GW-Basic Interpreter for MS-DOS, versions 3.2, 3.22 and 3.23

SUMMARY

To reliably test whether two floating-point variables or expressions are equal (using IEEE format or MBF), you must subtract the two variables being compared and test whether their difference is less than a value chosen at the limits of significance for single or double precision. NO OTHER TEST FOR EQUALITY WILL BE RELIABLE. The following formulas reliably test whether X and Y are equal:

1. For single precision, you must test whether the difference of X and Y is less than the value 7 significant digits smaller than X or Y. Divide X or Y by 10^7 to find the comparison value. For example:

         ' To try this example in VBDOS.EXE:
         ' 1. From the File menu, choose New Project.
         ' 2. Copy the code example to the Code window.
         ' 3. Press F5 to run the program.
   

         IF ABS(X! - Y!) <= (X! / 10^7) THEN
            PRINT "Equal within 7 digits"
         ENDIF
   
   

2. For double precision, you must test whether the difference of X and Y is less than the value 15 significant digits smaller than X or Y. Divide X or Y by 10^15 to find the comparison value. For example:

         ' To try this example in VBDOS.EXE:
         ' 1. From the File menu, choose New Project.
         ' 2. Copy the code example to the Code window.
         ' 3. Press F5 to run the program.
   

         IF ABS(X# - Y#) <= (X# / 10^15) THEN
            PRINT "Equal within 15 digits"
         ENDIF
   
   

MBF (Microsoft Binary Format) is found in Microsoft QuickBasic for MS-DOS (QB.EXE non-coprocessor version only), versions 1.0, 1.01, 2.0, 2.01, and 3.0, and in Microsoft GW-Basic Interpreter for MS-DOS, versions 3.2, 3.22, and 3.23.

The information in this article is also included in the Help file provided with the Standard and Professional Editions of Microsoft Visual Basic for MS-DOS, version 1.0.

MORE INFORMATION

   NOTE: Significant digits in a calculated number can be lost due to
   the following: multiple calculations, especially addition of
   numbers far apart in value, or subtraction of numbers similar in
   value. When a number results from multiple calculations, you may
   need to change your test for equality to use fewer significant
   digits to reflect the mathematical loss of significant digits. If
   your test of significance uses too many significant digits, you may
   fail to discover that numbers compared for equality are actually
   equal within the possible limit of accuracy.

   X = 25
   Y = 60.1
   IF 1502.5 = (X * Y) THEN PRINT "equal"

   Z = 25 * 60.1
   IF 1502.5 = Z THEN PRINT "equal"

   PRINT 69.82! + 1    ' Single precision, prints 70.82.
   PRINT 69.82# + 1    ' Double precision, prints 70.81999999999999.

Reference:

Both the IEEE and MBF standards attempt to balance accuracy and precision with numeric range and speed. Accuracy measures how many significant bits of precision are not lost in calculations. Precision refers to the number of bits in the mantissa, which determines how many decimal digits can be represented.

Both IEEE format and MBF store numbers of the form 1.x to the power of y (where x and y are base 2 numbers; x is the mantissa, and y is the exponent).

MBF single precision has 24 bits of mantissa, and double precision has 56 bits of mantissa. All MBF calculations are performed within just 24 or 56 bits.

IEEE single precision has 24 bits of mantissa, and double precision has 53 bits of mantissa. However, all single-precision and double-precision IEEE calculations in Visual Basic for MS-DOS, version 1.0; in QuickBasic for MS-DOS, versions 3.0/4.x; in Basic Compiler for MS-DOS, versions 6.0, and 6.0b; and in Basic PDS for MS-DOS, versions 7.0 and 7.1re performed in a 64-bit temporary register for greater accuracy. As a result, IEEE calculations are more accurate than MBF calculations, despite MBF's ability to represent more bits in double precision.

Most numbers in decimal (base 10) notation do NOT have an exact representation in the binary (base 2) floating-point storage format used in single-precision and double-precision data types. Both IEEE format and MBF cannot exactly represent (and must round off) all numbers that are not of the form 1.x to the power of y (where x and y are base 2 numbers). The numbers that can be exactly represented are spread out over a very wide range. A high density of representable numbers is near 1.0 and -1.0, but fewer and fewer representable numbers occur as the numbers go towards 0 or infinity.

The above limitations often cause Basic to return floating-point results different than you might expect. More information on this topic can be found in the Microsoft Knowledge Base by querying on the following words:

   floating and point and format and QuickBasic

1. Pages 16-21 of "Microsoft QuickBasic 4.0: Basic Language Reference" for QuickBasic for MS-DOS, versions 4.0 and 4.0b. Note that the 4.0 manuals need a correction on page 19 to the MHex$ code example, as explained in a separate article found in the Microsoft Knowledge Base by querying on the following words:

         MHex$ AND 19
   

2. Pages 12-17 of "Microsoft QuickBasic 4.5: Basic Language Reference" for QuickBasic for MS-DOS, version 4.5. Note that this optional manual must be ordered separately using an order card provided with version 4.5.

3. Pages 702-705 of the "Microsoft Basic 7.0: Programmer's Guide" for Microsoft Basic PDS for MS-DOS, versions 7.0 and 7.1.

   MBF AND conversion AND exponent