How to Avoid Rounding & Overflow Probs on Pentium Processors

Last reviewed: June 21, 1995
Article ID: Q126455

The information in this article applies to:

- Standard and Professional Editions of Microsoft Visual Basic for

  Windows, versions 2.0 and 3.0

SUMMARY

Intel Corporation has identified two minor bugs in the Pentium processor relating to the conversion of floating point values to integer numbers when using the Floating Point Integer Store (FIST) instruction. These bugs involve the processing of exceptions for a narrow range of bounds for floating point numbers and involve unexpected behavior in two rounding modes for six specific operands. Programmers can easily work around the first bug by range checking prior to converting the values. Programmers can avoid the second bug by not overriding the default rounding modes.

MORE INFORMATION

Different programming languages handle conversions from floating point to integer in different ways. C, C++, and FORTRAN programs are most likely safe; that is, they are not likely to be affected without explicit modification of rounding modes by the programmer. Basic programs are more likely to be affected by the bugs.

Application programs in general are not affected. Applications should include appropriate error checking and take appropriate actions to recover from exceptions. To do this, applications need to do explicit range checking prior to conversion or implement explicit exception handling. If an application relies on explicit exception handling, the possibility of encountering the bug exists. Applications written in languages that use exception handling will most likely use range checking, thereby allowing the application to be compiled with a no-check option for performance reasons.

FIST Overview

The FIST instruction is used to convert floating point numbers to signed integers. For example, in a C or FORTRAN program, when an integer variable is assigned a floating point value, that value must be truncated to an integer before being stored. Other conversions from floating point to integer are also possible; for example, Basic and Pascal round instead of truncating the fractional part.

The FIST instruction operates in one of four rounding modes: chop (also known as truncate), nearest (nearest or even), up, and down. The default type conversions done by C and FORTRAN use the chop rounding mode, whereas Basic uses nearest. The processor indicates whether the input operand is an integer by means of a precision exception (PE) flag. The PE flag is set to true if the number to be converted was not already an integer; that is, it had a fractional part. It is unchanged otherwise. The effect of rounding is shown in the rounding direction( C1) flag. If an input value is rounded to a number with a greater magnitude, the C1 flag is set; otherwise, it is cleared.

FIST converts floating point numbers to 16, 32, or 64-bit signed integers. Because the range of a floating point number is larger than any of these formats, some floating point numbers cannot be converted to integers. For example, the largest 16-bit integer is 32767, so an attempt to convert 32768.5 to a 16-bit integer results in an exception (in this case, the invalid operation exception (IE)). The processing of an exception is controlled by the exception mask in the floating point control register. If an invalid operation exception mask is set, the exception is masked, otherwise, it is unmasked.

For example, applications running under Microsoft Windows version 3.1 begin with all floating point exceptions masked. Different applications may change the exception masks and provide their own methods for handling exceptions. For example Microsoft Visual C++ sets the invalid operation exception to unmasked, while leaving the precision exception masked. An application developed in C++ will see a pop-up window appear when a FIST instruction attempts to convert a number that is out of bounds, unless the exception handling is changed.

Undetected Overflow

A bug has been identified by Intel in FIST instructions that convert a floating point number to either a 16- or 32-bit signed integer. The FISTP instruction that stores to a 64-bit integer is not affected. The error occurs when a rounding mode is set to "nearest" or "up" for one limited range of floating point values in the out of bounds region. For this range, regardless of the exception mask, the value 0 is written to memory, the PE flag is set, the IE flag is not set, and no IE exception is raised.

In the figures below, floating point values from A and B can be converted to integers. Values outside this range should raise an exception. This bug affects values in the C and D range only.

16-Bit FIST

                                                 65535.5 -
          -32768.5     0      32767.5            65536.0

<-------------[--------|--------]-------------------|-------->

   overflow       Normal Range        overflow
              A                 B                   C

32-Bit FIST

                                                          4,294,967,295.5 -
      -2,147,483,648.5     0      2,147,483,647.5         4,294,967,296.0

<-------------[------------|------------]-----------------------|-------->

   overflow           Normal Range             overflow
              A                         B                       D

For example, take a number outside the range of a 32-bit signed integer, such as 4,294,967,295.7. This is rounded mathematically to 4,294,967,296. Due to the bug, the Pentium processor does not raise an exception to this number. It simply writes a 0 into memory and does not transfer control to a handler or raise the IE flag.

Applications that Use Microsoft Visual Basic

Microsoft Visual Basic automatically promotes data types to handle mixed type operations. This avoids the overflow condition for many cases. For example:

   IntX = SngY*IntZ
   IntX = 65535.7*2

However, implicit conversions may be affected. The following Visual Basic version 3.0 code produces different results depending on the processor:

   Dim X As Integer
   Dim Y As Single
   Y = 65535.5
   X = Y
   MsgBox Cstr(X)

The above code should result in a run-time error 6 "Overflow" message. On a Pentium processor-based machine, this code results in a return of zero (0). A simple way to avoid this error is to include a range check in your code. For example:

   If (x)>=65535.5
   Then Error 6

The only programs affected would be those that explicitly trap error 6 (overflow) and involve either explicit assignment of a floating-point expression to an Integer data type or a Long data type or explicit conversion of a floating point expression using CInt() or CLng().

Programs that employ explicit bounds checking to prevent overflow conditions should not be affected.

If bounds checking adversely affects performance in a loop, migrating the code to a C language dynamic-link library (DLL) may be a better option.

Program Results

At the assembly level, the recommended workaround is the insertion of the FRNDINT (Floating Point Round to Integer) instruction immediately preceding the FIST instruction. FRNDINT will correctly round the floating point value before executing the FIST instruction (as opposed to FISTP); the correction will also need to preserve its input. Future versions of Visual Basic will include this fix.

NOTE: Any method that forces the processor to emulate floating point instructions will avoid the problem, at the cost of reduced performance and speed.

Rounding Mode Errors

For six specific operands, the results of the FIST instructions are not as expected. There are flag differences in all four rounding modes. There is also the possibility of an incorrect number being stored in the "up" and "down" rounding modes. In the "nearest" and "chop" rounding modes, which are most frequently used, the value stored to memory is correct. This bug affects the 16-, 32-, and 64-bit variants of the instruction.

The following table shows the affected numbers, rounding modes, and expected and actual values:

   Operand     | Rounding | Exp. | Actual | Exp. | Actual | Exp. | Actual
(any one of)   |  mode    | mode |  mode  |  PE  |   PE   |  C1  |   C1

1/16 (0.0625)  |  nearest |  0   |   0    |   1  |  unch. |  0   |   0
1/8 (0.125)    |   chop   |  0   |   0    |   1  |  unch. |  0   |   0
3/16 (0.1875)  |   down   |  0   |   0    |   1  |  unch. |  0   |   0
               |    up    |  1   |   0    |   1  |  unch. |  1   |   0
-1/16 (-0.0625)|  nearest |  0   |   0    |   1  |  unch. |  0   |   0
-1/8 (-0.125)  |   chop   |  0   |   0    |   1  |  unch. |  0   |   0
-3/16 (-0.1875)|   down   | -1   |   0    |   1  |  unch. |  1   |   0
               |    up    |  0   |   0    |   1  |  unch. |  0   |   0

For six numbers (1/16, 1/8, 3/16, -1/16, -1/8, and -3/16) incorrect results can occur. In all rounding modes, the PE flag is not set consistently with the rounding that occurred; it remains unchanged. For the three positive numbers in the "up" rounding mode, FIST stores an incorrect value (0 instead of 1); similarly, for the three negative numbers in the "down" rounding mode, FIST gives an incorrect result (0 instead of -1). In each of these cases, the C1 bit is also not set correctly. However, in the "nearest" and "chop" rounding modes, the value stored to memory is always correct, as is the C1 bit.

Program Results

Incorrect results from the second bug are only returned when the rounding mode is set to "up" or "down." Therefore, it should not affect an application unless the application explicitly changes the rounding mode from the language default. If an application changes to "up" or "down" rounding mode, it can no longer rely on language-provided floating-point- value-to-integer-number conversion facilities and must provide its own by, for example, using FRNDINT.

The incorrect status information in the PE flag, affecting all rounding modes, should be insignificant in most applications. If rounding occurs, the PE flag is already set. The PE exception is also typically masked. The C1 bit is only used by an exception handler, so the incorrect values are insignificant.

Workaround

At the assembly level the recommended workaround for the second FIST bug is the same for the first; inserting the FRNDINT instruction immediately preceding the FIST instruction.

Application programmers can avoid rounding errors in the second bug by not overriding the default rounding modes.

NOTE: Any method that forces the processor to emulate floating point instructions will avoid both problems, at the cost of reduced performance and speed.

Additional reference words: 3.00
KBCategory: kbhw kbprg kb3rdparty
KBSubcategory: RefsThird

THE INFORMATION PROVIDED IN THE MICROSOFT KNOWLEDGE BASE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. MICROSOFT DISCLAIMS ALL WARRANTIES, EITHER EXPRESS OR IMPLIED, INCLUDING THE WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL MICROSOFT CORPORATION OR ITS SUPPLIERS BE LIABLE FOR ANY DAMAGES WHATSOEVER INCLUDING DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL, LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF MICROSOFT CORPORATION OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. SOME STATES DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES SO THE FOREGOING LIMITATION MAY NOT APPLY.

Last reviewed: June 21, 1995
© 1998 Microsoft Corporation. All rights reserved. Terms of Use.