PRB: Format Function May Return Incorrect Values
ID: Q174155
|
The information in this article applies to:
-
Microsoft Visual Basic Control Creation, Learning, Professional, and Enterprise Editions for Windows, version 5.0
-
Microsoft Visual Basic Standard, Professional, and Enterprise Editions, 16-bit and 32-bit, for Windows, version 4.0
SYMPTOMS
Decimal values rounded by the Format function may be incorrect. For a
specific decimal number, the rounded value returned by the Format function
may differ for different data types and between different versions of
Microsoft Visual Basic.
CAUSE
When a variable is passed into the Format function, an algorithm is used to
determine the most appropriate numeric data-type to convert the value to,
and how to format it. This algorithm may not evaluate certain decimal
values as expected.
RESOLUTION
To avoid these unexpected conversions, explicitly convert the value to the
desired data type before using the Format function. For instance, use the
CCur function to convert the decimal value to a currency data-type that
maintains more accuracy in rounding.
STATUS
Microsoft has confirmed this to be a problem in the Microsoft products
listed at the beginning of this article. This problem does not occur in
Visual Basic 6.0.
MORE INFORMATION
Decimal values stored in single and double data-types are binary
representations of the number. Certain decimal values cannot be exactly
represented in binary and are approximately represented. When evaluated by
the Format function, these values may not be rounded as expected.
There are two methods recommended to minimize these errors:
- Use the Currency data type rather than singles or doubles. Currency
values are actually stored as integers with the decimal point scaled
four places.
- Convert the values to Currency before evaluation by the Format function.
Steps to Reproduce Behavior
- In Microsoft Visual Basic, create a new Standard EXE project. Form1 is
created by default.
- On Form1, add a textbox (Text1) and three labels (Label1, Label2, and
Label3).
- Set the Text property of Text1 to a zero and the Caption property of
each of the labels to nothing.
- Add the following code to the Change event of the textbox:
Dim dblValue As Double
Dim sngValue As Single
Dim curValue As Currency
dblValue = Text1.Text
sngValue = Text1.Text
curValue = Text1.Text
Label1.Caption = Format(dblValue, "#.000")
Label2.Caption = Format(sngValue, "#.000")
Label3.Caption = Format(curValue, "#.000")
- Run the project, and enter some decimal values in the textbox, such as:
0.9125
0.7125
0.0915
Note that the values displayed in the labels are different. The Format
function rounds some values up and some down based upon the data-type.
For instance, 0.9125 may be displayed as .912 and .913.
To correct this problem and to keep all values consistent, do one of the
following:
- Use the CCur() function to convert the values to a Currency data type
before using the Format() function: (This should not be used if the values will have more than 13 digits.)
TextDblCur = Format(CCur(dblValue), "#.000")
TextSngCur = Format(CCur(sngValue), "#.000")
- Declare the variables as Currency.
REFERENCES
For additional information, please see the following articles in the
Microsoft Knowledge Base:
(Complete) Tutorial to Understand IEEE Floating-Point Errors
HOWTO: Work Around Floating-Point Accuracy/Comparison Problems
Additional query words:
Precision
Keywords : kbVBp400 kbVBp500 kbVBp600fix PrgOther VBKBProgramming
Version : WINDOWS:4.0,5.0
Platform : WINDOWS
Issue type : kbbug