20021201
4 Number of Digits and Precision
k
Number of Digits
Standard Mode
The following applies when the check box next to the “Decimal Calculation” item on the
Basic Format dialog box is not selected.
• Up to 611 digits are stored in memory for integer values.
• Decimal values up to 15 digits are converted to fraction format and saved in memory. When
a mathematical expression cannot be converted to fraction format, the result is displayed in
decimal format.
• Values stored in memory are displayed as-is, regardless of how [Display] settings
(Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured (except when a decimal value is
displayed).
Decimal Mode
The following applies when the check box next to the “Decimal Calculation” item on the
Basic Format dialog box is selected.
• Values stored in Ans memory have the same number of digits as they have when
displayed. A value that is assigned to a variable has the same number of digits as defined
for Standard mode values.
• Values are displayed in accordance with how [Display] settings (Normal 1, Normal 2,
Fix 0 – 9, Sci 0 – 9) are configured.
• Displayed values are rounded to the appropriate number of decimal places.
• Some applications store values using a mantissa up to 15 digits long and a 3-digit
exponent.
k
Precision
• Internal calculations are performed using 15 digits.
• The error for a single mathematical expression (Decimal mode calculation error) is
±
1 at
the 10th digit. In the case of exponential display, calculation error is
±
1 at the least
significant digit. Note that performing consecutive calculations causes error to be
cumulative. Error is also cumulative for internal consecutive calculations performed for:
^(
x
y
),
x
,
x
!,
n
P
r
,
n
C
r
, etc.
• Error is cumulative and tends to be larger in the vicinity of a function’s singular point(s) and
inflection point(s), and the vicinity of zero. With sinh(
x
) and tanh(
x
), for example, the
inflection point occurs when
x
= 0. In this vicinity, error is cumulative and precision is poor.
α
-4-1
Number of Digits and Precision
