20060301
4 Number of Digit
s
and Preci
s
ion
I
Number of Digit
s
Standard Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic
Format dialog box is not selected.
• Up to 611 digits are stored in memory for integer values.
• Decimal values up to 15 digits are converted to fraction format and saved in memory. When
a mathematical expression cannot be converted to fraction format, the result is displayed in
decimal format.
• Values stored in memory are displayed as-is, regardless of how [Number Format] settings
(Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured (except when a decimal value is
displayed).
Decimal Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic
Format dialog box is selected.
• Values stored in Ans memory and values assigned to variables have the same number of
digits as defined for Standard mode values.
• Values are displayed in accordance with how [Number Format] settings (Normal 1, Normal 2,
Fix 0 – 9, Sci 0 – 9) are configured.
• Displayed values are rounded to the appropriate number of decimal places.
• Some applications store values using a mantissa up to 15 digits long and a 3-digit
exponent.
I
Preci
s
ion
• Internal calculations are performed using 15 digits.
• The error for a single mathematical expression (Decimal mode calculation error) is
p
1 at the
10th digit. In the case of exponential display, calculation error is
p
1 at the least significant
digit. Note that performing consecutive calculations causes error to be cumulative. Error
is also cumulative for internal consecutive calculations performed for: ^(
x
y
),
x
,
x
!,
n
P
r
,
n
C
r
, etc.
• Error is cumulative and tends to be larger in the vicinity of a function’s singular point(s) and
inflection point(s), and the vicinity of zero. With sinh(
x
) and tanh(
x
), for example, the
inflection point occurs when
x
= 0. In this vicinity, error is cumulative and precision is poor.
A
-4-1
Number of Digits and Precision
Summary of Contents for ClassPad 330
Page 11: ...20060301 20090601 ClassPad 330 ClassPad OS Version 3 04 ...
Page 277: ...20060301 3 3 10 Storing Functions 3 Tap AND Plot OR Plot ...
Page 779: ...20090601 S Graph Line Clustered C S Graph Line Stacked E 13 9 2 Graphing ...
Page 780: ...20090601 S Graph Line 100 Stacked F S Graph Column Clustered G 13 9 3 Graphing ...
Page 781: ...20090601 S Graph Column Stacked I S Graph Column 100 Stacked J 13 9 4 Graphing ...
Page 782: ...20090601 S Graph Bar Clustered K S Graph Bar Stacked 9 13 9 5 Graphing ...
