You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2020/07/01 14:16:07 UTC
Decimal128 scale limits
Hello,
Are there limits to the value of the scale for either decimal128 or
decimal? Can it be negative? Can it be greater than 38 (and/or lower
than -38)?
It's not clear from looking either at the spec or at the C++ code...
Regards
Antoine.
Re: Decimal128 scale limits
Posted by Micah Kornfield <em...@gmail.com>.
With regards to scale, my colleague discovered some inconsistencies and filed a JIRA with a proposed fix (a PR should be attached shortly).
I think this is an edge case that should be fixed but if someone with more historical context has opinions, I'd like to here them.
[1] https://issues.apache.org/jira/browse/ARROW-9913
On 2020/07/02 14:51:34, Wes McKinney <we...@gmail.com> wrote:
> I think the intention so far has been to support precision between 0
> and 38 and scale <= precision. 128-bit integers max out at 38 digits,
> I think that's the rationale for the limit. See e.g. the Impala docs
> (also uses 128-bit decimals) [1]
>
> [1]: https://impala.apache.org/docs/build/html/topics/impala_decimal.html
>
> On Wed, Jul 1, 2020 at 10:16 AM Kazuaki Ishizaki <IS...@jp.ibm.com> wrote:
> >
> > Hi,
> > According to https://arrow.apache.org/docs/cpp/api/utilities.html,
> > Decimal128 comes from the Apache ORC C++ implementation.
> >
> > When I see the Hive document at
> > https://hive.apache.org/javadocs/r1.2.2/api/index.html?org/apache/hadoop/hive/common/type/Decimal128.html
> > , there is the following statement. Does it help you?
> > > A 128-bit fixed-length Decimal value in the ANSI SQL Numeric semantics,
> > representing unscaledValue / 10**scale where scale is 0 or positive.
> >
> > Regards,
> > Kazuaki Ishizaki
> >
> >
> >
> > From: Jacek Pliszka <ja...@gmail.com>
> > To: dev@arrow.apache.org
> > Date: 2020/07/02 00:08
> > Subject: [EXTERNAL] Re: Decimal128 scale limits
> >
> >
> >
> > Hi!
> >
> > I am aware about at least 2 different decimal128 things:
> >
> > a) the one we have - where we use 128 bits to store integer which is
> > later shifted by scale - 38 is number of digits of significand i.e.
> > digits fitting in 128 bits
> > (2**128/10**38) - IMHO it is completely unrelated to scale which we
> > store separately
> >
> > b) IEEE 754 one which has exponent from -6143 to +6144
> >
> > BR,
> >
> > Jacek
> >
> > śr., 1 lip 2020 o 16:16 Antoine Pitrou <an...@python.org> napisał(a):
> > >
> > >
> > > Hello,
> > >
> > > Are there limits to the value of the scale for either decimal128 or
> > > decimal? Can it be negative? Can it be greater than 38 (and/or lower
> > > than -38)?
> > >
> > > It's not clear from looking either at the spec or at the C++ code...
> > >
> > > Regards
> > >
> > > Antoine.
> >
> >
> >
> >
>
Re: Decimal128 scale limits
Posted by Wes McKinney <we...@gmail.com>.
I think the intention so far has been to support precision between 0
and 38 and scale <= precision. 128-bit integers max out at 38 digits,
I think that's the rationale for the limit. See e.g. the Impala docs
(also uses 128-bit decimals) [1]
[1]: https://impala.apache.org/docs/build/html/topics/impala_decimal.html
On Wed, Jul 1, 2020 at 10:16 AM Kazuaki Ishizaki <IS...@jp.ibm.com> wrote:
>
> Hi,
> According to https://arrow.apache.org/docs/cpp/api/utilities.html,
> Decimal128 comes from the Apache ORC C++ implementation.
>
> When I see the Hive document at
> https://hive.apache.org/javadocs/r1.2.2/api/index.html?org/apache/hadoop/hive/common/type/Decimal128.html
> , there is the following statement. Does it help you?
> > A 128-bit fixed-length Decimal value in the ANSI SQL Numeric semantics,
> representing unscaledValue / 10**scale where scale is 0 or positive.
>
> Regards,
> Kazuaki Ishizaki
>
>
>
> From: Jacek Pliszka <ja...@gmail.com>
> To: dev@arrow.apache.org
> Date: 2020/07/02 00:08
> Subject: [EXTERNAL] Re: Decimal128 scale limits
>
>
>
> Hi!
>
> I am aware about at least 2 different decimal128 things:
>
> a) the one we have - where we use 128 bits to store integer which is
> later shifted by scale - 38 is number of digits of significand i.e.
> digits fitting in 128 bits
> (2**128/10**38) - IMHO it is completely unrelated to scale which we
> store separately
>
> b) IEEE 754 one which has exponent from -6143 to +6144
>
> BR,
>
> Jacek
>
> śr., 1 lip 2020 o 16:16 Antoine Pitrou <an...@python.org> napisał(a):
> >
> >
> > Hello,
> >
> > Are there limits to the value of the scale for either decimal128 or
> > decimal? Can it be negative? Can it be greater than 38 (and/or lower
> > than -38)?
> >
> > It's not clear from looking either at the spec or at the C++ code...
> >
> > Regards
> >
> > Antoine.
>
>
>
>
RE: Decimal128 scale limits
Posted by Kazuaki Ishizaki <IS...@jp.ibm.com>.
Hi,
According to https://arrow.apache.org/docs/cpp/api/utilities.html,
Decimal128 comes from the Apache ORC C++ implementation.
When I see the Hive document at
https://hive.apache.org/javadocs/r1.2.2/api/index.html?org/apache/hadoop/hive/common/type/Decimal128.html
, there is the following statement. Does it help you?
> A 128-bit fixed-length Decimal value in the ANSI SQL Numeric semantics,
representing unscaledValue / 10**scale where scale is 0 or positive.
Regards,
Kazuaki Ishizaki
From: Jacek Pliszka <ja...@gmail.com>
To: dev@arrow.apache.org
Date: 2020/07/02 00:08
Subject: [EXTERNAL] Re: Decimal128 scale limits
Hi!
I am aware about at least 2 different decimal128 things:
a) the one we have - where we use 128 bits to store integer which is
later shifted by scale - 38 is number of digits of significand i.e.
digits fitting in 128 bits
(2**128/10**38) - IMHO it is completely unrelated to scale which we
store separately
b) IEEE 754 one which has exponent from -6143 to +6144
BR,
Jacek
śr., 1 lip 2020 o 16:16 Antoine Pitrou <an...@python.org> napisał(a):
>
>
> Hello,
>
> Are there limits to the value of the scale for either decimal128 or
> decimal? Can it be negative? Can it be greater than 38 (and/or lower
> than -38)?
>
> It's not clear from looking either at the spec or at the C++ code...
>
> Regards
>
> Antoine.
Re: Decimal128 scale limits
Posted by Jacek Pliszka <ja...@gmail.com>.
Hi!
I am aware about at least 2 different decimal128 things:
a) the one we have - where we use 128 bits to store integer which is
later shifted by scale - 38 is number of digits of significand i.e.
digits fitting in 128 bits
(2**128/10**38) - IMHO it is completely unrelated to scale which we
store separately
b) IEEE 754 one which has exponent from -6143 to +6144
BR,
Jacek
śr., 1 lip 2020 o 16:16 Antoine Pitrou <an...@python.org> napisał(a):
>
>
> Hello,
>
> Are there limits to the value of the scale for either decimal128 or
> decimal? Can it be negative? Can it be greater than 38 (and/or lower
> than -38)?
>
> It's not clear from looking either at the spec or at the C++ code...
>
> Regards
>
> Antoine.