You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Dima Fomin <da...@yandex.ru> on 2019/10/24 12:47:10 UTC
ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
Hello!
First of all thank you very much for your efforts on Apache Arrow! This looks
very useful and prominent tech to be able use it in self-BI area.
I faced such issues - get Subj Error
> ArrowNotImplementedError: No cast implemented from decimal(2,2) to
decimal(12,2)
when trying to _pyarrow.Table.from_batches_ array of values defined in schema
as _pyarrow.decimal128(12,2)_.
Which values are in fact column of
_sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"_ values,
which I map in schema like this _pyarrow.decimal128(12,2)_.
What could be a reason of issue?
Could it be that somehow declared decimal type of data is reduced to minimum
value which met in data? :/
With best regards,
Dima Fomin
Re: ArrowNotImplementedError: No cast implemented from decimal(2,2)
to decimal(12,2)
Posted by Wes McKinney <we...@gmail.com>.
Apologies, I checked colab and it's 0.15.0 there, I confirm that
In [8]: pa.table([data2], schema=sss1)
does not work due to the missing cast implementation. It would be
great if someone would contribute an implementation for ARROW-6990
On Fri, Oct 25, 2019 at 9:42 AM Wes McKinney <we...@gmail.com> wrote:
>
> These code examples both work for me in 0.15.0 and master. What
> version of the project are you using?
>
> In [6]: pa.array(data1).type
> Out[6]: Decimal128Type(decimal(14, 2))
>
> In [7]: pa.table([data1], schema=sss1)
> Out[7]:
> pyarrow.Table
> decimal_col: decimal(22, 2)
>
> On Fri, Oct 25, 2019 at 8:46 AM Дима Фомин <da...@yandex.ru> wrote:
> >
> > Hello, Micah!
> >
> > Gotcha! Seems the problem in in pyarrow.array() handling of decimal values.
> >
> > data1 = [decimal.Decimal('234234234234.43')]
> >
> > #pa.array somehow broke precision and scale?
> > data2 = pa.array(data1)
> >
> > sss1 = pa.schema({'decimal_col': pa.decimal128(22,2)})
> >
> > pa.Table.from_arrays([data1], schema=sss1)
> >
> > is OK
> >
> > bur this one
> > pa.Table.from_arrays([data2], schema=sss1)
> >
> > raise ArrowNotImplementedError: No cast implemented from decimal(14, 2) to decimal(22, 2)
> >
> > see it here
> > https://colab.research.google.com/drive/1ejOyFBVs8GPFTosuRMDyTwPvt4145rgV
> >
> > Should I post in to JIRA as well? Or this is the same reason posted above?
> >
> >
> > 25.10.2019, 08:00, "Micah Kornfield" <em...@gmail.com>:
> >
> > Hi Dima,
> > Thanks for finding this bug would you mind filing a JIRA (https://issues.apache.org/jira).
> >
> > Note that this code works fine if you put quotes around the decimal value (e.g. Decimal('123.234')), but as you state it crashes python because the Decimal created without quotes requires more then 38 decimal places to represent.
> >
> > Thanks,
> > Micah
> >
> >
> >
> > On Thu, Oct 24, 2019 at 7:53 AM Дима Фомин <da...@yandex.ru> wrote:
> >
> > Digging this I just found that this code breaks my Python Kernel :(
> >
> > import sqlalchemy as sq
> > import pyarrow as pa
> >
> > aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234), sq.sql.sqltypes.decimal.Decimal(123.234)
> > ddd = [aaa, bbb]
> > pa.array(ddd)
> >
> > So seems pyarrow.array is limited with types of array elements, doesnt'it?
> >
> > WBR,
> > Dima Fomin
> >
> > 24.10.2019, 15:47, "Dima Fomin" <da...@yandex.ru>:
> >
> > Hello!
> > First of all thank you very much for your efforts on Apache Arrow! This looks very useful and prominent tech to be able use it in self-BI area.
> >
> > I faced such issues - get Subj Error
> >
> > ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
> >
> > when trying to pyarrow.Table.from_batches array of values defined in schema as pyarrow.decimal128(12,2).
> >
> > Which values are in fact column of sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)" values, which I map in schema like this pyarrow.decimal128(12,2).
> >
> > What could be a reason of issue?
> > Could it be that somehow declared decimal type of data is reduced to minimum value which met in data? :/
> >
> > With best regards,
> > Dima Fomin
Re: ArrowNotImplementedError: No cast implemented from decimal(2,2)
to decimal(12,2)
Posted by Wes McKinney <we...@gmail.com>.
These code examples both work for me in 0.15.0 and master. What
version of the project are you using?
In [6]: pa.array(data1).type
Out[6]: Decimal128Type(decimal(14, 2))
In [7]: pa.table([data1], schema=sss1)
Out[7]:
pyarrow.Table
decimal_col: decimal(22, 2)
On Fri, Oct 25, 2019 at 8:46 AM Дима Фомин <da...@yandex.ru> wrote:
>
> Hello, Micah!
>
> Gotcha! Seems the problem in in pyarrow.array() handling of decimal values.
>
> data1 = [decimal.Decimal('234234234234.43')]
>
> #pa.array somehow broke precision and scale?
> data2 = pa.array(data1)
>
> sss1 = pa.schema({'decimal_col': pa.decimal128(22,2)})
>
> pa.Table.from_arrays([data1], schema=sss1)
>
> is OK
>
> bur this one
> pa.Table.from_arrays([data2], schema=sss1)
>
> raise ArrowNotImplementedError: No cast implemented from decimal(14, 2) to decimal(22, 2)
>
> see it here
> https://colab.research.google.com/drive/1ejOyFBVs8GPFTosuRMDyTwPvt4145rgV
>
> Should I post in to JIRA as well? Or this is the same reason posted above?
>
>
> 25.10.2019, 08:00, "Micah Kornfield" <em...@gmail.com>:
>
> Hi Dima,
> Thanks for finding this bug would you mind filing a JIRA (https://issues.apache.org/jira).
>
> Note that this code works fine if you put quotes around the decimal value (e.g. Decimal('123.234')), but as you state it crashes python because the Decimal created without quotes requires more then 38 decimal places to represent.
>
> Thanks,
> Micah
>
>
>
> On Thu, Oct 24, 2019 at 7:53 AM Дима Фомин <da...@yandex.ru> wrote:
>
> Digging this I just found that this code breaks my Python Kernel :(
>
> import sqlalchemy as sq
> import pyarrow as pa
>
> aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234), sq.sql.sqltypes.decimal.Decimal(123.234)
> ddd = [aaa, bbb]
> pa.array(ddd)
>
> So seems pyarrow.array is limited with types of array elements, doesnt'it?
>
> WBR,
> Dima Fomin
>
> 24.10.2019, 15:47, "Dima Fomin" <da...@yandex.ru>:
>
> Hello!
> First of all thank you very much for your efforts on Apache Arrow! This looks very useful and prominent tech to be able use it in self-BI area.
>
> I faced such issues - get Subj Error
>
> ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
>
> when trying to pyarrow.Table.from_batches array of values defined in schema as pyarrow.decimal128(12,2).
>
> Which values are in fact column of sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)" values, which I map in schema like this pyarrow.decimal128(12,2).
>
> What could be a reason of issue?
> Could it be that somehow declared decimal type of data is reduced to minimum value which met in data? :/
>
> With best regards,
> Dima Fomin
Re: ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
Posted by Дима Фомин <da...@yandex.ru>.
Hello, Micah!
Gotcha! Seems the problem in in pyarrow.array() handling of decimal values.
data1 = [decimal.Decimal('234234234234.43')]
#pa.array somehow broke precision and scale?
data2 = pa.array(data1)
sss1 = pa.schema({'decimal_col': pa.decimal128(22,2)})
pa.Table.from_arrays([data1], schema=sss1)
is OK
bur this one
pa.Table.from_arrays([data2], schema=sss1)
raise ArrowNotImplementedError: No cast implemented from decimal(14, 2) to
decimal(22, 2)
see it here
https://colab.research.google.com/drive/1ejOyFBVs8GPFTosuRMDyTwPvt4145rgV
Should I post in to JIRA as well? Or this is the same reason posted above?
25.10.2019, 08:00, "Micah Kornfield" <em...@gmail.com>:
> Hi Dima,
>
> Thanks for finding this bug would you mind filing a JIRA
([https://issues.apache.org/jira](https://issues.apache.org/jira/secure/Dashboard.jspa)).
>
> Note that this code works fine if you put quotes around the decimal value
(e.g. Decimal('123.234')), but as you state it crashes python because the
Decimal created without quotes requires more then 38 decimal places to
represent.
>
> Thanks,
>
> Micah
>
> On Thu, Oct 24, 2019 at 7:53 AM Дима Фомин
<[datagy@yandex.ru](mailto:datagy@yandex.ru)> wrote:
>
>> Digging this I just found that this code breaks my Python Kernel :(
>>
>> import sqlalchemy as sq
>>
>> import pyarrow as pa
>>
>> aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234),
sq.sql.sqltypes.decimal.Decimal(123.234)
>>
>> ddd = [aaa, bbb]
>>
>> pa.array(ddd)
>>
>> So seems pyarrow.array is limited with types of array elements, doesnt'it?
>>
>> WBR,
>>
>> Dima Fomin
>>
>> 24.10.2019, 15:47, "Dima Fomin"
<[datagy@yandex.ru](mailto:datagy@yandex.ru)>:
>>
>>> Hello!
>>>
>>> First of all thank you very much for your efforts on Apache Arrow! This
looks very useful and prominent tech to be able use it in self-BI area.
>>>
>>> I faced such issues - get Subj Error
>>>
>>>> ArrowNotImplementedError: No cast implemented from decimal(2,2) to
decimal(12,2)
>>>
>>> when trying to _pyarrow.Table.from_batches_ array of values defined in
schema as _pyarrow.decimal128(12,2)_.
>>>
>>> Which values are in fact column of
_sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"_ values,
which I map in schema like this _pyarrow.decimal128(12,2)_.
>>>
>>> What could be a reason of issue?
>>>
>>> Could it be that somehow declared decimal type of data is reduced to
minimum value which met in data? :/
>>>
>>> With best regards,
>>>
>>> Dima Fomin
Re: ArrowNotImplementedError: No cast implemented from decimal(2,2)
to decimal(12,2)
Posted by Micah Kornfield <em...@gmail.com>.
Hi Dima,
Thanks for finding this bug would you mind filing a JIRA (
https://issues.apache.org/jira
<https://issues.apache.org/jira/secure/Dashboard.jspa>).
Note that this code works fine if you put quotes around the decimal value
(e.g. Decimal('123.234')), but as you state it crashes python because the
Decimal created without quotes requires more then 38 decimal places to
represent.
Thanks,
Micah
On Thu, Oct 24, 2019 at 7:53 AM Дима Фомин <da...@yandex.ru> wrote:
> Digging this I just found that this code breaks my Python Kernel :(
>
> import sqlalchemy as sq
> import pyarrow as pa
>
> aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234),
> sq.sql.sqltypes.decimal.Decimal(123.234)
> ddd = [aaa, bbb]
> pa.array(ddd)
>
> So seems pyarrow.array is limited with types of array elements, doesnt'it?
>
> WBR,
> Dima Fomin
>
> 24.10.2019, 15:47, "Dima Fomin" <da...@yandex.ru>:
>
> Hello!
> First of all thank you very much for your efforts on Apache Arrow! This
> looks very useful and prominent tech to be able use it in self-BI area.
>
> I faced such issues - get Subj Error
>
> ArrowNotImplementedError: No cast implemented from decimal(2,2) to
> decimal(12,2)
>
> when trying to *pyarrow.Table.from_batches* array of values defined in
> schema as *pyarrow.decimal128(12,2)*.
>
> Which values are in fact column of
> *sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"*
> values, which I map in schema like this *pyarrow.decimal128(12,2)*.
>
> What could be a reason of issue?
> Could it be that somehow declared decimal type of data is reduced to
> minimum value which met in data? :/
>
> With best regards,
> Dima Fomin
>
>
Re: ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
Posted by Дима Фомин <da...@yandex.ru>.
Digging this I just found that this code breaks my Python Kernel :(
import sqlalchemy as sq
import pyarrow as pa
aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234),
sq.sql.sqltypes.decimal.Decimal(123.234)
ddd = [aaa, bbb]
pa.array(ddd)
So seems pyarrow.array is limited with types of array elements, doesnt'it?
WBR,
Dima Fomin
24.10.2019, 15:47, "Dima Fomin" <da...@yandex.ru>:
> Hello!
>
> First of all thank you very much for your efforts on Apache Arrow! This
looks very useful and prominent tech to be able use it in self-BI area.
>
> I faced such issues - get Subj Error
>
>> ArrowNotImplementedError: No cast implemented from decimal(2,2) to
decimal(12,2)
>
> when trying to _pyarrow.Table.from_batches_ array of values defined in
schema as _pyarrow.decimal128(12,2)_.
>
> Which values are in fact column of
_sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"_ values,
which I map in schema like this _pyarrow.decimal128(12,2)_.
>
> What could be a reason of issue?
>
> Could it be that somehow declared decimal type of data is reduced to minimum
value which met in data? :/
>
> With best regards,
>
> Dima Fomin
Re: ArrowNotImplementedError: No cast implemented from decimal(2,2)
to decimal(12,2)
Posted by Micah Kornfield <em...@gmail.com>.
I filed:
https://issues.apache.org/jira/browse/ARROW-6990
https://issues.apache.org/jira/browse/ARROW-6989
On Thu, Oct 24, 2019 at 10:02 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Dima,
> Could you provide a minimal code sample for how you generated the error
> (the example below is a good one)?
>
> It sounds like it would be a good feature anyways to be able to cast
> between the decimals like this. Could you open a JIRA for this (
> https://issues.apache.org/jira
> <https://issues.apache.org/jira/secure/Dashboard.jspa>)
>
> Thanks,
> Micah
>
> On Thu, Oct 24, 2019 at 5:47 AM Dima Fomin <da...@yandex.ru> wrote:
>
>> Hello!
>> First of all thank you very much for your efforts on Apache Arrow! This
>> looks very useful and prominent tech to be able use it in self-BI area.
>>
>> I faced such issues - get Subj Error
>>
>> ArrowNotImplementedError: No cast implemented from decimal(2,2) to
>> decimal(12,2)
>>
>> when trying to *pyarrow.Table.from_batches* array of values defined in
>> schema as *pyarrow.decimal128(12,2)*.
>>
>> Which values are in fact column of
>> *sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"*
>> values, which I map in schema like this *pyarrow.decimal128(12,2)*.
>>
>> What could be a reason of issue?
>> Could it be that somehow declared decimal type of data is reduced to
>> minimum value which met in data? :/
>>
>> With best regards,
>> Dima Fomin
>>
>
Re: ArrowNotImplementedError: No cast implemented from decimal(2,2)
to decimal(12,2)
Posted by Micah Kornfield <em...@gmail.com>.
Hi Dima,
Could you provide a minimal code sample for how you generated the error
(the example below is a good one)?
It sounds like it would be a good feature anyways to be able to cast
between the decimals like this. Could you open a JIRA for this (
https://issues.apache.org/jira
<https://issues.apache.org/jira/secure/Dashboard.jspa>)
Thanks,
Micah
On Thu, Oct 24, 2019 at 5:47 AM Dima Fomin <da...@yandex.ru> wrote:
> Hello!
> First of all thank you very much for your efforts on Apache Arrow! This
> looks very useful and prominent tech to be able use it in self-BI area.
>
> I faced such issues - get Subj Error
>
> ArrowNotImplementedError: No cast implemented from decimal(2,2) to
> decimal(12,2)
>
> when trying to *pyarrow.Table.from_batches* array of values defined in
> schema as *pyarrow.decimal128(12,2)*.
>
> Which values are in fact column of
> *sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"*
> values, which I map in schema like this *pyarrow.decimal128(12,2)*.
>
> What could be a reason of issue?
> Could it be that somehow declared decimal type of data is reduced to
> minimum value which met in data? :/
>
> With best regards,
> Dima Fomin
>