You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Bhavitvya Malik <bh...@gmail.com> on 2021/02/11 10:33:25 UTC
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
Hi,
It's a follow up question for #9462
<https://github.com/apache/arrow/issues/9462>. Rewriting the issue here:
I came to know that pyarrow has this limitation of not storing
> N-dimensional array. After looking into this issue, I decided to represent
> a N-dimensional array as a list of arrays i.e.
> data = np.zeros((5, 3), dtype=np.uint8)
> data = list(data)
> inorder to preserve the dtype but when it comes to typecasting and writing
> it into array (from list) pyarrow.array(data, type=type) it gives the
> following error:
> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type
> numpy.ndarray: tried to convert to int
> Is there any way to avoid this issue? I just want to preserve the dtype
> from numpy array before converting it to list so that while writing it to
> pyarrow array format I can recognise its dtype and subsequently write it in
> that numpy dtype format.
I tried it with a 3D numpy array and it gave me this error even though it's
working fine with 2D numpy arrays. Can you please look into this?
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0 too
Thanks,
Bhavitvya
Re: pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
Posted by Wes McKinney <we...@gmail.com>.
We simply don't have conversions nor type inference implemented for
inner elements with dimension greater than 1. You're welcome to
propose this as a new feature / enhancement by opening a Jira issue.
On Thu, Feb 11, 2021 at 10:53 AM Bhavitvya Malik
<bh...@gmail.com> wrote:
>
> Sure, here it is:
>
> data = np.zeros((10,8), dtype=np.uint8)
> out = pa.array(list(data))
> out.type # ListType(list<item: uint8>)
>
> data = np.zeros((3,4,6), dtype=np.uint8)
> out = pa.array(list(data)) # Throws error ArrowInvalid: Can only convert 1-dimensional array values
>
> Even though it's working on 2D numpy arrays perfectly, it doesn't work on N-Dimensional numpy arrays (where N > 2). Why is it so?
>
>
> On Thu, 11 Feb 2021 at 21:18, Wes McKinney <we...@gmail.com> wrote:
>>
>> Can you provide more detail about what you are trying? You've showed
>> some exception here but haven't showed the exact code that results in
>> those exceptions
>>
>> On Thu, Feb 11, 2021 at 4:34 AM Bhavitvya Malik
>> <bh...@gmail.com> wrote:
>> >
>> > Hi,
>> > It's a follow up question for #9462. Rewriting the issue here:
>> >
>> >> I came to know that pyarrow has this limitation of not storing N-dimensional array. After looking into this issue, I decided to represent a N-dimensional array as a list of arrays i.e.
>> >> data = np.zeros((5, 3), dtype=np.uint8)
>> >> data = list(data)
>> >> inorder to preserve the dtype but when it comes to typecasting and writing it into array (from list) pyarrow.array(data, type=type) it gives the following error:
>> >> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type numpy.ndarray: tried to convert to int
>> >> Is there any way to avoid this issue? I just want to preserve the dtype from numpy array before converting it to list so that while writing it to pyarrow array format I can recognise its dtype and subsequently write it in that numpy dtype format.
>> >
>> >
>> > I tried it with a 3D numpy array and it gave me this error even though it's working fine with 2D numpy arrays. Can you please look into this?
>> >
>> > pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
>> >
>> >
>> > My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0 too
>> >
>> >
>> >
>> > Thanks,
>> > Bhavitvya
Re: pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
Posted by Bhavitvya Malik <bh...@gmail.com>.
Sure, here it is:
data = np.zeros((10,8), dtype=np.uint8)
out = pa.array(list(data))
out.type # ListType(list<item: uint8>)
data = np.zeros((3,4,6), dtype=np.uint8)
out = pa.array(list(data)) # Throws error ArrowInvalid: Can only
convert 1-dimensional array values
Even though it's working on 2D numpy arrays perfectly, it doesn't work on
N-Dimensional numpy arrays (where N > 2). Why is it so?
On Thu, 11 Feb 2021 at 21:18, Wes McKinney <we...@gmail.com> wrote:
> Can you provide more detail about what you are trying? You've showed
> some exception here but haven't showed the exact code that results in
> those exceptions
>
> On Thu, Feb 11, 2021 at 4:34 AM Bhavitvya Malik
> <bh...@gmail.com> wrote:
> >
> > Hi,
> > It's a follow up question for #9462. Rewriting the issue here:
> >
> >> I came to know that pyarrow has this limitation of not storing
> N-dimensional array. After looking into this issue, I decided to represent
> a N-dimensional array as a list of arrays i.e.
> >> data = np.zeros((5, 3), dtype=np.uint8)
> >> data = list(data)
> >> inorder to preserve the dtype but when it comes to typecasting and
> writing it into array (from list) pyarrow.array(data, type=type) it gives
> the following error:
> >> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type
> numpy.ndarray: tried to convert to int
> >> Is there any way to avoid this issue? I just want to preserve the dtype
> from numpy array before converting it to list so that while writing it to
> pyarrow array format I can recognise its dtype and subsequently write it in
> that numpy dtype format.
> >
> >
> > I tried it with a 3D numpy array and it gave me this error even though
> it's working fine with 2D numpy arrays. Can you please look into this?
> >
> > pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
> >
> >
> > My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0
> too
> >
> >
> >
> > Thanks,
> > Bhavitvya
>
Re: pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
Posted by Wes McKinney <we...@gmail.com>.
Can you provide more detail about what you are trying? You've showed
some exception here but haven't showed the exact code that results in
those exceptions
On Thu, Feb 11, 2021 at 4:34 AM Bhavitvya Malik
<bh...@gmail.com> wrote:
>
> Hi,
> It's a follow up question for #9462. Rewriting the issue here:
>
>> I came to know that pyarrow has this limitation of not storing N-dimensional array. After looking into this issue, I decided to represent a N-dimensional array as a list of arrays i.e.
>> data = np.zeros((5, 3), dtype=np.uint8)
>> data = list(data)
>> inorder to preserve the dtype but when it comes to typecasting and writing it into array (from list) pyarrow.array(data, type=type) it gives the following error:
>> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type numpy.ndarray: tried to convert to int
>> Is there any way to avoid this issue? I just want to preserve the dtype from numpy array before converting it to list so that while writing it to pyarrow array format I can recognise its dtype and subsequently write it in that numpy dtype format.
>
>
> I tried it with a 3D numpy array and it gave me this error even though it's working fine with 2D numpy arrays. Can you please look into this?
>
> pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
>
>
> My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0 too
>
>
>
> Thanks,
> Bhavitvya