You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Weston Pace <we...@gmail.com> on 2019/12/11 18:36:30 UTC
Efficiently allocating an empty vector (python)
I'm trying to combine multiple parquet files. They were produced at
different points in time and have different columns. For example, one has
columns A, B, C. Two has columns B, C, D. Three has columns C, D, E. I
want to concatenate all three into one table with columns A, B, C, D, E.
To do this I am adding the missing columns to each table. For example, I
am adding column D to table one and setting all values to null. In order
to do this I need to create a vector with length equal to one.num_rows and
set all values to null. The vector type is controlled by the type of D in
the other tables.
I am currently doing this by creating one large python list ahead of time
and using:
pa.array(big_list_of_nones, type=column_type, size=desired_size).slice(0,
desired_size)
However, this ends up being very slow. The calls to pa.array take longer
than reading the data in the first place.
I can build a large empty vector for every possible data type at the start
of my application but that seems inefficient.
Is there a good way to initialize a vector with all null values that I am
missing?
Re: Efficiently allocating an empty vector (python)
Posted by Weston Pace <we...@gmail.com>.
This works very well and is much simpler. Thank you for the workaround.
On Wed, Dec 11, 2019 at 10:29 AM Antoine Pitrou <an...@python.org> wrote:
>
> As a workaround, you can use the following hack:
>
> >>> arr = pa.Array.from_buffers(pa.null(), 123, [pa.py_buffer(b"")])
>
>
> >>> arr
>
>
> <pyarrow.lib.NullArray object at 0x7f9a84d79be8>
> 123 nulls
> >>> arr.cast(pa.int32())
>
>
> <pyarrow.lib.Int32Array object at 0x7f9a84d79d68>
> [
> null,
> null,
> null,
> null,
> null,
> null,
> null,
> null,
> null,
> null,
> ...
> null,
> null,
> null,
> null,
> null,
> null,
> null,
> null,
> null,
> null
> ]
>
>
> Regards
>
> Antoine.
>
>
> Le 11/12/2019 à 21:08, Weston Pace a écrit :
> > Thanks. Ted, I tried using numpy similar to your approach and had the
> same
> > performance. For the time being I am using a dictionary of data-type to
> > pre-allocated big empty array which should work for me in the meantime.
> >
> > On Wed, Dec 11, 2019 at 9:20 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> There's a C++ facility to do this, but it's not exposed in Python yet.
> >> I opened ARROW-7375 for it.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >> Le 11/12/2019 à 19:36, Weston Pace a écrit :
> >>> I'm trying to combine multiple parquet files. They were produced at
> >>> different points in time and have different columns. For example, one
> >> has
> >>> columns A, B, C. Two has columns B, C, D. Three has columns C, D,
> E. I
> >>> want to concatenate all three into one table with columns A, B, C, D,
> E.
> >>>
> >>> To do this I am adding the missing columns to each table. For
> example, I
> >>> am adding column D to table one and setting all values to null. In
> order
> >>> to do this I need to create a vector with length equal to one.num_rows
> >> and
> >>> set all values to null. The vector type is controlled by the type of D
> >> in
> >>> the other tables.
> >>>
> >>> I am currently doing this by creating one large python list ahead of
> time
> >>> and using:
> >>>
> >>> pa.array(big_list_of_nones, type=column_type,
> size=desired_size).slice(0,
> >>> desired_size)
> >>>
> >>> However, this ends up being very slow. The calls to pa.array take
> longer
> >>> than reading the data in the first place.
> >>>
> >>> I can build a large empty vector for every possible data type at the
> >> start
> >>> of my application but that seems inefficient.
> >>>
> >>> Is there a good way to initialize a vector with all null values that I
> am
> >>> missing?
> >>>
> >>
> >
>
Re: Efficiently allocating an empty vector (python)
Posted by Antoine Pitrou <an...@python.org>.
As a workaround, you can use the following hack:
>>> arr = pa.Array.from_buffers(pa.null(), 123, [pa.py_buffer(b"")])
>>> arr
<pyarrow.lib.NullArray object at 0x7f9a84d79be8>
123 nulls
>>> arr.cast(pa.int32())
<pyarrow.lib.Int32Array object at 0x7f9a84d79d68>
[
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
...
null,
null,
null,
null,
null,
null,
null,
null,
null,
null
]
Regards
Antoine.
Le 11/12/2019 à 21:08, Weston Pace a écrit :
> Thanks. Ted, I tried using numpy similar to your approach and had the same
> performance. For the time being I am using a dictionary of data-type to
> pre-allocated big empty array which should work for me in the meantime.
>
> On Wed, Dec 11, 2019 at 9:20 AM Antoine Pitrou <an...@python.org> wrote:
>
>>
>> There's a C++ facility to do this, but it's not exposed in Python yet.
>> I opened ARROW-7375 for it.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 11/12/2019 à 19:36, Weston Pace a écrit :
>>> I'm trying to combine multiple parquet files. They were produced at
>>> different points in time and have different columns. For example, one
>> has
>>> columns A, B, C. Two has columns B, C, D. Three has columns C, D, E. I
>>> want to concatenate all three into one table with columns A, B, C, D, E.
>>>
>>> To do this I am adding the missing columns to each table. For example, I
>>> am adding column D to table one and setting all values to null. In order
>>> to do this I need to create a vector with length equal to one.num_rows
>> and
>>> set all values to null. The vector type is controlled by the type of D
>> in
>>> the other tables.
>>>
>>> I am currently doing this by creating one large python list ahead of time
>>> and using:
>>>
>>> pa.array(big_list_of_nones, type=column_type, size=desired_size).slice(0,
>>> desired_size)
>>>
>>> However, this ends up being very slow. The calls to pa.array take longer
>>> than reading the data in the first place.
>>>
>>> I can build a large empty vector for every possible data type at the
>> start
>>> of my application but that seems inefficient.
>>>
>>> Is there a good way to initialize a vector with all null values that I am
>>> missing?
>>>
>>
>
Re: Efficiently allocating an empty vector (python)
Posted by Weston Pace <we...@gmail.com>.
Thanks. Ted, I tried using numpy similar to your approach and had the same
performance. For the time being I am using a dictionary of data-type to
pre-allocated big empty array which should work for me in the meantime.
On Wed, Dec 11, 2019 at 9:20 AM Antoine Pitrou <an...@python.org> wrote:
>
> There's a C++ facility to do this, but it's not exposed in Python yet.
> I opened ARROW-7375 for it.
>
> Regards
>
> Antoine.
>
>
> Le 11/12/2019 à 19:36, Weston Pace a écrit :
> > I'm trying to combine multiple parquet files. They were produced at
> > different points in time and have different columns. For example, one
> has
> > columns A, B, C. Two has columns B, C, D. Three has columns C, D, E. I
> > want to concatenate all three into one table with columns A, B, C, D, E.
> >
> > To do this I am adding the missing columns to each table. For example, I
> > am adding column D to table one and setting all values to null. In order
> > to do this I need to create a vector with length equal to one.num_rows
> and
> > set all values to null. The vector type is controlled by the type of D
> in
> > the other tables.
> >
> > I am currently doing this by creating one large python list ahead of time
> > and using:
> >
> > pa.array(big_list_of_nones, type=column_type, size=desired_size).slice(0,
> > desired_size)
> >
> > However, this ends up being very slow. The calls to pa.array take longer
> > than reading the data in the first place.
> >
> > I can build a large empty vector for every possible data type at the
> start
> > of my application but that seems inefficient.
> >
> > Is there a good way to initialize a vector with all null values that I am
> > missing?
> >
>
Re: Efficiently allocating an empty vector (python)
Posted by Antoine Pitrou <an...@python.org>.
There's a C++ facility to do this, but it's not exposed in Python yet.
I opened ARROW-7375 for it.
Regards
Antoine.
Le 11/12/2019 à 19:36, Weston Pace a écrit :
> I'm trying to combine multiple parquet files. They were produced at
> different points in time and have different columns. For example, one has
> columns A, B, C. Two has columns B, C, D. Three has columns C, D, E. I
> want to concatenate all three into one table with columns A, B, C, D, E.
>
> To do this I am adding the missing columns to each table. For example, I
> am adding column D to table one and setting all values to null. In order
> to do this I need to create a vector with length equal to one.num_rows and
> set all values to null. The vector type is controlled by the type of D in
> the other tables.
>
> I am currently doing this by creating one large python list ahead of time
> and using:
>
> pa.array(big_list_of_nones, type=column_type, size=desired_size).slice(0,
> desired_size)
>
> However, this ends up being very slow. The calls to pa.array take longer
> than reading the data in the first place.
>
> I can build a large empty vector for every possible data type at the start
> of my application but that seems inefficient.
>
> Is there a good way to initialize a vector with all null values that I am
> missing?
>
Re: Efficiently allocating an empty vector (python)
Posted by Ted Gooch <te...@gmail.com>.
Not sure if this is any better, but I have an open PR right now in Iceberg,
where we are doing something similar:
https://github.com/apache/incubator-iceberg/pull/544/commits/28166fd3f0e3a24863048a2721f1ae69f243e2af#diff-51d6edf951c105e1e62a3f1e8b4640aaR319-R341
@staticmethod
def create_null_column(reference_column, name, dtype_tuple):
dtype, init_val = dtype_tuple
chunk = pa.chunked_array([pa.array(np.full(len(c), init_val),
type=dtype, mask=[True] * len(c))
for c in reference_column.data.chunks],
type=dtype)
return pa.Column.from_array(name, chunk)
Note, that this is using the <0.15 column API, which has been deprecated.
On Wed, Dec 11, 2019 at 10:36 AM Weston Pace <we...@gmail.com> wrote:
> I'm trying to combine multiple parquet files. They were produced at
> different points in time and have different columns. For example, one has
> columns A, B, C. Two has columns B, C, D. Three has columns C, D, E. I
> want to concatenate all three into one table with columns A, B, C, D, E.
>
> To do this I am adding the missing columns to each table. For example, I
> am adding column D to table one and setting all values to null. In order
> to do this I need to create a vector with length equal to one.num_rows and
> set all values to null. The vector type is controlled by the type of D in
> the other tables.
>
> I am currently doing this by creating one large python list ahead of time
> and using:
>
> pa.array(big_list_of_nones, type=column_type, size=desired_size).slice(0,
> desired_size)
>
> However, this ends up being very slow. The calls to pa.array take longer
> than reading the data in the first place.
>
> I can build a large empty vector for every possible data type at the start
> of my application but that seems inefficient.
>
> Is there a good way to initialize a vector with all null values that I am
> missing?
>