Table.cast throws ArrowNotImplementedError (pyarrow==0.15.0)

[Lucas Pickup <Lu...@microsoft.com.INVALID>]

So it seems in 'pyarrow==0.15.0' `Table.columns` now returns ChunkedArray instead of Column. This has broken `Table.cast()` as it just calls `Table.itercolumns` and expects the yielded values to have a `.cast()` method, which ChunkedArray doesn't.

Was `Table.cast()` missed in cleaning up after removing Column? Or is casting still supposed to be possible on a Table (or even ChunkedArray) level?

Cheers,
Lucas

Re: Table.cast throws ArrowNotImplementedError (pyarrow==0.15.0)

[Joris Van den Bossche <jo...@gmail.com>]

Hi Lucas,

Do you have a small code example?

Trying the following worked in pyarrow 0.14, and still seems to work now:

In [1]: table = pa.table({'a': [1, 2, 3]})

In [2]: table
Out[2]:
pyarrow.Table
a: int64

In [3]: table.cast(pa.schema([('a', pa.int32())]))
Out[3]:
pyarrow.Table
a: int32

In [4]: pa.__version__
Out[4]: '0.15.1.dev73+g4db8f7ba2'

The ChunkedArray.cast() implementation is here:
https://github.com/apache/arrow/blob/4db8f7ba27ccfaceb0bf1638a54405039d164de0/python/pyarrow/table.pxi#L198-L223
and was added to ChunkedArray in the PR that Column was removed.

Best,
Joris


On Wed, 9 Oct 2019 at 02:19, Lucas Pickup
<Lu...@microsoft.com.invalid> wrote:

> So it seems in 'pyarrow==0.15.0' `Table.columns` now returns ChunkedArray
> instead of Column. This has broken `Table.cast()` as it just calls
> `Table.itercolumns` and expects the yielded values to have a `.cast()`
> method, which ChunkedArray doesn't.
>
> Was `Table.cast()` missed in cleaning up after removing Column? Or is
> casting still supposed to be possible on a Table (or even ChunkedArray)
> level?
>
> Cheers,
> Lucas
>