[jira] [Commented] (ARROW-17360) [Python] Reorder columns in pyarrow.feather.read_table

["Alenka Frim (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17625032#comment-17625032 ] 

Alenka Frim commented on ARROW-17360:
-------------------------------------

Unfortunately the easy solution for reordering the columns in pyarrow isn't feasible as {{pyarrow.Table.select()}} with "dotted path" doesn't work but "dotted path" can be used when selecting a column with {{{}ORCF.read(){}}}:
{code:java}
>       result4 = orc_file.read(columns=["struct.middle.inner"])

opt/conda/envs/arrow/lib/python3.8/site-packages/pyarrow/tests/test_orc.py:584: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
opt/conda/envs/arrow/lib/python3.8/site-packages/pyarrow/orc.py:189: in read
    table = table.select(columns)
pyarrow/table.pxi:3053: in pyarrow.lib.Table.select
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   KeyError: 'Field "struct.middle.inner" does not exist in table schema'
{code}
To close this issue I will add information to the {{orc.read_table()}} docstrings that we always follow the order of the file.

*A workaround for a user with ordering issue:*
 * add {{.select(columns= ['b', 'a']))}} after reading the {{Table}} from the orc file

_I still think a better solution would be that pandas starts using the new dataset API as mentioned above._

> [Python] Reorder columns in pyarrow.feather.read_table
> ------------------------------------------------------
>
>                 Key: ARROW-17360
>                 URL: https://issues.apache.org/jira/browse/ARROW-17360
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>    Affects Versions: 8.0.1
>            Reporter: Matthew Roeschke
>            Assignee: Alenka Frim
>            Priority: Major
>              Labels: orc, pull-request-available
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> xref [https://github.com/pandas-dev/pandas/issues/47944]
>  
> {code:java}
> In [1]: df = pd.DataFrame({"a": [1, 2, 3], "b": ["a", "b", "c"]})
> # pandas main branch / 1.5
> In [2]: df.to_orc("abc")
> In [3]: pd.read_orc("abc", columns=['b', 'a'])
> Out[3]:
>    a  b
> 0  1  a
> 1  2  b
> 2  3  c
> In [4]: import pyarrow.orc as orc
> In [5]: orc_file = orc.ORCFile("abc")
> # reordered to a, b
> In [6]: orc_file.read(columns=['b', 'a']).to_pandas()
> Out[6]:
>    a  b
> 0  1  a
> 1  2  b
> 2  3  c
> # reordered to a, b
> In [7]: orc_file.read(columns=['b', 'a'])
> Out[7]:
> pyarrow.Table
> a: int64
> b: string
> ----
> a: [[1,2,3]]
> b: [["a","b","c"]] {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)