You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Matthew Topol (Jira)" <ji...@apache.org> on 2022/11/09 17:17:00 UTC

[jira] [Updated] (ARROW-18288) [GO]: pqarrow (github.com/apache/arrow/go/v9/parquet/pqarrow) cannot handle arrow's DICTIONARY field

     [ https://issues.apache.org/jira/browse/ARROW-18288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthew Topol updated ARROW-18288:
----------------------------------
    Component/s: Parquet

> [GO]: pqarrow (github.com/apache/arrow/go/v9/parquet/pqarrow) cannot handle arrow's DICTIONARY field
> ----------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-18288
>                 URL: https://issues.apache.org/jira/browse/ARROW-18288
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Go, Parquet
>    Affects Versions: 9.0.0, 10.0.0
>            Reporter: Kevin Yang
>            Assignee: Matthew Topol
>            Priority: Minor
>
> Hey, Arrow Go Dev:
>  
> I was trying to save some arrow tables out to parquet files, with the help of the "[github.com/apache/arrow/go/v9/parquet/pqarrow|http://github.com/apache/arrow/go/v9/parquet/pqarrow]" package. btw, it's generally a great design (of Arrow) and a great Go implementation. 
>  
> However, one issue sticks out: in my original arrow Table I have some DICTIONARY fields, which pqarrow does NOT currently support.
>  
> I would assume supporting them will be quite straightward: just "denormalize" the DICTIONARY value into corresponding values (string, Timestamp, etc), and it's up to the Parquet to do the right trick (using DICTIONARY encoding, etc). 
>  
> I would have done this conversion on-the-fly by myself, by converting each DICTIONARY field into underlying values. However, the arrow table schema is dynamic and outside my control, and I need to iterate through fields (maybe structs) to locate those) -> it would be much better if pqarrow can support this natively. 
>  
> Can anyone help? thanks!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)