[jira] [Commented] (ARROW-15233) pyarrow.dataset.dataset loses type information when reading parquet files

["Jim Fulton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-15233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17467475#comment-17467475 ] 

Jim Fulton commented on ARROW-15233:
------------------------------------

This seems to depend on how the parquet file was written. I need to dig more.

Closing (at least for now).

> pyarrow.dataset.dataset loses type information when reading parquet files
> -------------------------------------------------------------------------
>
>                 Key: ARROW-15233
>                 URL: https://issues.apache.org/jira/browse/ARROW-15233
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 6.0.1
>         Environment: Ubuntu 20.2, Python 3.8.10
>            Reporter: Jim Fulton
>            Priority: Major
>
> When reading a parquet containing time data:
>  
> {{>>> import pyarrow.dataset}}
> {{>>> ds = pyarrow.dataset.dataset('foo.parquet', format='parquet')}}
> {{>>> ds.schema[1].type}}
> {{DataType(time32[ms])}}
> I get {{DataType}} rather than {{{}Time32Type{}}}, which means I can't query time units.
> I assume this is an issue for other non-basic types.
> I can write code to scrape the type's string representation. Is there a better way?
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)