You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Jim Fulton (Jira)" <ji...@apache.org> on 2022/01/01 00:49:00 UTC
[jira] [Created] (ARROW-15233) pyarrow.dataset.dataset loses type information when reading parquet files
Jim Fulton created ARROW-15233:
----------------------------------
Summary: pyarrow.dataset.dataset loses type information when reading parquet files
Key: ARROW-15233
URL: https://issues.apache.org/jira/browse/ARROW-15233
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 6.0.1
Environment: Ubuntu 20.2, Python 3.8.10
Reporter: Jim Fulton
When reading a parquet containing time data:
{{>>> import pyarrow.dataset}}
{{>>> ds = pyarrow.dataset.dataset('foo.parquet', format='parquet')}}
{{>>> ds.schema[1].type}}
{{DataType(time32[ms])}}
I get {{DataType}} rather than {{{}Time32Type{}}}, which means I can't query time units.
I assume this is an issue for other non-basic types.
I can write code to scrape the type's string representation. Is there a better way?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)