[jira] [Comment Edited] (ARROW-8964) [Python][Parquet] improve reading of partitioned parquet datasets whose schema changed

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129009#comment-17129009 ] 

Joris Van den Bossche edited comment on ARROW-8964 at 6/9/20, 8:52 AM:
-----------------------------------------------------------------------

I am not familiar with Impala. Looking at https://impala.apache.org/docs/build/html/topics/impala_parquet.html#parquet_data_types, it seems that the INT64 based timestamp types (the default when writing with pyarrow) is only supported in Impala > 3.2). If you need the older INT96 representation for timestamps, there is an option {{use_deprecated_int96_timestamps=True}} you can set in {{pq.write_table}}

bq. Should i close this task as it's pretty much a duplicate?

Yes, will close it



was (Author: jorisvandenbossche):
I am not familiar with Impala. Looking at https://impala.apache.org/docs/build/html/topics/impala_parquet.html#parquet_data_types, it seems that the INT64 based timestamp types (the default when writing with pyarrow) is only supported in Impala > 3.2). If you need the older INT96 representation for timestamps, there is an option {{use_deprecated_int96_timestamps=True}} you can set in {{pq.write_table}}

.bq Should i close this task as it's pretty much a duplicate?

Yes, will close it


> [Python][Parquet] improve reading of partitioned parquet datasets whose schema changed
> --------------------------------------------------------------------------------------
>
>                 Key: ARROW-8964
>                 URL: https://issues.apache.org/jira/browse/ARROW-8964
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>    Affects Versions: 0.17.1
>         Environment: Ubuntu 18.04, latest miniconda with python 3.7, pyarrow 0.17.1
>            Reporter: Ira Saktor
>            Priority: Major
>
> Hi there, i'm encountering the following issue when reading from HDFS:
>  
> *My situation:*
> I have a paritioned parquet dataset in HDFS, whose recent partitions contain parquet files with more columns than the older ones. When i try to read data using pyarrow.dataset.dataset and filter on recent data, i still get only the columns that are also contained in the old parquet files. I'd like to somehow merge the schema or use the schema from parquet files from which data ends up being loaded.
> *when using:*
> `pyarrow.dataset.dataset(path_to_hdfs_directory, paritioning = 'hive', filters = my_filter_expression).to_table().to_pandas()`
> Is there please a way to handle schema changes in a way, that the read data would contain all columns?
> everything works fine when i copy the needed parquet files into a separate folder, however it is very inconvenient way of working. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)