[jira] [Created] (ARROW-3728) Merging Parquet Files - Pandas Meta in Schema Mismatch

["Micah Williamson (JIRA)" <ji...@apache.org>]

Micah Williamson created ARROW-3728:
---------------------------------------

             Summary: Merging Parquet Files - Pandas Meta in Schema Mismatch
                 Key: ARROW-3728
                 URL: https://issues.apache.org/jira/browse/ARROW-3728
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 0.11.1, 0.11.0, 0.10.0
         Environment: Python 3.6.3
OSX 10.14
            Reporter: Micah Williamson


From: https://stackoverflow.com/questions/53214288/merging-parquet-files-pandas-meta-in-schema-mismatch
 
I am trying to merge multiple parquet files into one. Their schemas are identical field-wise but my {{ParquetWriter}} is complaining that they are not. After some investigation I found that the pandas meta in the schemas are different, causing this error.
 
Sample-

{code:python}
import pyarrow.parquet as pq

pq_tables=[]
for file_ in files:
    pq_table = pq.read_table(f'{MESS_DIR}/{file_}')
    pq_tables.append(pq_table)
    if writer is None:
        writer = pq.ParquetWriter(COMPRESSED_FILE, schema=pq_table.schema, use_deprecated_int96_timestamps=True)
    writer.write_table(table=pq_table)
{code}

The error-

{code}
Traceback (most recent call last):
  File "{PATH_TO}/main.py", line 68, in lambda_handler
    writer.write_table(table=pq_table)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyarrow/parquet.py", line 335, in write_table
    raise ValueError(msg)
ValueError: Table schema does not match schema used to create file:
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)