You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Micah Kornfield (Jira)" <ji...@apache.org> on 2019/09/25 04:38:00 UTC
[jira] [Resolved] (ARROW-6678) [C++] Regression in Parquet file
compatibility introduced by ARROW-3246
[ https://issues.apache.org/jira/browse/ARROW-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Micah Kornfield resolved ARROW-6678.
------------------------------------
Resolution: Fixed
Issue resolved by pull request 5493
[https://github.com/apache/arrow/pull/5493]
> [C++] Regression in Parquet file compatibility introduced by ARROW-3246
> -----------------------------------------------------------------------
>
> Key: ARROW-6678
> URL: https://issues.apache.org/jira/browse/ARROW-6678
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Wes McKinney
> Assignee: Wes McKinney
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.15.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> I randomly discovered that this script fails after applying the patch for ARROW-3246
> https://github.com/apache/arrow/commit/2ba0566b29312e84fafc987fd8dc9664748be96a
> {code}
> import pandas as pd
> import pyarrow as pa
> import pyarrow.parquet as pq
> import fastparquet as fp
> df = pd.util.testing.makeDataFrame()
> pq.write_table(pa.table(df), 'test.parquet')
> fp.ParquetFile('test.parquet')
> {code}
> with
> {code}
> Traceback (most recent call last):
> File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 110, in __init__
> with open_with(fn2, 'rb') as f:
> File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/util.py", line 38, in default_open
> return open(f, mode)
> NotADirectoryError: [Errno 20] Not a directory: 'test.parquet/_metadata'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "test.py", line 10, in <module>
> fp.ParquetFile('test.parquet')
> File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 116, in __init__
> self._parse_header(f, verify)
> File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 135, in _parse_header
> fmd = read_thrift(f, parquet_thrift.FileMetaData)
> File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/thrift_structures.py", line 25, in read_thrift
> obj.read(pin)
> File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py", line 1929, in read
> iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec])
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
> {code}
> I don't recall making any metadata-related changes but I'm going to review the patch to see if I can narrow down where the problem is to see whether it's a bug with Arrow/parquet-cpp or with the third party library
--
This message was sent by Atlassian Jira
(v8.3.4#803005)