You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Andy Reagan (JIRA)" <ji...@apache.org> on 2018/05/31 20:16:00 UTC
[jira] [Created] (ARROW-2654) pyarrow cannot load a file it created
Andy Reagan created ARROW-2654:
----------------------------------
Summary: pyarrow cannot load a file it created
Key: ARROW-2654
URL: https://issues.apache.org/jira/browse/ARROW-2654
Project: Apache Arrow
Issue Type: Bug
Reporter: Andy Reagan
I saved a file using pandas to_parquet method, but can't read it back in. Here's the full stack trace:
```
Traceback (most recent call last):
File "src/data/CLXP_pull.py", line 214, in <module>
main()
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "src/data/CLXP_pull.py", line 188, in main
results[fullname] = pd.read_parquet(os.path.join(project_dir, "data", "raw", fullname+".parquet"), engine="pyarrow")
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py", line 257, in read_parquet
return impl.read(path, columns=columns, **kwargs)
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py", line 130, in read
**kwargs).to_pandas()
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py", line 939, in read_table
pf = ParquetFile(source, metadata=metadata)
File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py", line 64, in __init__
self.reader.open(source, metadata=metadata)
File "_parquet.pyx", line 651, in pyarrow._parquet.ParquetReader.open
File "error.pxi", line 79, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Arrow error: IOError: [Errno 22] Invalid argument
```
Any ideas what could cause this? The file itself is 3.6GB.
I'm running pandas==0.22.0 and arrow==0.12.1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)