You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/06/18 13:44:59 UTC
[GitHub] [arrow] Demetrio92 commented on issue #1688: Possible to read categoricals back into Pandas from Parquet using Pyarrow?
Demetrio92 commented on issue #1688:
URL: https://github.com/apache/arrow/issues/1688#issuecomment-646026392
Stumbled across this bug again.
`pyarrow` preserves `category` as `dtype`, `fastparquet` **does not**.
Docs don't mention it. They even kinda mislead the users:
https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-parquet

^ this works, because the second option has `engine='pyarrow'`!!!
So, the default `auto`, which uses `fastparquet` would actually result in those being `object`.
_This took me a bit, before I noticed where and why do I have memory leaks despite optimizing my dataframes..._
```python
pd.__version__
Out[3]: '1.0.4'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org