You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Jakub Okoński (JIRA)" <ji...@apache.org> on 2018/09/03 17:06:00 UTC
[jira] [Commented] (ARROW-3139) [Python]ArrowIOError: Arrow error:
Capacity error during read
[ https://issues.apache.org/jira/browse/ARROW-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602353#comment-16602353 ]
Jakub Okoński commented on ARROW-3139:
--------------------------------------
This looks similar to PARQUET-1239 Are they related?
> [Python]ArrowIOError: Arrow error: Capacity error during read
> -------------------------------------------------------------
>
> Key: ARROW-3139
> URL: https://issues.apache.org/jira/browse/ARROW-3139
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.10.0
> Environment: pandas=0.23.1=py36h637b7d7_0
> pyarrow==0.10.0
> Reporter: Frédérique Vanneste
> Priority: Major
>
> My assumption: the problem is caused by a large object column containing strings up to 27 characters long. (so that column is much larger than 2GB of strings, chunking issue)
> looks similar as https://issues.apache.org/jira/browse/ARROW-2227?focusedCommentId=16379574&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16379574
> Code
> * basket_plateau= pq.read_table("basket_plateau.parquet")
> * basket_plateau = pd.read_parquet("basket_plateau.parquet")
> Error produced
> * ArrowIOError: Arrow error: Capacity error: BinaryArray cannot contain more than 2147483646 bytes, have 2147483655
> Dataset
> * Pandas dataframe (pandas=0.23.1=py36h637b7d7_0)
> * 2.7 billion record, 4 columns ( int64/object/datetime64/float64)
> * aprox 90GB in memory
> * example of object col: "Fresh Vegetables", "Alcohol Beers", ... (think food retail categories)
> History to bug:
> * was using older version of pyarrow
> * tried writing dataset to disk (parquet) and failed
> * stumbled on https://issues.apache.org/jira/browse/ARROW-2227
> * upgraded to 0.10
> * tried writing dataset to disk (parquet) and succeeded
> * tried reading dataset and failed
> * looks like a similar case as: https://issues.apache.org/jira/browse/ARROW-2227?focusedCommentId=16379574&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16379574
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)