You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Yevgeni Litvin (Jira)" <ji...@apache.org> on 2019/11/20 07:57:00 UTC
[jira] [Updated] (ARROW-7214) [Python] unpickling a pyarrow table
with dictionary fields crashes
[ https://issues.apache.org/jira/browse/ARROW-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yevgeni Litvin updated ARROW-7214:
----------------------------------
Description:
The following code crashes on this check:
{code:java}
F1120 07:51:37.523720 12432 array.cc:773] Check failed: (data->dictionary) != (nullptr)
{code}
{code:java}
import cPickle as pickle
import pandas as pd
import pyarrow as pa
df = pd.DataFrame([{"cat": "a", "val":1},{"cat": "b", "val":2} ])
df["cat"] = df["cat"].astype('category')index_table = pa.Table.from_pandas(df, preserve_index=False)
with open('/tmp/zz.pickle', 'wb') as f:
pickle.dump(index_table, f, protocol=2)
with open('/tmp/zz.pickle', 'rb') as f:
index_table = pickle.load(f)
{code}
Used Python2 with the following environment:
{code:java}
Package Version
--------------- -------
enum34 1.1.6
futures 3.3.0
numpy 1.16.5
pandas 0.24.2
pip 19.3.1
pyarrow 0.14.1 (0.14.0 and up suffer from this issue)
python-dateutil 2.8.1
pytz 2019.3
setuptools 41.6.0
six 1.13.0
{code}
was:
The following code crashes on this check:
{code:java}
F1120 07:51:37.523720 12432 array.cc:773] Check failed: (data->dictionary) != (nullptr)
{code}
Used pandas 0.24.2.
{code:java}
import cPickle as pickle
import pandas as pd
import pyarrow as pa
df = pd.DataFrame([{"cat": "a", "val":1},{"cat": "b", "val":2} ])
df["cat"] = df["cat"].astype('category')index_table = pa.Table.from_pandas(df, preserve_index=False)
with open('/tmp/zz.pickle', 'wb') as f:
pickle.dump(index_table, f, protocol=2)
with open('/tmp/zz.pickle', 'rb') as f:
index_table = pickle.load(f)
{code}
> [Python] unpickling a pyarrow table with dictionary fields crashes
> ------------------------------------------------------------------
>
> Key: ARROW-7214
> URL: https://issues.apache.org/jira/browse/ARROW-7214
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.14.0, 0.14.1, 0.15.0, 0.15.1
> Reporter: Yevgeni Litvin
> Priority: Major
>
> The following code crashes on this check:
> {code:java}
> F1120 07:51:37.523720 12432 array.cc:773] Check failed: (data->dictionary) != (nullptr)
> {code}
>
> {code:java}
> import cPickle as pickle
> import pandas as pd
> import pyarrow as pa
> df = pd.DataFrame([{"cat": "a", "val":1},{"cat": "b", "val":2} ])
> df["cat"] = df["cat"].astype('category')index_table = pa.Table.from_pandas(df, preserve_index=False)
> with open('/tmp/zz.pickle', 'wb') as f:
> pickle.dump(index_table, f, protocol=2)
> with open('/tmp/zz.pickle', 'rb') as f:
> index_table = pickle.load(f)
> {code}
>
> Used Python2 with the following environment:
> {code:java}
> Package Version
> --------------- -------
> enum34 1.1.6
> futures 3.3.0
> numpy 1.16.5
> pandas 0.24.2
> pip 19.3.1
> pyarrow 0.14.1 (0.14.0 and up suffer from this issue)
> python-dateutil 2.8.1
> pytz 2019.3
> setuptools 41.6.0
> six 1.13.0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)