You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "MIkhail Osckin (JIRA)" <ji...@apache.org> on 2017/10/09 14:34:00 UTC
[jira] [Created] (ARROW-1660) pandas field values are messed up
across rows
MIkhail Osckin created ARROW-1660:
-------------------------------------
Summary: pandas field values are messed up across rows
Key: ARROW-1660
URL: https://issues.apache.org/jira/browse/ARROW-1660
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.7.1
Environment: 4.4.0-72-generic #93-Ubuntu SMP x86_64, python3
Reporter: MIkhail Osckin
I have the following scala case class to store sparse matrix data to read it later using python
case class CooVector(
id: Int,
row_ids: Seq[Int],
rowsIdx: Seq[Int],
colIdx: Seq[Int],
data: Seq[Double])
I save the dataset of this type to multiple parquet files using spark and then read it using pyarrow.parquet and convert the result to pandas dataset.
The problem i have is that some values end up in wrong rows, for example, row_ids might end up in wrong cooVector row. I have no idea what the reason is but might be it is related to the fact that the fields are of variable sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)