[jira] [Created] (ARROW-15142) Cannot mix struct and non-struct, non-null values error when saving nested types with PyArrow

["Karthik (Jira)" <ji...@apache.org>]

Karthik created ARROW-15142:
-------------------------------

             Summary: Cannot mix struct and non-struct, non-null values error when saving nested types with PyArrow 
                 Key: ARROW-15142
                 URL: https://issues.apache.org/jira/browse/ARROW-15142
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 6.0.1
            Reporter: Karthik


When trying to save a Pandas dataframe with a nested type (list within list, list within dict) using pyarrow engine, the following error is encountered

{color:#e75c58}ArrowInvalid{color}: ('cannot mix list and non-list, non-null values', 'Conversion failed for column A with type object')

 

Repro:
{code:java}
import pandas as pd
x = pd.DataFrame({"A": [[24, 27, [1, 1]]]})
x.to_parquet('/tmp/a.pqt', engine="pyarrow")  {code}
Doing a bit of googling, it appears that this is a known Arrow shortcoming. However, this is a commonly encountered datastructure, and 'fastparquet' handles this seamlessly. Is there a proposed timeline/plan for fixing this?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)