[I] "ValueError: Table schema does not match schema used to create file" when field is null [arrow]

["ShaimaaSabry (via GitHub)" <gi...@apache.org>]

ShaimaaSabry opened a new issue, #43165:
URL: https://github.com/apache/arrow/issues/43165

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   I have the following schema:
   `bankrupcy: struct<bankrupcy_flag: bool, bankrupcy_records: list<item: struct<value: string>>>`
   
   and the following data:
   `[{'bankrupcy': {'bankrupcy_flag': False, 'bankrupcy_records': []}}]`
   where "bankrupcy_records" happens to be an empty array.
   
   I am trying to write the data to parquet using this code:
   `  headers = ["bankrupcy"]
       data = [
           {
               "bankrupcy": {
                   "bankrupcy_flag": False,
                   "bankrupcy_records": []
               }
           }
       ]
   
       data_transposed = defaultdict(list)
       for row in data:
           for header in headers:
               value = row.get(header)
               value = value.as_py() if isinstance(value, Scalar) else value
               data_transposed[header].append(value)
   
       table = Table.from_pydict(data_transposed)
       
       writer = ParquetWriter('example.parquet', schema, version="1.0")
       writer.write_table(table)
       writer.close()` 
   
   I am getting this error:
   `ValueError: Table schema does not match schema used to create file: 
   table:
   bankrupcy: struct<bankrupcy_flag: bool, bankrupcy_records: list<item: null>>
     child 0, bankrupcy_flag: bool
     child 1, bankrupcy_records: list<item: null>
         child 0, item: null vs. 
   file:
   bankrupcy: struct<bankrupcy_flag: bool, bankrupcy_records: list<item: struct<value: string>>>
     child 0, bankrupcy_flag: bool
     child 1, bankrupcy_records: list<item: struct<value: string>>
         child 0, item: struct<value: string>
             child 0, value: string`
   
   How can I define the schema so that pyarrow would allow the value of "bankrupcy_records"  to be an empty list sometimes?
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] "ValueError: Table schema does not match schema used to create file" when field is null [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic closed issue #43165: "ValueError: Table schema does not match schema used to create file" when field is null
URL: https://github.com/apache/arrow/issues/43165


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] "ValueError: Table schema does not match schema used to create file" when field is null [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #43165:
URL: https://github.com/apache/arrow/issues/43165#issuecomment-3547051640

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] "ValueError: Table schema does not match schema used to create file" when field is null [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #43165:
URL: https://github.com/apache/arrow/issues/43165#issuecomment-3670578337

   Closing as stale usage question with no recent activity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org