You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Deuxis (via GitHub)" <gi...@apache.org> on 2025/08/20 16:02:18 UTC
[I] [Python] RecordBatch.as_pylist(maps_as_pydicts="strict") option doesn't work for some nested maps [arrow]
Deuxis opened a new issue, #47380:
URL: https://github.com/apache/arrow/issues/47380
### Describe the bug, including details regarding any error messages, version, and platform.
I have a `RecordBatch` `rb` with the following schema:
```
schema = pa.schema([
("col1", pa.map_("string", "string")),
("col2", pa.struct([
("f1", pa.int32()),
("f2", pa.map_("string", pa.struct([
("item", pa.list_(pa.struct([
("raw", "string"),
("component", pa.map_("string", "string"))
])))
])))
]))
])
```
When using `rb.to_pylist(maps_as_pydicts="strict")`, the map in `col1` exports correctly, as `{key:val}` dict, as does the map at `$['col2']['f2']` , while the map at `$['col2']['f2'][*]['item'][*]['component']` still exports as a `[(key, val)]` list of tuples:
```
[{'col1': {'A': 'B'},
'col2': {'f1': 1,
'f2': {'C': {'item': [{'raw': 'D', 'component': [('E', 'F')]}]}}}}]
```
Verified on pyarrow 21.0.0 on x86_64 Linux and Windows.
Use case and issue:
I need to adapt a pipeline to serialize a parquet file to JSON and deserialize back into a parquet file. Tuples serialize to JSON arrays and sadly even with the schema provided `RecordBatch.from_pylist` keels over and dies upon encountering a list of lists where it expected the Map. (`ArrowTypeError: Could not convert 'E' with type str: was expecting tuple of (key, value) pair`) Trying to transform the JSON into JSONL and load it via `pyarrow.json.read_json` has even worse results, as it straight-up doesn't support loading Maps and gives up when I hand it the schema.
The maps correctly exported to dicts->json objects import with no problem, so fixing this bug will solve my issue.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] RecordBatch.as_pylist(maps_as_pydicts="strict") option doesn't work for some nested maps [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #47380: [Python] RecordBatch.as_pylist(maps_as_pydicts="strict") option doesn't work for some nested maps
URL: https://github.com/apache/arrow/issues/47380
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] RecordBatch.as_pylist(maps_as_pydicts="strict") option doesn't work for some nested maps [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #47380:
URL: https://github.com/apache/arrow/issues/47380#issuecomment-3375527480
@jo-migo can you add a comment on the issue so we can assign it to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] RecordBatch.as_pylist(maps_as_pydicts="strict") option doesn't work for some nested maps [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #47380:
URL: https://github.com/apache/arrow/issues/47380#issuecomment-3375530027
Issue resolved by pull request 47454
https://github.com/apache/arrow/pull/47454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org