You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "edmcman (via GitHub)" <gi...@apache.org> on 2025/04/07 14:05:09 UTC
[I] Flexible handling of conflicts during JSON type inference [arrow]
edmcman opened a new issue, #46040:
URL: https://github.com/apache/arrow/issues/46040
### Describe the enhancement requested
JSON is extremely flexible, and arrow's [automatic type inference rules](https://arrow.apache.org/docs/python/json.html#automatic-type-inference) often fails to find a schema.
Here is a simple example on which `pyarrow.json.read_json` returns `ArrowInvalid: JSON parse error: Column(/lol/[]) changed from number to array in row 0`
```json
{ "lol": [42, []]}
```
I'm attempting to help solve [this issue](https://github.com/huggingface/datasets/issues/5950) which is basically to cope with such "unstructured" JSON formats. AFAICT Arrow doesn't make this easy, because parsing/conversion simply fails without an obvious way to move forward. It would be nice if there was a way to proceed even if type inference fails. Here are a few ideas:
* If type inference fails, optionally use a general container type (`JsonType`?) instead. This could perhaps be a ParseOption passed to `read_json`.
* Allow the user to augment (e.g., a callback) the type inference process, especially when it would otherwise fail. Again, this could be a ParseOption.
* Expose the type inference algorithm as a separate function, with a way to explicitly indicate failures in the schema. In other words, don't throw an exception, but rather mark that part of the schema in some way as needing additional input from the caller.
If I missed an obvious way of coping with these unstructured JSON files, please let me know!
### Component(s)
Python, Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Flexible handling of conflicts during JSON type inference [arrow]
Posted by "edmcman (via GitHub)" <gi...@apache.org>.
edmcman commented on issue #46040:
URL: https://github.com/apache/arrow/issues/46040#issuecomment-3176394622
I'm not working on it due to time constraints. But I think it's a great idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Flexible handling of conflicts during JSON type inference [arrow]
Posted by "antoniobadia (via GitHub)" <gi...@apache.org>.
antoniobadia commented on issue #46040:
URL: https://github.com/apache/arrow/issues/46040#issuecomment-5283267935
I think this issue is too important to leave unattended; unfortunately, I can't currently address it myself. I'd like to add the 'Status: needs champion' label, but I don't think I have the permissions (no labels option). If anyone with access agrees, please do. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Flexible handling of conflicts during JSON type inference [arrow]
Posted by "antoniobadia (via GitHub)" <gi...@apache.org>.
antoniobadia commented on issue #46040:
URL: https://github.com/apache/arrow/issues/46040#issuecomment-3176227414
Arrow cannot parse JSON correctly when the conversion to column fails. That is, it expects every path in JSON to be associated to a unique data type. It fails in all the following (examples produced with pyarrow):
"name": {
"firstName": "Duckota",
"lastName": "Fanning"
},
...
"name": "Jim Jones",
#pyarrow.lib.ArrowInvalid: JSON parse error: Column(/name) changed from object to string in row 3
"dimensions": {
"height": "six foot",
"weight": "165 lbs"
},
...
"dimensions": {
"height": 6.2,
"weight": 185
},
#pyarrow.lib.ArrowInvalid: JSON parse error: Column(/dimensions/height) changed from string to number in row 3
{"a":1,"b":"foo"}
{"a":2,"b":"bar"}
{"a":3, "c":{"d":4, "e":5}}
#pyarrow.lib.ArrowInvalid: JSON parse error: Column(/d/[]) changed from number to string in row 3
{"a":11, "d":[12, "h"]}
{"a":6, "d":[7, 8, 9]}
#pyarrow.lib.ArrowInvalid: JSON parse error: Column(/d/[]) changed from number to string in row 4
{"a":10, "d":["f", "g"]}
This may be a limitation made on purpose because of the difficulties of transforming into columnar storage. I'm currently working on a project to take care of this by using UNIONs automatically in such cases. Please let me know if this is already being done/addressed (or if you are interested in joining the project)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Flexible handling of conflicts during JSON type inference [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46040:
URL: https://github.com/apache/arrow/issues/46040#issuecomment-5266105003
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org