You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "0x26res (via GitHub)" <gi...@apache.org> on 2024/12/05 10:56:07 UTC
[I] read_json error message not clear when duplicate field in schema [arrow]
0x26res opened a new issue, #44949:
URL: https://github.com/apache/arrow/issues/44949
### Describe the bug, including details regarding any error messages, version, and platform.
I had mistakenly put a duplicate field in my schema when calling `pyarrow.json.read_json` with `ParseOptions.explicit_schema`
I was either getting seg faults or an error message that wasn't clear.
Given the nature of json, there shouldn't be duplicate fields in the schema. `pyarrow.json.ParseOptions` should throw if there's any duplicate field in the `explicit_schema`.
I can send an MR, I'm just wondering if there's anywhere else where we do similar checks?
```python
import io
import pyarrow as pa
import pyarrow.json
SCHEMA = pa.schema(
[
pa.field("foo", pa.bool_()),
pa.field("foo", pa.bool_()),
]
)
with io.BytesIO(b'{"foo": true,"other": "bar"}') as buffer:
buffer.seek(0)
pyarrow.json.read_json(
buffer,
parse_options=pyarrow.json.ParseOptions(explicit_schema=SCHEMA),
)
```
```
pyarrow.json.read_json(
File "pyarrow/_json.pyx", line 308, in pyarrow._json.read_json
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Failed to convert JSON to bool from dictionary<values=string, indices=int32, ordered=0>
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2536701799
> Thanks for filing. A few thoughts:
>
> 1. If you are able to share a minimum reproduction of a segfault, please do and we can make a separate ticket for it.
I haven't been able to. I've seen it happen with my production data, but I haven't been able to reproduce.
> 2. Is it possible to just improve the error message? I'm not sure if it'd be best to have read_json validate the schema in this way.
I guess we can improve the message. But at the moment the error happens when parsing the data, I don't think there's a way to prevent the error without validating the schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2539245710
> Only when passed to read_json we should try to validate that fields are unique.
That seems pretty reasonable to me. Re: "I'm just wondering if there's anywhere else where we do similar checks?" I'm not sure. PR welcome but hopefully others can chime in here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2540826744
In general duplicate fields are indeed allowed in the arrow spec (although I am not sure we are handling it always nicely in pyarrow), but given the specific contex of json, I am personally fine with just checking for this in the json parser code (e.g. when constructing the options object when the `explicit_schema` is specified?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] read_json error message not clear when duplicate field in schema [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2532675339
Thanks for filing. A few thoughts:
1. If you are able to share a minimum reproduction of a segfault, please do and we can make a separate ticket for it.
2. Is it possible to just improve the error message? I'm not sure if it'd be best to have read_json validate the schema in this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2540835350
> I'd be curious what we do for parsers other than the JSON one in Arrow
In the case of other parsers, for example for CSV we do read files with duplicate field names correctly, but in this case of course the _format_ allows it (not that CSV is a well-defined format though ..).
It does seem that specifying a specific schema using `ConvertOptions(column_types=..)` with duplicate fields does not entirely work though, as it just takes the type of last occurrence in the schema for a given name (but it does not error or crash)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2538858953
I think a schema with a repeated field is valid, in the context of arrow.
But it's not valid in the context of json. Only when passed to read_json we should try to validate that fields are unique.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2540994469
Would it make sense to unify how we handle JSON and CSV parsing a bit and also check for schema/column duplicates in the csv parser code? Maybe to only add a warning in case columns are duplicated and `column_types` is defined in `ConvertOptions`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2541771747
From what @jorisvandenbossche is saying it looks like the behaviour, regarding duplicate field is different.
For `read_csv` it is allowed, though it looks like it will only work correctly if all field of the same name have got the same type.
For `read_json` it will fail.
In which case I don't think we should share the schema validation between read_json and read_csv.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] read_json error message not clear when duplicate field in schema [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #44949:
URL: https://github.com/apache/arrow/issues/44949#issuecomment-2536721845
The reason I brought up improving the message only is because I don't think we want to consider a schema with repeated field names invalid since I'm not sure it is. It happens to mix poorly with the JSON parser here which is probably the better place to make an improvement. I'd be curious what we do for parsers other than the JSON one in Arrow. cc @jorisvandenbossche @AlenkaF for thoughts here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org