You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "shashbha14 (via GitHub)" <gi...@apache.org> on 2026/02/07 14:50:19 UTC
[PR] GH-49158: Allow type conversion in JSON parser when explicit schema is provided [arrow]
shashbha14 opened a new pull request, #49177:
URL: https://github.com/apache/arrow/pull/49177
Fixes #49158
The issue: when you provide an explicit schema to the JSON parser, it errors if JSON types don't exactly match schema types, even when conversion is straightforward.
For example, if you have:n
{"_id": "152934"}
{"_id": 152934}And your schema says `_id` should be string, it fails on row 1 with "Column changed from string to number" instead of converting 152934 to "152934".
I fixed this by making the parser attempt type conversion when an explicit schema is provided. Before erroring on a type mismatch, it checks if we have an explicit schema and tries to convert the value to match the expected type.
Changes:
- Store explicit_schema in HandlerBase so we can access it during parsing
- Modified AppendScalar() to try conversion before erroring when explicit schema exists
- Added TryConvertAndAppend() helper that handles the conversion logic
- Updated Bool() handler to also support conversion
- Added tests for number->string and string->number cases
Conversions that work now:
- Number -> String (152934 -> "152934")
- String -> Number (when the string is numeric)
- Boolean conversions to/from string and number
- Number -> Boolean (0 is false, non-zero is true)
This only happens when explicit schema is provided, so it's backward compatible. All existing tests still pass.
Fixes #49158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #49177:
URL: https://github.com/apache/arrow/pull/49177#issuecomment-3883605300
Thanks for replying @shashbha14! Feel free to propose a PR when we have consensus in the discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #49177:
URL: https://github.com/apache/arrow/pull/49177#issuecomment-3878956583
@shashbha14 closing this now for inactivity. Feel free to reopen this PR and engage us in review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #49177:
URL: https://github.com/apache/arrow/pull/49177#issuecomment-3878474863
Since @shashbha14 seems inactive we best close this and leave the issue unassigned for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "shashbha14 (via GitHub)" <gi...@apache.org>.
shashbha14 commented on PR #49177:
URL: https://github.com/apache/arrow/pull/49177#issuecomment-3883539575
I had an initial PR (#49177) for this but it mixed in unrelated commits and the
JSON parser behavior is still being discussed. I’m happy to revisit this once
there’s consensus on how coercion with explicit schema should behave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #49177:
URL: https://github.com/apache/arrow/pull/49177#issuecomment-3865125390
Thanks for jumping on this @shashbha14. However please note we might not want to change the json parser right now depending of the outcome of discussion on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok closed pull request #49177: GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided
URL: https://github.com/apache/arrow/pull/49177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #49177:
URL: https://github.com/apache/arrow/pull/49177#issuecomment-3878912317
I agree with you. I've removed the assignee from the issue. Happy to close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49158: [Python] Allow type conversion in JSON parser when explicit schema is provided [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #49177:
URL: https://github.com/apache/arrow/pull/49177#discussion_r2784888177
##########
python/pyarrow/table.pxi:
##########
@@ -1934,6 +1936,9 @@ cdef class _Tabular(_PandasConvertible):
If not passed, will be inferred from the Mapping values.
metadata : dict or Mapping, default None
Optional metadata for the schema (if inferred).
+ personal_data : bool, default None
+ Whether the table/batch contains personal data. If True, adds
+ b'ARROW:personal_data': b'true' to the metadata.
Review Comment:
Out of curiosity, why is `personal_data` needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org