You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2026/05/05 09:54:26 UTC
[PR] [VL] Fix json_tuple rewrite producing incompatible JSON path for Spark JVM fallback [gluten]
Zouxxyy opened a new pull request, #12038:
URL: https://github.com/apache/gluten/pull/12038
## What changes are proposed in this pull request?
When `json_tuple` is rewritten to `get_json_object` calls by `PullOutGenerateProjectHelper.pullOutPreProject`, the generated JSON path used bare bracket notation `$[key]` (without quotes). This works fine in Velox/simdjson, but when the expression falls back to Spark JVM execution (e.g., `get_json_object` is blacklisted or validation fails), Spark's `JsonPathParser` rejects `$[key]` and returns NULL.
This PR changes the generated path format from `$[key]` to `$['key']` (single-quoted bracket notation), which is accepted by both Velox/simdjson and Spark JVM's JsonPathParser.
**Root cause:**
- Velox's `JsonPathNormalizer` normalizes `$['key']` → `$[key]` internally for simdjson, so both forms work.
- Spark JVM only accepts `$.name` or `$['name']`, and rejects `$[name]` (bare brackets without quotes), returning NULL directly.
## How was this patch tested?
Added unit tests in `MiscOperatorSuite` covering:
- Basic single key extraction with fallback
- Dot-containing field names (e.g., `a.b`) — the core scenario for bracket notation
- Multiple keys extraction
- Non-existent keys returning null
- Mix of existing and non-existing keys
- NULL JSON input handling
## Was this patch authored or co-authored using generative AI tooling?
Yes
Generated-by: Qoder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Fix json_tuple rewrite producing incompatible JSON path in fallback scenarios [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he merged PR #12038:
URL: https://github.com/apache/gluten/pull/12038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Fix json_tuple rewrite producing incompatible JSON path for Spark JVM fallback [gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #12038:
URL: https://github.com/apache/gluten/pull/12038#issuecomment-4385627882
The failure ci should be unrelated, CC @lyy-pineapple @philo-he for a look thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Fix json_tuple rewrite producing incompatible JSON path for Spark JVM fallback [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12038:
URL: https://github.com/apache/gluten/pull/12038#issuecomment-4409807088
Thank you for the PR.
`MiscOperatorSuite.scala` generally holds some basic tests for expressions/operators. I would recommend to move the proposed test to a new test suite, named like `JsonTuplePathRewriteSuite.scala`. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org