You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "n0r0shi (via GitHub)" <gi...@apache.org> on 2026/02/28 10:49:39 UTC
[PR] [GLUTEN-7600][VL] Add monotonically_increasing_id function mapping [incubator-gluten]
n0r0shi opened a new pull request, #11674:
URL: https://github.com/apache/incubator-gluten/pull/11674
## Summary
- Adds `Sig[MonotonicallyIncreasingID]` to `ExpressionMappings.SCALAR_SIGS` so the function is offloaded to Velox instead of falling back to vanilla Spark.
- Sets Velox's `expression.dedup_non_deterministic` to `false` to match Spark semantics — Spark never deduplicates non-deterministic expressions, each call has independent state.
- Un-ignores and fixes the test in `ScalarFunctionsValidateSuite`.
## Context
PR #10097 previously attempted this but was closed because of a result mismatch (#7628): `SELECT monotonically_increasing_id(), monotonically_increasing_id()` returned
```
┌─────┬───────┬───────┐
│ Row │ Col 1 │ Col 2 │
├─────┼───────┼───────┤
│ 0 │ 0 │ 2 │
├─────┼───────┼───────┤
│ 1 │ 1 │ 3 │
└─────┴───────┴───────┘
```
instead of Spark's expected
```
┌─────┬───────┬───────┐
│ Row │ Col 1 │ Col 2 │
├─────┼───────┼───────┤
│ 0 │ 0 │ 0 │
├─────┼───────┼───────┤
│ 1 │ 1 │ 1 │
└─────┴───────┴───────┘
```
The root cause was Velox's expression compiler deduplicating the two structurally identical calls into one shared counter instance.
Velox has since added the `expression.dedup_non_deterministic` config (facebookincubator/velox#15008) to control this behavior. This PR sets it to `false` for Gluten. This only affects non-deterministic expressions — deterministic expression deduplication is unchanged.
**Question for reviewers:** Is setting `expression.dedup_non_deterministic = false` globally the right approach? An alternative would be conditionally disabling it only when stateful expressions like `monotonically_increasing_id` are detected in the plan, but we believe the global approach is correct since Spark semantics never deduplicate non-deterministic expressions.
Closes #7628
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7600][VL] Add monotonically_increasing_id function mapping [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11674: [GLUTEN-7600][VL] Add monotonically_increasing_id function mapping
URL: https://github.com/apache/gluten/pull/11674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7600][VL] Add monotonically_increasing_id function mapping [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11674:
URL: https://github.com/apache/gluten/pull/11674#issuecomment-4340371303
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7600][VL] Add monotonically_increasing_id function mapping [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11674:
URL: https://github.com/apache/incubator-gluten/pull/11674#issuecomment-3976991168
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7600][VL] Add monotonically_increasing_id function mapping [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11674:
URL: https://github.com/apache/gluten/pull/11674#issuecomment-4274989371
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org