You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "rdtr (via GitHub)" <gi...@apache.org> on 2026/10/13 08:02:20 UTC

[PR] [GLUTEN-7628][VL] Support monotonically_increasing_id function [gluten]

rdtr opened a new pull request, #13268:
URL: https://github.com/apache/gluten/pull/13268

   ## What changes are proposed in this pull request?
   
   This is a new version of #11674, which was closed as stale without a review.
   
   `monotonically_increasing_id` falls back to vanilla Spark in the Velox backend. Velox has a Spark implementation, and Gluten already passes the partition ID it uses (`SparkQueryConfig::kPartitionId`), but the function is not mapped. An earlier attempt (#10097) was closed because of a result mismatch (#7628): with two calls in one query, such as `SELECT monotonically_increasing_id(), monotonically_increasing_id()`, the second column returned 2, 3, ... instead of 0, 1, .... Velox deduplicated the two identical calls into one, so they shared one counter.
   
   Velox has since added `expression.dedup_non_deterministic` (facebookincubator/velox#15008) to turn this off. This PR:
   
   - Maps `MonotonicallyIncreasingID` in the Velox backend's expression mappings.
   - Sets `expression.dedup_non_deterministic` to `false`. Spark keeps each call of a non-deterministic function separate, with its own state. Deterministic expressions are still deduplicated.
   - Re-enables the `monotonically_increasing_id` test in `ScalarFunctionsValidateSuite`, fixes its name, and adds a query with two calls.
   
   Question for reviewers: the config is set for all queries, not only the ones that use `monotonically_increasing_id`. I think that matches Spark, which never merges non-deterministic calls, but please let me know if you'd prefer to limit it.
   
   Closes #7628.
   
   ## How was this patch tested?
   
   Re-enabled and extended the `monotonically_increasing_id` test in `ScalarFunctionsValidateSuite`.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7628][VL] Support monotonically_increasing_id function [gluten]

Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he merged PR #13268:
URL: https://github.com/apache/gluten/pull/13268


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org