You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "anshulbaliga7 (via GitHub)" <gi...@apache.org> on 2026/09/25 11:12:30 UTC

[PR] [VL] Enforce format_number restrictions on the JVM side [gluten]

anshulbaliga7 opened a new pull request, #13137:
URL: https://github.com/apache/gluten/pull/13137

   ## What changes are proposed in this pull request?
   
   `FormatNumberRestrictions` is registered in `ExpressionRestrictions.listAllRestrictions`, so the generated function support docs advertise that `format_number` falls back for `DecimalType` input and for a string format argument. Nothing enforced those restrictions on the JVM side, though: the expression went straight to `GenericExpressionTransformer` and only fell back because native validation found no matching Velox signature. The fallback reason was therefore a generic native validation failure rather than the documented sentence, and each such query paid a Substrait serialization and JNI validation round trip for a case known to be unsupported before planning started. It is the only entry in `listAllRestrictions` that is documented but not enforced.
   
   This PR routes `FormatNumber` through a new `genFormatNumberTransformer` hook on `SparkPlanExecApi`, following the pattern already used for `str_to_map`. The default returns `GenericExpressionTransformer`, so ClickHouse and Bolt are unaffected. The Velox override rejects `DecimalType` input and a string second argument with the
   matching `FormatNumberRestrictions` message via `GlutenExceptionUtil.throwsNotFullySupported`.
   
   ## How was this patch tested?
   
   Extended the `format_number` test in `ScalarFunctionsValidateSuite`: the two existing fallback cases now also assert that the fallback reason recorded on the executed plan contains the documented restriction, read from the fallback tag the same way `GlutenFallbackReporter` does. Existing offload cases are unchanged.
   
   Built locally with `mvn -Pbackends-velox -Pspark-3.5 -pl backends-velox -am -DskipTests clean test-compile`. The suite itself runs in the Velox backend CI jobs across Spark 3.4, 3.5, 4.0 and 4.1.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude claude-opus-4-8


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Enforce format_number restrictions on the JVM side [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13137:
URL: https://github.com/apache/gluten/pull/13137#issuecomment-5862862745

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Enforce format_number restrictions on the JVM side [gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan merged PR #13137:
URL: https://github.com/apache/gluten/pull/13137


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Enforce format_number restrictions on the JVM side [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13137:
URL: https://github.com/apache/gluten/pull/13137#issuecomment-5833150038

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Enforce format_number restrictions on the JVM side [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13137:
URL: https://github.com/apache/gluten/pull/13137#issuecomment-5831432365

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Enforce format_number restrictions on the JVM side [gluten]

Posted by "anshulbaliga7 (via GitHub)" <gi...@apache.org>.
anshulbaliga7 commented on PR #13137:
URL: https://github.com/apache/gluten/pull/13137#issuecomment-5831492572

   @yikf This completes the `FormatNumberRestrictions` you added in [#12754](https://github.com/apache/gluten/issues/12754). The restrictions were published in the generated function support docs but not enforced on the JVM side, so a decimal `format_number` only fell back via native validation with a generic reason. This routes `FormatNumber` through a backend hook, mirroring `str_to_map`, so the Velox backend rejects it up front with the documented message. Would appreciate a look when you have time. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org