You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "PHILO-HE (via GitHub)" <gi...@apache.org> on 2025/07/07 08:25:47 UTC
[I] [VL] Add ANSI mode support [incubator-gluten]
PHILO-HE opened a new issue, #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134
### Description
In this issue, we will list and track all tasks for ANSI mode support.
There are two Spark configurations directly related to ANSI usage.
* `spark.sql.ansi.enabled` (default is true since Spark 4.0)
* `spark.sql.storeAssignmentPolicy` (default is ANSI since Spark 3.0)
See Spark ANSI compliance: https://github.com/apache/spark/blob/v4.0.0/docs/sql-ref-ansi-compliance.md
Related discussion: https://github.com/apache/incubator-gluten/issues/4740.
### Gluten version
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3051449821
@PHILO-HE are you going to work on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "minni31 (via GitHub)" <gi...@apache.org>.
minni31 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-4251701173
Hello @philo-he I raised a PR for cast NumericType to long/int/short/byte https://github.com/facebookincubator/velox/pull/16962
Please help with reviw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "infvg (via GitHub)" <gi...@apache.org>.
infvg commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3767532822
Raised a PR for `cast string to timestamp`:
https://github.com/facebookincubator/velox/pull/16058/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3051494254
> [@PHILO-HE](https://github.com/PHILO-HE) are you going to work on this?
@FelixYBW, some developers in the community are interested in this work. I will work with them. I assume it requires a lot of small changes distributed in multiple files in Velox.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-4229851939
GlutenCastWithAnsiOnSuite has 4 ANSI Cast failures that are already fixed by this issue. No additional work needed for ANSI mode under the Spark 4.x tracking effort.
Linking to the Spark 4.x feature tracking issue #11910 as parent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "ReemaAlzaid (via GitHub)" <gi...@apache.org>.
ReemaAlzaid commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-4397969545
I opened a pr for this `STORE_ASSIGNMENT_POLICY defaults to ANSI` https://github.com/apache/gluten/pull/12051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "Smallfu666 (via GitHub)" <gi...@apache.org>.
Smallfu666 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-5215966687
Yes, it came up in the ANSI validate suite (`ArithmeticAnsiValidateSuite`, run with ansiFallback disabled), not on any production path. Agreed it's benign — no need for a separate sub-issue.
I'll keep the regression lenient: it asserts the overflow still throws and that the query stays on `ProjectExecTransformer` (native checked arithmetic actually runs), and on Spark 4.0 it intercepts `SparkException` rather than requiring `ArithmeticException` in the cause chain. That passes on both 4.0 and 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "minni31 (via GitHub)" <gi...@apache.org>.
minni31 commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3988881023
Hey @PHILO-HE, I'll be picking up cast NumericType to long/int/short/byte. Looking forward to making my first contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3853450258
PR raised for `cast string to long/int/short/byte`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "ReemaAlzaid (via GitHub)" <gi...@apache.org>.
ReemaAlzaid commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-6269206822
Hi @philo-he, I picked up the decimal pmod item from this issue since the previous [Velox PR #16406](https://github.com/facebookincubator/velox/pull/16406) was closed.
I opened [Velox PR #19522](https://github.com/facebookincubator/velox/pull/19522) to add decimal pmod and checked_pmod, and have addressed the review feedback. Gluten integration will follow separately.
Could you update the tracking link and take a look when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3051512350
Yes, several customer required it. I added to project track. Could you update the status weekly or bi-weekly?
We may need to enable the ANSI UT firstly and list the failure cases. Also create the sub-issue for work to do. I think some of work is already done in velox.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3052136260
> Yes, several customer required it. I added to project track. Could you update the status weekly or bi-weekly?
>
> We may need to enable the ANSI UT firstly and list the failure cases. Also create the sub-issue for work to do. I think some of work is already done in velox.
@FelixYBW, I will update the status bi-weekly and will also create some subtasks. Yes, there are some relevant work already done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "Smallfu666 (via GitHub)" <gi...@apache.org>.
Smallfu666 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-5083248428
Hi @philo-he , I reproduced a remaining ANSI exception-mapping issue on current main (8b701473f) with Spark 4.0 and fallback disabled.
SELECT 2147483647 + 1 is confirmed offloaded to Velox and checked_add detects the overflow, but the cause chain becomes:
SparkException -> GlutenException -> GlutenException
This appears to be a JNI/Java exception-translation issue rather than missing arithmetic overflow support. Would you prefer a separate sub-issue for this? I can take it and add focused regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3784108025
I am picking up `Cast string to date`
@PHILO-HE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3796194592
Picking up `Cast string to time`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "minni31 (via GitHub)" <gi...@apache.org>.
minni31 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-4074146276
Sure, thanks @malinjawi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "minni31 (via GitHub)" <gi...@apache.org>.
minni31 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-4251704627
I'm picking up cast string to double/float.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "Harthi7 (via GitHub)" <gi...@apache.org>.
Harthi7 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-4231543973
Hello @philo-he I raised a PR for `cast string to timestamp` https://github.com/facebookincubator/velox/pull/17102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3753604735
Hey @PHILO-HE @FelixYBW I will be picking up `cast string to boolean`. Looking forward to making my first contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3765063742
Hey @PHILO-HE, I opened [PR #11437](https://github.com/apache/incubator-gluten/pull/11437) for Spark ANSI cast string ti boolean. I implemented it in Gluten by routing ANSI casts to a custom function and added tests. Would you prefer this to live in Gluten, or should we move the behavior into Velox’s Spark CAST (so ANSI is handled in the velox repo)? I’m happy to refactor based on your guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3866879359
[PR](https://github.com/facebookincubator/velox/pull/16092) was merged for `Validate Spark string-to-date cast`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3889357953
Picking up `Arithmetic Functions (ANSI Overflow Check)` @PHILO-HE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "nimesh1601 (via GitHub)" <gi...@apache.org>.
nimesh1601 commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3051570813
@PHILO-HE We at Uber also require this support, so feel free to include me too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-5114731085
> Hi [@philo-he](https://github.com/philo-he), I reproduced a narrow Spark 4.0 exception-chain mismatch on current main ([8b70147](https://github.com/apache/gluten/commit/8b701473f465e4d7eb05335b0529bad9327db8b3)) with ANSI fallback disabled. For SELECT 2147483647 + 1, execution is confirmed through ProjectExecTransformer, and the native log shows checked_add(2147483647, 1). On Spark 4.0, the resulting cause chain is: SparkException -> GlutenException -> GlutenException with no ArithmeticException. The same regression test passes on Spark 4.1.1, so this appears specific to the Spark 4.0 exception propagation path rather than missing arithmetic implementation. Would you prefer a separate sub-issue for this Spark 4.0 compatibility gap? I can take it and add focused regression coverage.
@Smallfu666, thanks for the feedback. Did this come up in the Gluten tests? Since it looks benign, could we ignore the Spark 4.0 exception-chain mismatch and just adjust the test to pass?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-4014006273
Hey @minni31 — happy to see your interest in this issue!
Just a heads-up: I opened a draft PR for NumericType → long/int/short/byte here a while ago:
https://github.com/facebookincubator/velox/pull/16341
If it helps with your first contribution, feel free to pick it up from where I left off. I'm also happy to assist if you have any questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "infvg (via GitHub)" <gi...@apache.org>.
infvg commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3772692261
Raised a PR for ``cast timestamp to int``:
https://github.com/facebookincubator/velox/pull/16061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "Mariamalmesfer (via GitHub)" <gi...@apache.org>.
Mariamalmesfer commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3783983613
Hi @PHILO-HE, I’m planning to pick up [Cast decimal to string](https://github.com/apache/incubator-gluten/issues/10134#:~:text=Cast%20decimal%20to,expressions/Cast.scala#L678). PR coming soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [incubator-gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on issue #10134:
URL: https://github.com/apache/incubator-gluten/issues/10134#issuecomment-3851695388
@PHILO-HE `cast string to boolean` [PR](https://github.com/facebookincubator/velox/pull/16059) was merged we can mark as complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "Smallfu666 (via GitHub)" <gi...@apache.org>.
Smallfu666 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-5083002678
Hi @philo-he, I reproduced a narrow remaining part of this tracker on current main (`8b701473f4654e7eb05335b0529bad9327db8b3`) with Spark 4.0/4.1 and ANSI fallback disabled.
For `SELECT 2147483647 + 1`, the expression is executed by `ProjectExecTransformer` (native Velox execution confirmed via `Top-level Expression: checked_add(2147483647:INTEGER, 1:INTEGER)`), but the exception cause chain is:
```
org.apache.spark.SparkException
-> org.apache.gluten.exception.GlutenException
-> org.apache.gluten.exception.GlutenException
```
Vanilla Spark exposes an `ArithmeticException` (or its Spark subclass) in the cause chain. Gluten currently maps every native `std::exception` to `GlutenException` via `JNI_METHOD_END` in `cpp/core/jni/JniError.h`, so the existing Spark 4.x test only intercepting the outer `SparkException` does not detect this mismatch.
Would you prefer a separate sub-issue for a narrowly scoped fix that maps only Velox `ARITHMETIC_ERROR` to Java `ArithmeticException`, while preserving `GlutenException` for all other errors? I would cover Spark 4.0/4.1, ANSI on/off, TRY semantics, cause-chain type, and native offload.
Confirmed reproduction on both Spark 4.0 and 4.1 with the same exception chain. This is only about the exception class mapping — I am not claiming the full 24-item WRONG_EXCEPTION list from historical PR #11975 still exists on current main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "Harthi7 (via GitHub)" <gi...@apache.org>.
Harthi7 commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-4263752141
I picked up `cast float/double to timestamp` and raised a PR https://github.com/facebookincubator/velox/pull/17219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Add ANSI mode support [gluten]
Posted by "rdtr (via GitHub)" <gi...@apache.org>.
rdtr commented on issue #10134:
URL: https://github.com/apache/gluten/issues/10134#issuecomment-5967065903
@PHILO-HE I went through the remaining items. A few can be checked off, could you update the checklist?
Can be checked off:
- Abs: done in https://github.com/facebookincubator/velox/pull/14346 (merged 2025-09-18). Spark's `abs` throws on integer overflow in ANSI mode, and the ANSI CI has no `abs` failures.
- VAR_POP, VAR_SAMP, STDDEV_POP, STDDEV_SAMP: these have no ANSI behavior in Spark. Their final divisions are guarded by `If(n === 0.0, ...)`, and the result for a zero divisor (NULL or NaN) depends on `spark.sql.legacy.statisticalAggregate`, which Gluten already passes to Velox.
- RoundCeil, RoundFloor: no ANSI behavior either (`RoundBase.ansiEnabled` is false; only `Round` and `BRound` override it). Offloading them is in progress in #12777.
I've also proposed one direction for passing the ANSI mode to Velox in https://github.com/facebookincubator/velox/issues/3869, since several of these PRs depend on it. Please check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org