You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "bersprockets (via GitHub)" <gi...@apache.org> on 2023/09/08 14:27:18 UTC
[GitHub] [spark] bersprockets opened a new pull request, #42857: [SPARK-45106][SQL] `PercentileCont` should check user supplied input
bersprockets opened a new pull request, #42857:
URL: https://github.com/apache/spark/pull/42857
### What changes were proposed in this pull request?
Change `PercentileCont` to explicitly check user-supplied input by calling `checkInputDataTypes` on the replacement.
### Why are the changes needed?
`PercentileCont` does not currently check the user's input. If the runtime replacement (an instance of `Percentile`) rejects the user's input, the runtime replacement ends up unresolved.
For example, this query throws an internal error rather than producing a useful error message:
```
select percentile_cont(b) WITHIN GROUP (ORDER BY a DESC) as x
from (values (12, 0.25), (13, 0.25), (22, 0.25)) as (a, b);
[INTERNAL_ERROR] Cannot resolve the runtime replaceable expression "percentile_cont(a, b)". The replacement is unresolved: "percentile(a, b, 1)".
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot resolve the runtime replaceable expression "percentile_cont(a, b)". The replacement is unresolved: "percentile(a, b, 1)".
at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$6(CheckAnalysis.scala:313)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$6$adapted(CheckAnalysis.scala:277)
...
```
With this PR, the above query will produce the following error message:
```
[DATATYPE_MISMATCH.NON_FOLDABLE_INPUT] Cannot resolve "percentile_cont(a, b)" due to data type mismatch: the input percentage should be a foldable "DOUBLE" expression; however, got "b".; line 1 pos 7;
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #42857: [SPARK-45106][SQL] `PercentileCont` should check user supplied input
Posted by "bersprockets (via GitHub)" <gi...@apache.org>.
bersprockets commented on PR #42857:
URL: https://github.com/apache/spark/pull/42857#issuecomment-1712060165
cc @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #42857: [SPARK-45106][SQL] `PercentileCont` should check user supplied input
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #42857:
URL: https://github.com/apache/spark/pull/42857#issuecomment-1712140666
Merged to master/3.5.
There is a conflict from branch-3.4. Could you make a PR for branch-3.4 and 3.3, please, @bersprockets ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #42857: [SPARK-45106][SQL] `PercentileCont` should check user supplied input
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #42857: [SPARK-45106][SQL] `PercentileCont` should check user supplied input
URL: https://github.com/apache/spark/pull/42857
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org