You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/17 13:11:37 UTC
[GitHub] [spark] ulysses-you opened a new pull request, #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
ulysses-you opened a new pull request, #37207:
URL: https://github.com/apache/spark/pull/37207
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add a new expression `DecimalDivideWithOverflowCheck` to replace the previous CheckOverflowInSum + Divide + Cast.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
If the result data type is decimal, the Average will first calculate the result using the default precison and scale of divide, then cast to the result data type. We should do calculate and return the result data type directly so that we can avoid the precision loss. It can also save one unnecessary cast.
And for the overflow check, we should check the result of divide whether overflow instead of the dividend.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
yes, a small bug fix
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
add a test and fix test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r924039193
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala:
##########
@@ -247,3 +247,77 @@ case class DecimalAddNoOverflowCheck(
newLeft: Expression, newRight: Expression): DecimalAddNoOverflowCheck =
copy(left = newLeft, right = newRight)
}
+
+/**
+ * A divide expression for decimal values which is only used internally by Avg.
+ *
+ * It will fail when nullOnOverflow is false follows:
+ * - left (sum in avg) is null due to over the max precision 38,
+ * the right (count in avg) should never be null
+ * - the result of divide is overflow
+ */
+case class DecimalDivideWithOverflowCheck(
+ left: Expression,
+ right: Expression,
+ override val dataType: DecimalType,
+ avgQueryContext: String,
+ nullOnOverflow: Boolean) extends BinaryExpression with SupportQueryContext {
+ override def nullable: Boolean = nullOnOverflow
+ override def initQueryContext(): String = avgQueryContext
+ def decimalMethod: String = "$div"
+
+ override def eval(input: InternalRow): Any = {
+ val value1 = left.eval(input)
+ if (value1 == null) {
+ if (nullOnOverflow) {
+ null
+ } else {
+ throw QueryExecutionErrors.overflowInSumOfDecimalError(queryContext)
+ }
+ } else {
+ val value2 = right.eval(input)
+ dataType.fractional.asInstanceOf[Fractional[Any]].div(value1, value2).asInstanceOf[Decimal]
+ .toPrecision(dataType.precision, dataType.scale, Decimal.ROUND_HALF_UP, nullOnOverflow,
+ queryContext)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val errorContextCode = if (nullOnOverflow) {
+ "\"\""
+ } else {
+ ctx.addReferenceObj("errCtx", queryContext)
+ }
+ val nullHandling = if (nullOnOverflow) {
+ ""
+ } else {
+ s"throw QueryExecutionErrors.overflowInSumOfDecimalError($errorContextCode);"
+ }
+
+ val eval1 = left.genCode(ctx)
+ val eval2 = right.genCode(ctx)
+
+ // scalastyle:off line.size.limit
+ val code =
+ code"""
+ |${eval1.code}
+ |${eval2.code}
+ |boolean ${ev.isNull} = ${eval1.isNull};
+ |${CodeGenerator.javaType(dataType)} ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
+ |if (${eval1.isNull}) {
+ | $nullHandling
+ |} else {
+ | ${ev.value} = ${eval1.value}.$decimalMethod(${eval2.value}).toPrecision(
Review Comment:
sorry, not quite follow the suggestion. This expression is internally used only by average so
- no caller side will invoke this checkInputDataTypes method
- the only chance it can fail is we write a bug code at average
but if you think it's necessary, I can add some code like:
```
override def checkInputDataTypes(): TypeCheckResult = (left.dataType, right.dataType) match {
case (l: DecimalType, r: DecimalType) if inputType.acceptsType(l) && inputType.acceptsType(r) =>
TypeCheckResult.TypeCheckSuccess
case _ => throw new IllegalStateException("Both left and right child must be decimal type.")
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #37207:
URL: https://github.com/apache/spark/pull/37207#issuecomment-1189711015
thank you @gengliangwang and @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r924019925
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala:
##########
@@ -88,9 +88,14 @@ abstract class AverageBase
// We can't directly use `/` as it throws an exception under ansi mode.
protected def getEvaluateExpression(queryContext: String) = child.dataType match {
case _: DecimalType =>
- Divide(
- CheckOverflowInSum(sum, sumDataType.asInstanceOf[DecimalType], !useAnsiAdd, queryContext),
- count.cast(DecimalType.LongDecimal), failOnError = false).cast(resultType)
Review Comment:
@ulysses-you I mean, the `cast` here will check the overflow of the division result under ANSI mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #37207:
URL: https://github.com/apache/spark/pull/37207#issuecomment-1188527477
thank you @gengliangwang for the review. For your concerns:
> I think the issue is that the expression Divide allows precision loss by default when handling decimals. Setting the flag spark.sql.decimalOperations.allowPrecisionLoss as false will produce the same result in the test case https://github.com/apache/spark/pull/37207/files#diff-450322f1656760b5126d42d2334685a9b488540a267c1557c82f8e01f723a672R1026. Thus the current behavior is sort of expected, it is consistent with the result of dividing a decimal.
This main issue is, in average evaluation we change the precison twice. The first time we change the result to the precision of divide which apply the config you point out. The second time we change the result to the precison of average which is different with divide. To solve it, this pr added a new decimal divide to avoid the first precision change. I'm not sure the precison of average should be same with divide, if so the result decimal type of average is wrong.
> Besides, if we have to change the behavior of Average, we can consider having a decimal dividing expression that doesn't lose precision instead.
That's what this pr did. This pr combines the decimal devide and overflow checking into one expression.
> I think that's what the current code did. If there is a difference after this PR, please create a new test case.
no, the current code checks the sum value if is overflow and this pr changes to check the result of average if is overflow. The sum data type is `DecimalType.bounded(p + 10, s)` which means it will overflow with almost 1 billion rows, so sorry for that it really hard to add the test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r923093897
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala:
##########
@@ -247,3 +247,76 @@ case class DecimalAddNoOverflowCheck(
newLeft: Expression, newRight: Expression): DecimalAddNoOverflowCheck =
copy(left = newLeft, right = newRight)
}
+
+/**
+ * A divide expression for decimal values which is only used internally by Avg.
+ *
+ * It will fail when nullOnOverflow is false follows:
+ * - left is null, as the right side should never be null
Review Comment:
@cloud-fan in fact, it will produce null value if sum overflow. We do not check the overflow during merging, but the max precison of Spark is limited by 38.. which means the sum value may over 1E38BD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #37207:
URL: https://github.com/apache/spark/pull/37207#issuecomment-1186691951
cc @gengliangwang @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r924028738
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala:
##########
@@ -88,9 +88,14 @@ abstract class AverageBase
// We can't directly use `/` as it throws an exception under ansi mode.
protected def getEvaluateExpression(queryContext: String) = child.dataType match {
case _: DecimalType =>
- Divide(
- CheckOverflowInSum(sum, sumDataType.asInstanceOf[DecimalType], !useAnsiAdd, queryContext),
- count.cast(DecimalType.LongDecimal), failOnError = false).cast(resultType)
Review Comment:
I see, then it's also a unnecessary overflow check for average before this pr. if sum is not overflow, sum / count will never overflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r924040583
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala:
##########
@@ -247,3 +247,77 @@ case class DecimalAddNoOverflowCheck(
newLeft: Expression, newRight: Expression): DecimalAddNoOverflowCheck =
copy(left = newLeft, right = newRight)
}
+
+/**
+ * A divide expression for decimal values which is only used internally by Avg.
+ *
+ * It will fail when nullOnOverflow is false follows:
+ * - left (sum in avg) is null due to over the max precision 38,
+ * the right (count in avg) should never be null
+ * - the result of divide is overflow
+ */
+case class DecimalDivideWithOverflowCheck(
+ left: Expression,
+ right: Expression,
+ override val dataType: DecimalType,
+ avgQueryContext: String,
+ nullOnOverflow: Boolean) extends BinaryExpression with SupportQueryContext {
+ override def nullable: Boolean = nullOnOverflow
+ override def initQueryContext(): String = avgQueryContext
+ def decimalMethod: String = "$div"
+
+ override def eval(input: InternalRow): Any = {
+ val value1 = left.eval(input)
+ if (value1 == null) {
+ if (nullOnOverflow) {
+ null
+ } else {
+ throw QueryExecutionErrors.overflowInSumOfDecimalError(queryContext)
+ }
+ } else {
+ val value2 = right.eval(input)
+ dataType.fractional.asInstanceOf[Fractional[Any]].div(value1, value2).asInstanceOf[Decimal]
+ .toPrecision(dataType.precision, dataType.scale, Decimal.ROUND_HALF_UP, nullOnOverflow,
+ queryContext)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val errorContextCode = if (nullOnOverflow) {
+ "\"\""
+ } else {
+ ctx.addReferenceObj("errCtx", queryContext)
+ }
+ val nullHandling = if (nullOnOverflow) {
+ ""
+ } else {
+ s"throw QueryExecutionErrors.overflowInSumOfDecimalError($errorContextCode);"
+ }
+
+ val eval1 = left.genCode(ctx)
+ val eval2 = right.genCode(ctx)
+
+ // scalastyle:off line.size.limit
+ val code =
+ code"""
+ |${eval1.code}
+ |${eval2.code}
+ |boolean ${ev.isNull} = ${eval1.isNull};
+ |${CodeGenerator.javaType(dataType)} ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
+ |if (${eval1.isNull}) {
+ | $nullHandling
+ |} else {
+ | ${ev.value} = ${eval1.value}.$decimalMethod(${eval2.value}).toPrecision(
Review Comment:
Yes, I meant we should check the input so that the expression won't be used in the wrong place in future development.
The trait `ExpectsInputTypes` is an abstraction for the code you posted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r922980643
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -2166,7 +2166,7 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
df.queryExecution.optimizedPlan.collect {
case _: DataSourceV2ScanRelation =>
val expected_plan_fragment =
- "PushedAggregates: [SUM(PRICE), COUNT(PRICE)]"
+ "PushedAggregates: [COUNT(PRICE), SUM(PRICE)]"
Review Comment:
just for curiosity: what caused this different order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r924032530
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala:
##########
@@ -247,3 +247,77 @@ case class DecimalAddNoOverflowCheck(
newLeft: Expression, newRight: Expression): DecimalAddNoOverflowCheck =
copy(left = newLeft, right = newRight)
}
+
+/**
+ * A divide expression for decimal values which is only used internally by Avg.
+ *
+ * It will fail when nullOnOverflow is false follows:
+ * - left (sum in avg) is null due to over the max precision 38,
+ * the right (count in avg) should never be null
+ * - the result of divide is overflow
+ */
+case class DecimalDivideWithOverflowCheck(
+ left: Expression,
+ right: Expression,
+ override val dataType: DecimalType,
+ avgQueryContext: String,
+ nullOnOverflow: Boolean) extends BinaryExpression with SupportQueryContext {
+ override def nullable: Boolean = nullOnOverflow
+ override def initQueryContext(): String = avgQueryContext
+ def decimalMethod: String = "$div"
+
+ override def eval(input: InternalRow): Any = {
+ val value1 = left.eval(input)
+ if (value1 == null) {
+ if (nullOnOverflow) {
+ null
+ } else {
+ throw QueryExecutionErrors.overflowInSumOfDecimalError(queryContext)
+ }
+ } else {
+ val value2 = right.eval(input)
+ dataType.fractional.asInstanceOf[Fractional[Any]].div(value1, value2).asInstanceOf[Decimal]
+ .toPrecision(dataType.precision, dataType.scale, Decimal.ROUND_HALF_UP, nullOnOverflow,
+ queryContext)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val errorContextCode = if (nullOnOverflow) {
+ "\"\""
+ } else {
+ ctx.addReferenceObj("errCtx", queryContext)
+ }
+ val nullHandling = if (nullOnOverflow) {
+ ""
+ } else {
+ s"throw QueryExecutionErrors.overflowInSumOfDecimalError($errorContextCode);"
+ }
+
+ val eval1 = left.genCode(ctx)
+ val eval2 = right.genCode(ctx)
+
+ // scalastyle:off line.size.limit
+ val code =
+ code"""
+ |${eval1.code}
+ |${eval2.code}
+ |boolean ${ev.isNull} = ${eval1.isNull};
+ |${CodeGenerator.javaType(dataType)} ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
+ |if (${eval1.isNull}) {
+ | $nullHandling
+ |} else {
+ | ${ev.value} = ${eval1.value}.$decimalMethod(${eval2.value}).toPrecision(
Review Comment:
nit: I suggest extending `ExpectsInputTypes` for input data type check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #37207:
URL: https://github.com/apache/spark/pull/37207#issuecomment-1188390096
> If the result data type is decimal, the Average will first calculate the result using the default precison and scale of divide, then cast to the result data type. We should do calculate and return the result data type directly so that we can avoid the precision loss. It can also save one unnecessary cast.
@ulysses-you Actually, it took me some time to understand what the issue is from your statement. I think the issue is about the expression `Divide` allows precision loss by default when handling decimals. Setting the flag `spark.sql.decimalOperations.allowPrecisionLoss` as false will produce the same result as your PR.
Shall we consider having a decimal divide which doesn't lose precision instead?
> And for the overflow check, we should check the result of divide whether overflow instead of the dividend.
I think that's what the current code did. If there is a difference after this PR, please create a new test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r923008573
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala:
##########
@@ -247,3 +247,76 @@ case class DecimalAddNoOverflowCheck(
newLeft: Expression, newRight: Expression): DecimalAddNoOverflowCheck =
copy(left = newLeft, right = newRight)
}
+
+/**
+ * A divide expression for decimal values which is only used internally by Avg.
+ *
+ * It will fail when nullOnOverflow is false follows:
+ * - left is null, as the right side should never be null
Review Comment:
ah, yes. We skip the count == 0 and the merging expression do not check the overflow of sum. That said here the sum will never be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r923012513
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -2166,7 +2166,7 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
df.queryExecution.optimizedPlan.collect {
case _: DataSourceV2ScanRelation =>
val expected_plan_fragment =
- "PushedAggregates: [SUM(PRICE), COUNT(PRICE)]"
+ "PushedAggregates: [COUNT(PRICE), SUM(PRICE)]"
Review Comment:
Since `DecimalDivideWithOverflowCheck` is a dedicated expression for `AVG`, shall we put the `count == 0` check in it as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
URL: https://github.com/apache/spark/pull/37207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #37207:
URL: https://github.com/apache/spark/pull/37207#issuecomment-1188791526
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r924028738
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala:
##########
@@ -88,9 +88,14 @@ abstract class AverageBase
// We can't directly use `/` as it throws an exception under ansi mode.
protected def getEvaluateExpression(queryContext: String) = child.dataType match {
case _: DecimalType =>
- Divide(
- CheckOverflowInSum(sum, sumDataType.asInstanceOf[DecimalType], !useAnsiAdd, queryContext),
- count.cast(DecimalType.LongDecimal), failOnError = false).cast(resultType)
Review Comment:
I see, then it's also a unnecessary overflow check for average. if sum is not overflow, sum / count will never overflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #37207:
URL: https://github.com/apache/spark/pull/37207#issuecomment-1188546620
@ulysses-you I see, Thanks for the explanation. It makes sense to me now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r924043508
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala:
##########
@@ -247,3 +247,77 @@ case class DecimalAddNoOverflowCheck(
newLeft: Expression, newRight: Expression): DecimalAddNoOverflowCheck =
copy(left = newLeft, right = newRight)
}
+
+/**
+ * A divide expression for decimal values which is only used internally by Avg.
+ *
+ * It will fail when nullOnOverflow is false follows:
+ * - left (sum in avg) is null due to over the max precision 38,
+ * the right (count in avg) should never be null
+ * - the result of divide is overflow
+ */
+case class DecimalDivideWithOverflowCheck(
+ left: Expression,
+ right: Expression,
+ override val dataType: DecimalType,
+ avgQueryContext: String,
+ nullOnOverflow: Boolean) extends BinaryExpression with SupportQueryContext {
+ override def nullable: Boolean = nullOnOverflow
+ override def initQueryContext(): String = avgQueryContext
+ def decimalMethod: String = "$div"
+
+ override def eval(input: InternalRow): Any = {
+ val value1 = left.eval(input)
+ if (value1 == null) {
+ if (nullOnOverflow) {
+ null
+ } else {
+ throw QueryExecutionErrors.overflowInSumOfDecimalError(queryContext)
+ }
+ } else {
+ val value2 = right.eval(input)
+ dataType.fractional.asInstanceOf[Fractional[Any]].div(value1, value2).asInstanceOf[Decimal]
+ .toPrecision(dataType.precision, dataType.scale, Decimal.ROUND_HALF_UP, nullOnOverflow,
+ queryContext)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val errorContextCode = if (nullOnOverflow) {
+ "\"\""
+ } else {
+ ctx.addReferenceObj("errCtx", queryContext)
+ }
+ val nullHandling = if (nullOnOverflow) {
+ ""
+ } else {
+ s"throw QueryExecutionErrors.overflowInSumOfDecimalError($errorContextCode);"
+ }
+
+ val eval1 = left.genCode(ctx)
+ val eval2 = right.genCode(ctx)
+
+ // scalastyle:off line.size.limit
+ val code =
+ code"""
+ |${eval1.code}
+ |${eval2.code}
+ |boolean ${ev.isNull} = ${eval1.isNull};
+ |${CodeGenerator.javaType(dataType)} ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
+ |if (${eval1.isNull}) {
+ | $nullHandling
+ |} else {
+ | ${ev.value} = ${eval1.value}.$decimalMethod(${eval2.value}).toPrecision(
Review Comment:
got it ! addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r922979739
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala:
##########
@@ -247,3 +247,76 @@ case class DecimalAddNoOverflowCheck(
newLeft: Expression, newRight: Expression): DecimalAddNoOverflowCheck =
copy(left = newLeft, right = newRight)
}
+
+/**
+ * A divide expression for decimal values which is only used internally by Avg.
+ *
+ * It will fail when nullOnOverflow is false follows:
+ * - left is null, as the right side should never be null
Review Comment:
since we only invoke `DecimalDivideWithOverflowCheck` when count > 0, I think left (`sum` in AVG) will never be null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a diff in pull request #37207: [SPARK-39792][SQL] Add DecimalDivideWithOverflowCheck for decimal average
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #37207:
URL: https://github.com/apache/spark/pull/37207#discussion_r923007976
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -2166,7 +2166,7 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
df.queryExecution.optimizedPlan.collect {
case _: DataSourceV2ScanRelation =>
val expected_plan_fragment =
- "PushedAggregates: [SUM(PRICE), COUNT(PRICE)]"
+ "PushedAggregates: [COUNT(PRICE), SUM(PRICE)]"
Review Comment:
This pr makes the count ahead of sum, and the ordering is decided by this code:
https://github.com/apache/spark/blob/e4ca8424474e571d8e137388fe5d54732b68c2f3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala#L135-L147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org