You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "XiDuo You (Jira)" <ji...@apache.org> on 2022/05/27 08:51:00 UTC
[jira] [Updated] (SPARK-39316) Merge PromotePrecision and CheckOverflow into decimal binary arithmetic
[ https://issues.apache.org/jira/browse/SPARK-39316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
XiDuo You updated SPARK-39316:
------------------------------
Description:
Merge {{PromotePrecision}} into {{{}dataType{}}}, for example, {{{}Add{}}}:
{code:java}
override def dataType: DataType = (left, right) match {
case (DecimalType.Expression(p1, s1), DecimalType.Expression(p2, s2)) =>
val resultScale = max(s1, s2)
if (allowPrecisionLoss) {
DecimalType.adjustPrecisionScale(max(p1 - s1, p2 - s2) + resultScale + 1,
resultScale)
} else {
DecimalType.bounded(max(p1 - s1, p2 - s2) + resultScale + 1, resultScale)
}
case _ => super.dataType
} {code}
Merge {{{}CheckOverflow{}}}, for example, {{Add}} eval:
{code:java}
dataType match {
case decimalType: DecimalType =>
val value = numeric.plus(input1, input2)
checkOverflow(value.asInstanceOf[Decimal], decimalType)
...
} {code}
was:
Merge `PromotePrecision` into `dataType`, so every arithmetic should report the accurate decimal type.
For example, `Add`:
{code:java}
override def dataType: DataType = (left, right) match {
case (DecimalType.Expression(p1, s1), DecimalType.Expression(p2, s2)) =>
val resultScale = max(s1, s2)
if (allowPrecisionLoss) {
DecimalType.adjustPrecisionScale(max(p1 - s1, p2 - s2) + resultScale + 1,
resultScale)
} else {
DecimalType.bounded(max(p1 - s1, p2 - s2) + resultScale + 1, resultScale)
}
case _ => super.dataType
} {code}
Merge `CheckOverflow` into eval and code-gen code path, so every arithmetic can handle the overflow case during runtime.
For example, `Add`:
{code:java}
dataType match {
case decimalType: DecimalType =>
val value = numeric.plus(input1, input2)
checkOverflow(value.asInstanceOf[Decimal], decimalType)
...
} {code}
> Merge PromotePrecision and CheckOverflow into decimal binary arithmetic
> -----------------------------------------------------------------------
>
> Key: SPARK-39316
> URL: https://issues.apache.org/jira/browse/SPARK-39316
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: XiDuo You
> Priority: Major
>
> Merge {{PromotePrecision}} into {{{}dataType{}}}, for example, {{{}Add{}}}:
> {code:java}
> override def dataType: DataType = (left, right) match {
> case (DecimalType.Expression(p1, s1), DecimalType.Expression(p2, s2)) =>
> val resultScale = max(s1, s2)
> if (allowPrecisionLoss) {
> DecimalType.adjustPrecisionScale(max(p1 - s1, p2 - s2) + resultScale + 1,
> resultScale)
> } else {
> DecimalType.bounded(max(p1 - s1, p2 - s2) + resultScale + 1, resultScale)
> }
> case _ => super.dataType
> } {code}
> Merge {{{}CheckOverflow{}}}, for example, {{Add}} eval:
> {code:java}
> dataType match {
> case decimalType: DecimalType =>
> val value = numeric.plus(input1, input2)
> checkOverflow(value.asInstanceOf[Decimal], decimalType)
> ...
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org