You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (Jira)" <ji...@apache.org> on 2022/11/21 01:44:00 UTC
[jira] [Updated] (SPARK-41207) Regression in IntegralDivide
[ https://issues.apache.org/jira/browse/SPARK-41207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-41207:
--------------------------------
Fix Version/s: (was: 3.4.0)
> Regression in IntegralDivide
> ----------------------------
>
> Key: SPARK-41207
> URL: https://issues.apache.org/jira/browse/SPARK-41207
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Raza Jafri
> Priority: Minor
>
> There has been a regression in Integral Divide after the removal of PromotePrecision from Spark 3.4.0.
> {code:java}
> scala> val data = Seq(Row(BigDecimal("-7.70892"), BigDecimal("4.27138661282262736522411173299611831E+40")))
> scala> val simpleSchema = StructType(Array(
> | StructField("a", DecimalType(6,5),true),
> | StructField("b", DecimalType(36,-5), true)))
> scala> val df = spark.createDataFrame(spark.sparkContext.parallelize(data), simpleSchema)
> {code}
> The above statements result in an AnalysisException thrown
>
> {code:java}
> org.apache.spark.sql.AnalysisException: Decimal scale (0) cannot be greater than precision (-4).
> at org.apache.spark.sql.errors.QueryCompilationErrors$.decimalCannotGreaterThanPrecisionError(QueryCompilationErrors.scala:2237)
> at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:49)
> at org.apache.spark.sql.types.DecimalType$.bounded(DecimalType.scala:164)
> at org.apache.spark.sql.catalyst.expressions.IntegralDivide.resultDecimalType(arithmetic.scala:868)
> at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.dataType(arithmetic.scala:238)
> at org.apache.spark.sql.catalyst.expressions.IntegralDivide.org$apache$spark$sql$catalyst$expressions$DivModLike$$super$dataType(arithmetic.scala:842)
> {code}
> I believe this is happening because we aren't promoting the precision like we were before this [PR|https://github.com/apache/spark/commit/301a13963808d1ad44be5cacf0a20f65b853d5a2] went in. Without promoting precision the resultDecimalType in the example above tries to return a Decimal with precision of -4 and scale of 0 which is invalid
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org