You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Philip Dodds (JIRA)" <ji...@apache.org> on 2015/12/03 15:32:10 UTC
[jira] [Commented] (SPARK-12128) Multiplication on decimals in
dataframe returns null
[ https://issues.apache.org/jira/browse/SPARK-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037839#comment-15037839 ]
Philip Dodds commented on SPARK-12128:
--------------------------------------
I tried to add a testcase, but when I do it using a simplified approach
{code}
test("SPARK-12128: Multiplication of decimals in dataframe returning null") {
withTempTable("t") {
Seq((Decimal(2), Decimal(2)), (Decimal(3), Decimal(3))).toDF("a", "b").registerTempTable("t")
checkAnswer(sql("SELECT a*b FROM t"),
Seq(Row(Decimal(4.0).toBigDecimal),
Row(Decimal(9.0).toBigDecimal)))
}
}
{code}
It then appears to work, though i did need to covert the target result to big decimal
> Multiplication on decimals in dataframe returns null
> ----------------------------------------------------
>
> Key: SPARK-12128
> URL: https://issues.apache.org/jira/browse/SPARK-12128
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.5.0, 1.5.1, 1.5.2
> Environment: Scala 2.11/Spark 1.5.0/1.5.1/1.5.2
> Reporter: Philip Dodds
>
> I hit a weird issue when I tried to multiply to decimals in a select (either in scala or as SQL), and Im assuming I must be missing the point.
> The issue is fairly easy to recreate with something like the following:
> {code:java}
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> import sqlContext.implicits._
> import org.apache.spark.sql.types.Decimal
> case class Trade(quantity: Decimal,price: Decimal)
> val data = Seq.fill(100) {
> val price = Decimal(20+scala.util.Random.nextInt(10))
> val quantity = Decimal(20+scala.util.Random.nextInt(10))
> Trade(quantity, price)
> }
> val trades = sc.parallelize(data).toDF()
> trades.registerTempTable("trades")
> trades.select(trades("price")*trades("quantity")).show
> sqlContext.sql("select price/quantity,price*quantity,price+quantity,price-quantity from trades").show
> {code}
> The odd part is if you run it you will see that the addition/division and subtraction works but the multiplication returns a null.
> Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)
> ie.
> {code}
> +------------------+
> |(price * quantity)|
> +------------------+
> | null|
> | null|
> | null|
> | null|
> | null|
> +------------------+
> +--------------------+----+--------------------+--------------------+
> | _c0| _c1| _c2| _c3|
> +--------------------+----+--------------------+--------------------+
> |0.952380952380952381|null|41.00000000000000...|-1.00000000000000...|
> |1.380952380952380952|null|50.00000000000000...|8.000000000000000000|
> |1.272727272727272727|null|50.00000000000000...|6.000000000000000000|
> |0.833333333333333333|null|44.00000000000000...|-4.00000000000000...|
> |1.000000000000000000|null|58.00000000000000...| 0E-18|
> +--------------------+----+--------------------+--------------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org