RE : 3 equalTo "3.15" = true

[Denis DEBARBIEUX <dd...@norsys.fr>]

I am confused since the two column have the same name.


________________________________________
De : Artur Sukhenko [artur.sukhenko@gmail.com]
Date d'envoi : mercredi 6 février 2019 17:32
À : Russell Spitzer
Cc : user@spark.apache.org
Objet : Re: 3 equalTo "3.15" = true

scala> df.select(colString, colShort, colShort.equalTo(colString)).explain
== Physical Plan ==
LocalTableScan [tier_id#3, tier_id#56, (CAST(tier_id AS SMALLINT) = tier_id)#50]


On Wed, Feb 6, 2019 at 6:19 PM Russell Spitzer <ru...@gmail.com>> wrote:
Run an "explain" instead of show, i'm betting it's casting tier_id to a small_int to do the comparison

On Wed, Feb 6, 2019 at 9:31 AM Artur Sukhenko <ar...@gmail.com>> wrote:
Hello guys,
I am migrating from Spark 1.6 to 2.2 and have this issue:
I am casting string to short and comparing them with equal .
Original code is:
... when(col(fieldName).equalTo(castedValueCol), castedValueCol).

  otherwise(defaultErrorValueCol)

Reproduce (version 2.3.0.cloudera4):
scala> val df = Seq("3.15").toDF("tier_id")
df: org.apache.spark.sql.DataFrame = [tier_id: string]

scala> val colShort = col("tier_id").cast(ShortType)
colShort: org.apache.spark.sql.Column = CAST(tier_id AS SMALLINT)

scala> val colString = col("tier_id")
colString: org.apache.spark.sql.Column = tier_id

scala> res4.select(colString, colShort, colShort.equalTo(colString)).show
+-------+-------+-------------------------------------+
|tier_id|tier_id|(CAST(tier_id AS SMALLINT) = tier_id)|
+-------+-------+-------------------------------------+
|   3.15|      3|                                 true|
+-------+-------+-------------------------------------+
scala>

Why is this?
--
--
Artur Sukhenko
--
--
Artur Sukhenko

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org