You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "koert kuipers (Jira)" <ji...@apache.org> on 2020/06/26 18:53:00 UTC
[jira] [Updated] (SPARK-32109) SQL hash function handling of nulls
makes collision too likely
[ https://issues.apache.org/jira/browse/SPARK-32109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
koert kuipers updated SPARK-32109:
----------------------------------
Description:
this ticket is about org.apache.spark.sql.functions.hash and sparks handling of nulls when hashing sequences.
{code:java}
scala> spark.sql("SELECT hash('bar', null)").show()
+---------------+
|hash(bar, NULL)|
+---------------+
| -1808790533|
+---------------+
scala> spark.sql("SELECT hash(null, 'bar')").show()
+---------------+
|hash(NULL, bar)|
+---------------+
| -1808790533|
+---------------+
{code}
these are differences sequences. e.g. these could be positions 0 and 1 in a dataframe which are diffferent columns with entirely different meanings. the hashes should not be the same.
another example:
{code:java}
scala> Seq(("john", null), (null, "john")).toDF("name", "alias").withColumn("hash", hash(col("name"), col("alias"))).show
+----+-----+---------+
|name|alias| hash|
+----+-----+---------+
|john| null|487839701|
|null| john|487839701|
+----+-----+---------+ {code}
instead of ignoring nulls each null show do a transform to the hash so that the order of elements including the nulls matters for the outcome.
was:
this ticket is about org.apache.spark.sql.functions.hash and sparks handling of nulls when hashing sequences.
{code:java}
scala> spark.sql("SELECT hash('bar', null)").show()
+---------------+
|hash(bar, NULL)|
+---------------+
| -1808790533|
+---------------+
scala> spark.sql("SELECT hash(null, 'bar')").show()
+---------------+
|hash(NULL, bar)|
+---------------+
| -1808790533|
+---------------+
{code}
these are differences sequences. e.g. these could be positions 0 and 1 in a dataframe which are diffferent columns with entirely different meanings. the hashes should bot be the same.
another example:
{code:java}
scala> Seq(("john", null), (null, "john")).toDF("name", "alias").withColumn("hash", hash(col("name"), col("alias"))).show
+----+-----+---------+
|name|alias| hash|
+----+-----+---------+
|john| null|487839701|
|null| john|487839701|
+----+-----+---------+ {code}
> SQL hash function handling of nulls makes collision too likely
> --------------------------------------------------------------
>
> Key: SPARK-32109
> URL: https://issues.apache.org/jira/browse/SPARK-32109
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: koert kuipers
> Priority: Minor
>
> this ticket is about org.apache.spark.sql.functions.hash and sparks handling of nulls when hashing sequences.
> {code:java}
> scala> spark.sql("SELECT hash('bar', null)").show()
> +---------------+
> |hash(bar, NULL)|
> +---------------+
> | -1808790533|
> +---------------+
> scala> spark.sql("SELECT hash(null, 'bar')").show()
> +---------------+
> |hash(NULL, bar)|
> +---------------+
> | -1808790533|
> +---------------+
> {code}
> these are differences sequences. e.g. these could be positions 0 and 1 in a dataframe which are diffferent columns with entirely different meanings. the hashes should not be the same.
> another example:
> {code:java}
> scala> Seq(("john", null), (null, "john")).toDF("name", "alias").withColumn("hash", hash(col("name"), col("alias"))).show
> +----+-----+---------+
> |name|alias| hash|
> +----+-----+---------+
> |john| null|487839701|
> |null| john|487839701|
> +----+-----+---------+ {code}
> instead of ignoring nulls each null show do a transform to the hash so that the order of elements including the nulls matters for the outcome.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org