You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2022/05/25 01:52:00 UTC
[jira] [Resolved] (SPARK-39220) codegen cause NullPointException
[ https://issues.apache.org/jira/browse/SPARK-39220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-39220.
----------------------------------
Resolution: Cannot Reproduce
> codegen cause NullPointException
> --------------------------------
>
> Key: SPARK-39220
> URL: https://issues.apache.org/jira/browse/SPARK-39220
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0, 2.4.6, 3.2.1
> Reporter: chenxusheng
> Priority: Major
>
> The following code raises NullPointException
> {code:sql}
> SELECT
> fk4c7a8cfc,
> fka54f2a73,
> fk37e266f7
> FROM
> be2a04fad4a24848bee641825e5b3466
> WHERE
> (
> fk4c7a8cfc is not null
> and fk4c7a8cfc<> ''
> )
> LIMIT
> 1000
> {code}
> However, if so, it is normal
> {code:sql}
> SELECT
> fk4c7a8cfc,
> fka54f2a73,
> fk37e266f7
> FROM
> be2a04fad4a24848bee641825e5b3466
> WHERE
> (
> fk4c7a8cfc is not null
> and '' <> fk4c7a8cfc
> )
> LIMIT
> 1000
> {code}
> I just put the '' in where in front.
> The reason for this problem is that the data contains null values.
> *_org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext#genEqual_*
> {code:scala}
> def genEqual(dataType: DataType, c1: String, c2: String): String = dataType match {
> case BinaryType => s"java.util.Arrays.equals($c1, $c2)"
> case FloatType =>
> s"((java.lang.Float.isNaN($c1) && java.lang.Float.isNaN($c2)) || $c1 == $c2)"
> case DoubleType =>
> s"((java.lang.Double.isNaN($c1) && java.lang.Double.isNaN($c2)) || $c1 == $c2)"
> case dt: DataType if isPrimitiveType(dt) => s"$c1 == $c2"
> case dt: DataType if dt.isInstanceOf[AtomicType] => s"$c1.equals($c2)"
> case array: ArrayType => genComp(array, c1, c2) + " == 0"
> case struct: StructType => genComp(struct, c1, c2) + " == 0"
> case udt: UserDefinedType[_] => genEqual(udt.sqlType, c1, c2)
> case NullType => "false"
> case _ =>
> throw new IllegalArgumentException(
> "cannot generate equality code for un-comparable type: " + dataType.catalogString)
> }
> {code}
> {code:scala}
> case dt: DataType if dt.isInstanceOf[AtomicType] => s"$c1.equals($c2)"
> {code}
> Missing null value judgment?
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org