You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/02/21 11:37:00 UTC

[jira] [Assigned] (SPARK-38206) Relax the requirement of data type comparison for keys in stream-stream join

     [ https://issues.apache.org/jira/browse/SPARK-38206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-38206:
------------------------------------

    Assignee:     (was: Apache Spark)

> Relax the requirement of data type comparison for keys in stream-stream join
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-38206
>                 URL: https://issues.apache.org/jira/browse/SPARK-38206
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.1.2, 3.2.1, 3.3.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> Currently, stream-stream join checks for the data type compatible between left keys and right keys. It is done as "strict" checking, requiring nullability as same for both sides. This leads to throw assertion error if optimizer turns some columns in one side from nullable to non-nullable but not touching opposite side.
> If it is logically correct to relax the nullability check (with deciding proper type on output schema), we should do it to avoid any possible issue from optimization.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org