You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2024/03/07 06:13:00 UTC
[jira] [Resolved] (SPARK-47305) PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation
[ https://issues.apache.org/jira/browse/SPARK-47305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-47305.
----------------------------------
Fix Version/s: 3.4.3
3.5.2
4.0.0
Resolution: Fixed
Issue resolved by pull request 45406
[https://github.com/apache/spark/pull/45406]
> PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-47305
> URL: https://issues.apache.org/jira/browse/SPARK-47305
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 4.0.0, 3.5.1
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.3, 3.5.2, 4.0.0
>
>
> This seems to be a very old bug in optimizer. Related ticket: https://issues.apache.org/jira/browse/SPARK-21765
> When filter is evaluated to be always false, PruneFilters replaces the filter with empty LocalRelation, which effectively prunes filter. The logic cares about migration of the isStreaming flag, but incorrectly migrated in some case, via picking up the value of isStreaming flag from root node rather than filter (or child).
> isStreaming flag is true if the value of isStreaming flag from any of children is true. Flipping the coin, some children might have isStreaming flag as "false". If the filter being pruned is a descendant to such children (in other word, ancestor of streaming node), LocalRelation is incorrectly tagged as streaming where it should be batch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org