You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2024/09/30 13:20:00 UTC
[jira] [Updated] (SPARK-49836) The parent operator is lost when window function receives time window as parameter
[ https://issues.apache.org/jira/browse/SPARK-49836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim updated SPARK-49836:
---------------------------------
Description:
This issue is from Spark 3.4.0+. I just marked the highest bugfix version for each version line to ease of marking all versions.
This is a silly bug and I'm not sure how return works in Scala after figuring out what happened, but here's my observation.
[https://github.com/apache/spark/blob/c54c017e93090a5fb2edf1b5ef029561b6387a3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala#L90-L94]
This lambda is a partial function - I can't find the exact behavior definition of calling "return" in the partial function. Looks like calling return does not only complete the execution of current context (here it was Aggregate), but somehow complete the execution in broader context and the above operator is "lost".
Thanks [~azera] for the report of the broken test!
EDIT: It does not look like happening for every case - we have relevant tests in MultiStatefulOperatorsSuite and they have been passing for a while.
Looks like the issue is coupled with subquery.
was:
This issue is from Spark 3.4.0+. I just marked the highest bugfix version for each version line to ease of marking all versions.
This is a silly bug and I'm not sure how return works in Scala after figuring out what happened, but here's my observation.
[https://github.com/apache/spark/blob/c54c017e93090a5fb2edf1b5ef029561b6387a3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala#L90-L94]
This lambda is a partial function - I can't find the exact behavior definition of calling "return" in the partial function. Looks like calling return does not only complete the execution of current context (here it was Aggregate), but somehow complete the execution in broader context and the above operator is "lost".
Thanks [~azera] for the report of the broken test!
> The parent operator is lost when window function receives time window as parameter
> ----------------------------------------------------------------------------------
>
> Key: SPARK-49836
> URL: https://issues.apache.org/jira/browse/SPARK-49836
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.0.0, 3.4.3, 3.5.3
> Reporter: Jungtaek Lim
> Priority: Blocker
>
> This issue is from Spark 3.4.0+. I just marked the highest bugfix version for each version line to ease of marking all versions.
> This is a silly bug and I'm not sure how return works in Scala after figuring out what happened, but here's my observation.
> [https://github.com/apache/spark/blob/c54c017e93090a5fb2edf1b5ef029561b6387a3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala#L90-L94]
> This lambda is a partial function - I can't find the exact behavior definition of calling "return" in the partial function. Looks like calling return does not only complete the execution of current context (here it was Aggregate), but somehow complete the execution in broader context and the above operator is "lost".
> Thanks [~azera] for the report of the broken test!
> EDIT: It does not look like happening for every case - we have relevant tests in MultiStatefulOperatorsSuite and they have been passing for a while.
> Looks like the issue is coupled with subquery.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org