You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/10/23 23:54:00 UTC
[jira] [Commented] (SPARK-40892) Loosen the requirement of window_time rule - allow multiple window_time calls
[ https://issues.apache.org/jira/browse/SPARK-40892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17622896#comment-17622896 ]
Apache Spark commented on SPARK-40892:
--------------------------------------
User 'HeartSaVioR' has created a pull request for this issue:
https://github.com/apache/spark/pull/38361
> Loosen the requirement of window_time rule - allow multiple window_time calls
> -----------------------------------------------------------------------------
>
> Key: SPARK-40892
> URL: https://issues.apache.org/jira/browse/SPARK-40892
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> SPARK-40821 introduces a new SQL function "window_time" to extract the representative time from window (which also carries over the event time metadata as well if feasible).
> SPARK-40821 followed the existing rule of time window / session window which only allows a single function call in a same projection (strictly saying, it considers the call of function as once if the function is called with same parameters).
> For existing rules, the restriction makes sense since allowing this would produce cartesian product of rows (although Spark can handle it). But given that window_time only produces one value, the restriction no longer makes sense.
> It would be better to unlock the functionality. Note that this means the resulting column of "window_time()" is no longer be "window_time". (Note that this is the practice most of function calls do. The rules time window and session window don't follow the practice so arguably they have a bug, but fixing the bug would bring backward incompatibility...)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org