You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mike Sukmanowsky (JIRA)" <ji...@apache.org> on 2018/08/10 16:16:00 UTC
[jira] [Commented] (SPARK-10816) EventTime based sessionization
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576503#comment-16576503 ]
Mike Sukmanowsky commented on SPARK-10816:
------------------------------------------
Voted, but want to express that we're definitely interested in native support for session windows/merging windows in structured streaming. We're heavy PySpark users and mapGroupsWithState is Java/Scala only so being able to support this via SQL or PySpark DSL would be a great win.
Apache Beam supports a syntax like so (see https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102):
{code:java}
PCollection<KV<String, Integer>> scores = input
.apply(Window.into(Sessions.withGapDuration(Duration.standardMinutes(1)))
.triggering(
AtWatermark()
.withEarlyFirings(AtPeriod(Duration.standardMinutes(1)))
.withLateFirings(AtCount(1)))
.accumulatingAndRetractingFiredPanes())
.apply(Sum.integersPerKey());{code}
> EventTime based sessionization
> ------------------------------
>
> Key: SPARK-10816
> URL: https://issues.apache.org/jira/browse/SPARK-10816
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Reporter: Reynold Xin
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org