You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/10/12 03:45:00 UTC
[jira] [Commented] (SPARK-29426) Watermark does not take effect
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949892#comment-16949892 ]
Jungtaek Lim commented on SPARK-29426:
--------------------------------------
Could you attach query listener and see the actual event time in each batch as well? And I suggest you to provide simple and complete reproducer, as to reproduce mes has to exist.
> Watermark does not take effect
> ------------------------------
>
> Key: SPARK-29426
> URL: https://issues.apache.org/jira/browse/SPARK-29426
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.3
> Reporter: jingshanglu
> Priority: Major
>
> I use withWatermark and window to express windowed aggregations, but the Watermark does not take effect.
> my code:
> {code:java}
> // code placeholder
> Dataset<Row> clientSqlIpCount = mes.withWatermark("timestamp","1 minute")
> .groupBy(
> functions.window(mes.col("timestamp"),"10 minutes","5 minutes"),
> mes.col("sql"),mes.col("client"),mes.col("ip"))
> .count();
> StreamingQuery query = clientSqlIpCount
> .writeStream()
> .outputMode("Update")
> .format("console")
> .start();
> spark.streams().awaitAnyTermination();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org