You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zsxwing <gi...@git.apache.org> on 2017/01/01 19:59:04 UTC
[GitHub] spark pull request #16449: [SPARK-19050][SS]Fix EventTimeWatermarkSuite 'del...
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/16449
[SPARK-19050][SS]Fix EventTimeWatermarkSuite 'delay in months and years handled correctly
## What changes were proposed in this pull request?
`monthsSinceEpoch` in this test is like `math.floor(num)`, so `monthDiff` has two possible values`
## How was this patch tested?
Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark watermark-test-hotfix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16449.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16449
----
commit c456c6e28894e4fe2fc6f1ea8228375765820cb4
Author: Shixiong Zhu <sh...@databricks.com>
Date: 2017-01-01T19:55:17Z
Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'del...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16449
It sounds like our watermarkTime delay calculation causes this issue. Below are two typical cases:
Case 1: when setting the watermark delay to 1 month interval:
```Scala
.withWatermark("eventTime", "1 months")
```
the `watermarket` time is `Thu Dec 01 20:05:34 PST 2016` and the current time is `Sun Jan 01 20:05:34 PST 2017`
Case 2: when setting the watermark delay to 1 month interval:
```Scala
.withWatermark("eventTime", "29 months")
```
the `watermarket` time is `Thu Jul 17 21:10:50 PDT 2014` and the current time is `Sun Jan 01 20:10:50 PST 2017`
It sounds like it is caused by our intentional over-estimation (that is, by [using 31 days per month](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/EventTimeWatermarkExec.scala#L88))?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS]Fix EventTimeWatermarkSuite 'delay in m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16449
cc @gatorsmile @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSui...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16449
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'del...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16449
Merging to master and 2.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS]Fix EventTimeWatermarkSuite 'delay in m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16449
I will merge this PR once this test passes since the master is broken now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'del...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16449
Yeah, agree. This is a bug in the test cases. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'del...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16449
**[Test build #70777 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70777/testReport)** for PR 16449 at commit [`c456c6e`](https://github.com/apache/spark/commit/c456c6e28894e4fe2fc6f1ea8228375765820cb4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS]Fix EventTimeWatermarkSuite 'delay in m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16449
**[Test build #70777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70777/testReport)** for PR 16449 at commit [`c456c6e`](https://github.com/apache/spark/commit/c456c6e28894e4fe2fc6f1ea8228375765820cb4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'del...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16449
> the watermarket time is Thu Jul 17 21:10:50 PDT 2014 and the current time is Sun Jan 01 20:10:50 PST 2017
> It sounds like it is caused by our intentional over-estimation (that is, by using 31 days per month)?
@gatorsmile this is an expected behavior. It's intentional and it's correct as per the comment in `Dataset.withWatermark`,
> the actual watermark used is only guaranteed to be at least `delayThreshold` behind the actual event time. In some cases we may still process records that arrive more than `delayThreshold` late.
If the user wants to filter data accurately, they need to use `filter` explicitly.
That's why I changed the test rather than the watermark calculation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'del...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16449
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16449: [SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'del...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16449
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70777/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org