You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/24 06:51:16 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
HeartSaVioR opened a new pull request #27333:
URL: https://github.com/apache/spark/pull/27333
### What changes were proposed in this pull request?
This patch adds new UTs to prevent SPARK-29438 for streaming aggregation as well as flatMapGroupsWithState, as we agree about the review comment quote here:
https://github.com/apache/spark/pull/26162#issuecomment-576929692
> LGTM for this PR. But on a additional note, this is a very subtle and easy-to-make bug with TaskContext.getPartitionId. I wonder if this bug is present in any other stateful operation. I wonder if this bug is present in any other stateful operation. Can you please verify how partitionId is used in the other stateful operations?
For now they're not broken, but even better if we have UTs to prevent the case for the future.
### Why are the changes needed?
New UTs will prevent streaming aggregation and flatMapGroupsWithState to be broken in future where it is placed on the right side of UNION and the number of partition is changing on the left side of UNION. Please refer SPARK-29438 for more details.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added UTs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675252970
**[Test build #127536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127536/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya closed pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
viirya closed pull request #27333:
URL: https://github.com/apache/spark/pull/27333
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675450958
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675250950
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716032349
Will merge this once jenkins pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643729369
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675450958
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715667921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675301413
**[Test build #127551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127551/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692387249
**[Test build #128677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128677/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643706079
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-622044614
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634491784
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634420114
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643756712
**[Test build #124001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124001/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715810779
Accidentally closed this - don't get the UX on Github mobile app. (Don't know how to reopen in the app.) Will reopen soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675252970
**[Test build #127536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127536/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643728387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123990/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675297620
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715740603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130222/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675297154
**[Test build #127536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127536/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634422033
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716063066
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27333:
URL: https://github.com/apache/spark/pull/27333#discussion_r511492575
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
##########
@@ -1019,6 +1019,56 @@ class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest {
spark.createDataset(Seq(("a", 2), ("b", 1))).toDF)
}
+ testWithAllStateVersions("SPARK-29438: ensure UNION doesn't lead (flat)MapGroupsWithState" +
+ " to use shifted partition IDs") {
+ val stateFunc = (key: String, values: Iterator[String], state: GroupState[RunningCount]) => {
+ val count = state.getOption.map(_.count).getOrElse(0L) + values.size
+ state.update(RunningCount(count))
+ (key, count.toString)
+ }
+
+ def constructUnionDf(desiredPartitionsForInput1: Int)
+ : (MemoryStream[String], MemoryStream[String], DataFrame) = {
+ val input1 = MemoryStream[String](desiredPartitionsForInput1)
+ val input2 = MemoryStream[String]
+ val df1 = input1.toDF()
+ .select($"value", $"value")
+ val df2 = input2.toDS()
+ .groupByKey(x => x)
+ .mapGroupsWithState(stateFunc) // Types = State: MyState, Out: (Str, Str)
+ .toDF()
+
+ // Unioned DF would have columns as (String, String)
+ (input1, input2, df1.union(df2))
+ }
+
+ withTempDir { checkpointDir =>
+ val (input1, input2, unionDf) = constructUnionDf(2)
+ testStream(unionDf, Update)(
+ StartStream(checkpointLocation = checkpointDir.getAbsolutePath),
+ MultiAddData(input1, "input1-a")(input2, "input2-a"),
+ CheckNewAnswer(("input1-a", "input1-a"), ("input2-a", "1")),
+ StopStream
+ )
+
+ // We're restoring the query with different number of partitions in left side of UNION,
+ // which may lead right side of union to have mismatched partition IDs (e.g. if it relies on
+ // TaskContext.partitionId()). This test will verify (flat)MapGroupsWithState doesn't have
+ // such issue.
+
+ val (newInput1, newInput2, newUnionDf) = constructUnionDf(3)
+
+ newInput1.addData("input1-a")
+ newInput2.addData("input2-a")
Review comment:
Oh I got it. We reuse the same checkpoint directory and so we need to fill same batches of previous stream query. Actually we even don't need to put the same data.
Then the new query begins from the new added data, and read the same state store.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716041745
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34837/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715667918
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34823/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716032552
**[Test build #130237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130237/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715667921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715740574
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675302013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657320994
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715665750
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34823/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657271102
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634471090
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643706522
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643729615
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634421671
**[Test build #123163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123163/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643728220
**[Test build #123990 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123990/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715661004
**[Test build #130222 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130222/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643706441
**[Test build #123990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123990/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657271823
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634307498
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634470896
**[Test build #123163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123163/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643728385
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634422033
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-622042868
**[Test build #122132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122132/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692496346
**[Test build #128677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128677/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27333:
URL: https://github.com/apache/spark/pull/27333#discussion_r511309080
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
##########
@@ -1019,6 +1019,56 @@ class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest {
spark.createDataset(Seq(("a", 2), ("b", 1))).toDF)
}
+ testWithAllStateVersions("SPARK-29438: ensure UNION doesn't lead (flat)MapGroupsWithState" +
+ " to use shifted partition IDs") {
+ val stateFunc = (key: String, values: Iterator[String], state: GroupState[RunningCount]) => {
+ val count = state.getOption.map(_.count).getOrElse(0L) + values.size
+ state.update(RunningCount(count))
+ (key, count.toString)
+ }
+
+ def constructUnionDf(desiredPartitionsForInput1: Int)
+ : (MemoryStream[String], MemoryStream[String], DataFrame) = {
+ val input1 = MemoryStream[String](desiredPartitionsForInput1)
+ val input2 = MemoryStream[String]
+ val df1 = input1.toDF()
+ .select($"value", $"value")
+ val df2 = input2.toDS()
+ .groupByKey(x => x)
+ .mapGroupsWithState(stateFunc) // Types = State: MyState, Out: (Str, Str)
+ .toDF()
+
+ // Unioned DF would have columns as (String, String)
+ (input1, input2, df1.union(df2))
+ }
+
+ withTempDir { checkpointDir =>
+ val (input1, input2, unionDf) = constructUnionDf(2)
+ testStream(unionDf, Update)(
+ StartStream(checkpointLocation = checkpointDir.getAbsolutePath),
+ MultiAddData(input1, "input1-a")(input2, "input2-a"),
+ CheckNewAnswer(("input1-a", "input1-a"), ("input2-a", "1")),
+ StopStream
+ )
+
+ // We're restoring the query with different number of partitions in left side of UNION,
+ // which may lead right side of union to have mismatched partition IDs (e.g. if it relies on
+ // TaskContext.partitionId()). This test will verify (flat)MapGroupsWithState doesn't have
+ // such issue.
+
+ val (newInput1, newInput2, newUnionDf) = constructUnionDf(3)
+
+ newInput1.addData("input1-a")
+ newInput2.addData("input2-a")
Review comment:
Why we need to add data here? Don't we rely on `MultiAddData` below to add more data to `newInput1` and `newInput2`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675449886
**[Test build #127551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127551/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634644182
**[Test build #123174 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123174/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692387684
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692387249
**[Test build #128677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128677/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657271658
**[Test build #125729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125729/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675301413
**[Test build #127551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127551/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643706441
**[Test build #123990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123990/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634307505
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675302013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634305431
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715740574
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634492056
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675297631
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127536/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716084716
Thanks all for reviewing and merging!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692387061
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643729502
**[Test build #124001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124001/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-621815608
**[Test build #122132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122132/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692497937
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675250942
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643728385
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-622044614
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-621816542
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716063066
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715660771
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675297620
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716038768
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34837/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634495561
**[Test build #123174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123174/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-621815608
**[Test build #122132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122132/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692497937
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716041751
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715739957
**[Test build #130222 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130222/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643729615
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657320994
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-621816542
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634495561
**[Test build #123174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123174/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643729502
**[Test build #124001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124001/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716064216
Thanks. Merging to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-692387684
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716032205
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716062887
**[Test build #130237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130237/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634471090
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643706522
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634306981
**[Test build #123144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123144/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716032552
**[Test build #130237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130237/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634645187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643756928
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634492056
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634421671
**[Test build #123163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123163/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675250318
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-621811785
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657320554
**[Test build #125729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125729/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634492065
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27802/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-716041751
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-675299031
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-715661004
**[Test build #130222 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130222/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634645187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657271823
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-643756928
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #27333:
URL: https://github.com/apache/spark/pull/27333
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634306981
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-634471099
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123163/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #27333:
URL: https://github.com/apache/spark/pull/27333#issuecomment-657271658
**[Test build #125729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125729/testReport)** for PR 27333 at commit [`466363e`](https://github.com/apache/spark/commit/466363edb22ea83a81e21a72f1b983dc7b5a733e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #27333: [SPARK-29438][SS][FOLLOWUP] Add regression tests for Streaming Aggregation and flatMapGroupsWithState
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27333:
URL: https://github.com/apache/spark/pull/27333#discussion_r511329896
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
##########
@@ -1019,6 +1019,56 @@ class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest {
spark.createDataset(Seq(("a", 2), ("b", 1))).toDF)
}
+ testWithAllStateVersions("SPARK-29438: ensure UNION doesn't lead (flat)MapGroupsWithState" +
+ " to use shifted partition IDs") {
+ val stateFunc = (key: String, values: Iterator[String], state: GroupState[RunningCount]) => {
+ val count = state.getOption.map(_.count).getOrElse(0L) + values.size
+ state.update(RunningCount(count))
+ (key, count.toString)
+ }
+
+ def constructUnionDf(desiredPartitionsForInput1: Int)
+ : (MemoryStream[String], MemoryStream[String], DataFrame) = {
+ val input1 = MemoryStream[String](desiredPartitionsForInput1)
+ val input2 = MemoryStream[String]
+ val df1 = input1.toDF()
+ .select($"value", $"value")
+ val df2 = input2.toDS()
+ .groupByKey(x => x)
+ .mapGroupsWithState(stateFunc) // Types = State: MyState, Out: (Str, Str)
+ .toDF()
+
+ // Unioned DF would have columns as (String, String)
+ (input1, input2, df1.union(df2))
+ }
+
+ withTempDir { checkpointDir =>
+ val (input1, input2, unionDf) = constructUnionDf(2)
+ testStream(unionDf, Update)(
+ StartStream(checkpointLocation = checkpointDir.getAbsolutePath),
+ MultiAddData(input1, "input1-a")(input2, "input2-a"),
+ CheckNewAnswer(("input1-a", "input1-a"), ("input2-a", "1")),
+ StopStream
+ )
+
+ // We're restoring the query with different number of partitions in left side of UNION,
+ // which may lead right side of union to have mismatched partition IDs (e.g. if it relies on
+ // TaskContext.partitionId()). This test will verify (flat)MapGroupsWithState doesn't have
+ // such issue.
+
+ val (newInput1, newInput2, newUnionDf) = constructUnionDf(3)
+
+ newInput1.addData("input1-a")
+ newInput2.addData("input2-a")
Review comment:
We're initializing the different memory streams here, which are not aware of previous data and require us to fill the same data again on restore. (Each addition to the memory stream represents a batch, hence you'll need to add the same for the new memory stream.)
Probably there could be the another kind of small improvement for tests if we have convenient way to push all the input rows we put before to the new memory stream. That may be dealt with minor PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org