You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/04 07:49:07 UTC
[GitHub] [spark] xuanyuanking opened a new pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
xuanyuanking opened a new pull request #28725:
URL: https://github.com/apache/spark/pull/28725
### What changes were proposed in this pull request?
Add compatibility tests for streaming aggregation state store format.
### Why are the changes needed?
After SPARK-31894, we have a validation checking for the streaming state store. It's better to add integrated tests in the PR builder as soon as the breaking changes introduced.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Test only.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-638673339
**[Test build #123524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123524/testReport)** for PR 28725 at commit [`b6b7b18`](https://github.com/apache/spark/commit/b6b7b18766e655128e895fc349e51eb2e2c5deae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646514143
**[Test build #124276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124276/testReport)** for PR 28725 at commit [`9ad2c06`](https://github.com/apache/spark/commit/9ad2c068226ffc03b139b1c0a99226ac8c36b8b7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #28725:
URL: https://github.com/apache/spark/pull/28725#discussion_r441377492
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingStateStoreFormatCompatibilitySuite.scala
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.commons.io.FileUtils
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.streaming.InternalOutputModes.Complete
+import org.apache.spark.sql.execution.streaming.MemoryStream
+import org.apache.spark.sql.functions._
+import org.apache.spark.util.Utils
+
+/**
+ * An integrated test for streaming state store format compatibility.
+ * For each PR breaks this test, we need to pay attention to the underlying unsafe row format
+ * changing. All the checkpoint dirs were generated based on Spark version 2.4.5. If we accept the
+ * changes, it means the checkpoint for Structured Streaming will become non-reusable. Please add
+ * a new test for the issue, just like the test suite "SPARK-28067 changed the sum decimal unsafe
+ * row format".
+ */
+class StreamingStateStoreFormatCompatibilitySuite extends StreamTest {
+ import testImplicits._
+
+ private def prepareCheckpointDir(testName: String): File = {
+ val resourceUri = this.getClass.getResource("/structured-streaming/" +
+ s"checkpoint-version-2.4.5-for-compatibility-test-${testName}").toURI
+ val checkpointDir = Utils.createTempDir().getCanonicalFile
+ FileUtils.copyDirectory(new File(resourceUri), checkpointDir)
+ checkpointDir
+ }
+
+ test("common functions") {
+ val inputData = MemoryStream[Int]
+ val aggregated =
+ inputData.toDF().toDF("value")
+ .selectExpr(
+ "value",
+ "value % 5 AS id",
+ "CAST(value AS STRING) as str",
+ "CAST(value AS FLOAT) as f",
+ "CAST(value AS DOUBLE) as d",
+ "CAST(value AS DECIMAL) as dec",
+ "value % 3 AS mod",
+ "named_struct('key', CAST(value AS STRING), 'value', value) AS s")
+ .groupBy($"id")
+ .agg(
+ avg($"value").as("avg_v"),
+ avg($"f").as("avg_f"),
+ avg($"d").as("avg_d"),
+ avg($"dec").as("avg_dec"),
+ count($"value").as("cnt"),
+ first($"value").as("first_v"),
+ first($"s").as("first_s"),
+ last($"value").as("last_v"),
+ last($"s").as("last_s"),
+ min(struct("value", "str")).as("min_struct"),
+ max($"value").as("max_v"),
+ sum($"value").as("sum_v"),
+ sum($"f").as("sum_f"),
+ sum($"d").as("sum_d"),
+ // The test for sum decimal broke by SPARK-28067, use separated test for it
+ // sum($"dec").as("sum_dec"),
+ collect_list($"value").as("col_list"),
+ collect_set($"mod").as("col_set"))
+ .select("id", "avg_v", "avg_f", "avg_d", "avg_dec", "cnt", "first_v", "first_s.value",
+ "last_v", "last_s.value", "min_struct.value", "max_v", "sum_v", "sum_f", "sum_d",
+ "col_list", "col_set")
+
+ val checkpointDir = prepareCheckpointDir("common-functions")
+ inputData.addData(0 to 9: _*)
+
+ testStream(aggregated, Complete)(
+ StartStream(checkpointLocation = checkpointDir.getAbsolutePath),
+ /*
+ Note: The checkpoint was generated using the following input in Spark version 2.4.5
+ AddData(inputData, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9),
+ CheckAnswer(
+ Row(0, 2.5, 2.5F, 2.5, 2.5000, 2, 0, 0, 5, 5, 0, 5, 5, 5.0, 5.0, Seq(0, 5),
+ Seq(0, 2)),
+ Row(1, 3.5, 3.5F, 3.5, 3.5000, 2, 1, 1, 6, 6, 1, 6, 7, 7.0, 7.0, Seq(1, 6),
+ Seq(0, 1)),
+ Row(2, 4.5, 4.5F, 4.5, 4.5000, 2, 2, 2, 7, 7, 2, 7, 9, 9.0, 9.0, Seq(2, 7),
+ Seq(1, 2)),
+ Row(3, 5.5, 5.5F, 5.5, 5.5000, 2, 3, 3, 8, 8, 3, 8, 11, 11.0, 11.0, Seq(3, 8),
+ Seq(0, 2)),
+ Row(4, 6.5, 6.5F, 6.5, 6.5000, 2, 4, 4, 9, 9, 4, 9, 13, 13.0, 13.0, Seq(4, 9),
+ Seq(0, 1)))
+ */
+ AddData(inputData, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19),
+ CheckAnswer(
+ Row(0, 7.5, 7.5, 7.5, 7.5000, 4, 0, 0, 15, 15, 0, 15, 30, 30.0, 30.0,
+ Seq(0, 5, 10, 15), Seq(0, 1, 2)),
+ Row(1, 8.5, 8.5, 8.5, 8.5000, 4, 1, 1, 16, 16, 1, 16, 34, 34.0, 34.0,
+ Seq(1, 6, 11, 16), Seq(0, 1, 2)),
+ Row(2, 9.5, 9.5, 9.5, 9.5000, 4, 2, 2, 17, 17, 2, 17, 38, 38.0, 38.0,
+ Seq(2, 7, 12, 17), Seq(0, 1, 2)),
+ Row(3, 10.5, 10.5, 10.5, 10.5000, 4, 3, 3, 18, 18, 3, 18, 42, 42.0, 42.0,
+ Seq(3, 8, 13, 18), Seq(0, 1, 2)),
+ Row(4, 11.5, 11.5, 11.5, 11.5000, 4, 4, 4, 19, 19, 4, 19, 46, 46.0, 46.0,
+ Seq(4, 9, 14, 19), Seq(0, 1, 2)))
+ )
+ }
+
+ test("statistical functions") {
+ val inputData = MemoryStream[Long]
+ val aggregated =
+ inputData.toDF().toDF("value")
+ .selectExpr(
+ "value",
+ "value % 5 AS id",
+ "CAST(value AS STRING) as str",
+ "CAST(value AS FLOAT) as f",
+ "CAST(value AS DOUBLE) as d",
+ "CAST(value AS DECIMAL) as dec",
+ "value % 3 AS mod")
+ .groupBy($"id")
+ .agg(
+ kurtosis($"d").as("kts"),
+ skewness($"d").as("skew"),
+ approx_count_distinct($"mod").as("approx_cnt"),
+ approx_count_distinct($"f").as("approx_cnt_f"),
+ approx_count_distinct($"d").as("approx_cnt_d"),
+ approx_count_distinct($"dec").as("approx_cnt_dec"),
+ approx_count_distinct($"str").as("approx_cnt_str"),
+ stddev_pop($"d").as("stddev_pop"),
+ stddev_samp($"d").as("stddev_samp"),
+ var_pop($"d").as("var_pop"),
+ var_samp($"d").as("var_samp"),
+ covar_pop($"value", $"mod").as("covar_pop"),
+ covar_samp($"value", $"mod").as("covar_samp"),
+ corr($"value", $"mod").as("corr"))
+ .select("id", "kts", "skew", "approx_cnt", "approx_cnt_f", "approx_cnt_d",
+ "approx_cnt_dec", "approx_cnt_str", "stddev_pop", "stddev_samp", "var_pop", "var_samp",
+ "covar_pop", "covar_samp", "corr")
+
+ val checkpointDir = prepareCheckpointDir("statistical-functions")
+ inputData.addData(0L to 9L: _*)
+
+ testStream(aggregated, Complete)(
+ StartStream(checkpointLocation = checkpointDir.getAbsolutePath),
+ /*
+ Note: The checkpoint was generated using the following input in Spark version 2.4.5
+ AddData(inputData, 0L to 9L: _*),
+ CheckAnswer(
+ Row(0, -2.0, 0.0, 2, 2, 2, 2, 2, 2.5, 3.5355339059327378, 6.25, 12.5, 2.5, 5.0, 1.0),
+ Row(1, -2.0, 0.0, 2, 2, 2, 2, 2, 2.5, 3.5355339059327378, 6.25, 12.5, -1.25, -2.5, -1.0),
+ Row(2, -2.0, 0.0, 2, 2, 2, 2, 2, 2.5, 3.5355339059327378, 6.25, 12.5, -1.25, -2.5, -1.0),
+ Row(3, -2.0, 0.0, 2, 2, 2, 2, 2, 2.5, 3.5355339059327378, 6.25, 12.5, 2.5, 5.0, 1.0),
+ Row(4, -2.0, 0.0, 2, 2, 2, 2, 2, 2.5, 3.5355339059327378, 6.25, 12.5, -1.25, -2.5, -1.0))
+ */
+
+ AddData(inputData, 10L to 19L: _*),
+ CheckAnswer(
+ Row(0, -1.36, 0.0, 3, 4, 4, 4, 4, 5.5901699437494745, 6.454972243679028, 31.25,
+ 41.666666666666664, -0.625, -0.8333333333333334, -0.13483997249264842),
+ Row(1, -1.36, 0.0, 3, 4, 4, 4, 4, 5.5901699437494745, 6.454972243679028, 31.25,
+ 41.666666666666664, 1.25, 1.6666666666666667, 0.31622776601683794),
+ Row(2, -1.36, 0.0, 3, 4, 4, 4, 4, 5.5901699437494745, 6.454972243679028, 31.25,
+ 41.666666666666664, -0.625, -0.8333333333333334, -0.13483997249264842),
+ Row(3, -1.36, 0.0, 3, 4, 4, 4, 4, 5.5901699437494745, 6.454972243679028, 31.25,
+ 41.666666666666664, -0.625, -0.8333333333333334, -0.13483997249264842),
+ Row(4, -1.36, 0.0, 3, 4, 4, 4, 4, 5.5901699437494745, 6.454972243679028, 31.25,
+ 41.666666666666664, 1.25, 1.6666666666666667, 0.31622776601683794))
+ )
+ }
+
+ test("deduplicate with all columns") {
Review comment:
cc @maropu @dongjoon-hyun
This new integrated test will return the wrong answer without #28830.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646514690
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-644983504
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-644982503
**[Test build #124127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124127/testReport)** for PR 28725 at commit [`f311e82`](https://github.com/apache/spark/commit/f311e8290f7be10493871626467854bf91e95f11).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class StreamingStateStoreFormatCompatibilitySuite extends StreamTest `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-644825819
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646626984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28725:
URL: https://github.com/apache/spark/pull/28725#discussion_r442723369
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingStateStoreFormatCompatibilitySuite.scala
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.commons.io.FileUtils
+
+import org.apache.spark.SparkException
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.streaming.InternalOutputModes.Complete
+import org.apache.spark.sql.execution.streaming.MemoryStream
+import org.apache.spark.sql.functions._
+import org.apache.spark.util.Utils
+
+/**
+ * An integrated test for streaming state store format compatibility.
+ * For each PR breaks this test, we need to pay attention to the underlying unsafe row format
+ * changing. All the checkpoint dirs were generated based on Spark version 2.4.5. If we accept the
+ * changes, it means the checkpoint for Structured Streaming will become non-reusable. Please add
+ * a new test for the issue, just like the test suite "SPARK-28067 changed the sum decimal unsafe
+ * row format".
+ */
+class StreamingStateStoreFormatCompatibilitySuite extends StreamTest {
Review comment:
can we explain how the legacy checkpoints are generated? We can either put it in doc, or an ignored test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-638673832
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-638673339
**[Test build #123524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123524/testReport)** for PR 28725 at commit [`b6b7b18`](https://github.com/apache/spark/commit/b6b7b18766e655128e895fc349e51eb2e2c5deae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-647352013
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #28725:
URL: https://github.com/apache/spark/pull/28725
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646514690
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646625818
**[Test build #124276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124276/testReport)** for PR 28725 at commit [`9ad2c06`](https://github.com/apache/spark/commit/9ad2c068226ffc03b139b1c0a99226ac8c36b8b7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-638816806
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-644825084
**[Test build #124127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124127/testReport)** for PR 28725 at commit [`f311e82`](https://github.com/apache/spark/commit/f311e8290f7be10493871626467854bf91e95f11).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-638673832
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646626984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-644825084
**[Test build #124127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124127/testReport)** for PR 28725 at commit [`f311e82`](https://github.com/apache/spark/commit/f311e8290f7be10493871626467854bf91e95f11).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-644825819
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-644983504
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #28725: [SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-647367303
Thanks for the review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646515571
#28707 has been merged, these integration tests are ready for review.
cc @cloud-fan @HeartSaVioR for taking a look when you have time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-638816806
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-638815815
**[Test build #123524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123524/testReport)** for PR 28725 at commit [`b6b7b18`](https://github.com/apache/spark/commit/b6b7b18766e655128e895fc349e51eb2e2c5deae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28725:
URL: https://github.com/apache/spark/pull/28725#issuecomment-646514143
**[Test build #124276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124276/testReport)** for PR 28725 at commit [`9ad2c06`](https://github.com/apache/spark/commit/9ad2c068226ffc03b139b1c0a99226ac8c36b8b7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org