You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jose-torres <gi...@git.apache.org> on 2018/01/16 20:47:46 UTC

[GitHub] spark pull request #20282: [SPARK-23093][SS] Don't change run id when reconf...

GitHub user jose-torres opened a pull request:

    https://github.com/apache/spark/pull/20282

    [SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query.

    ## What changes were proposed in this pull request?
    
    Keep the run ID static, using a different ID for the epoch coordinator to avoid cross-execution message contamination.
    
    ## How was this patch tested?
    
    existing unit tests

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jose-torres/spark fix-runid

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20282.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20282
    
----
commit a7c51813d36bdb09b98e119cd50ae9e8afcaca98
Author: Jose Torres <jo...@...>
Date:   2018-01-16T20:30:00Z

    don't change run id when reconfiguring

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86285/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86198/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    LGTM. Merging to master and 2.3.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86285/testReport)** for PR 20282 at commit [`f9b2fd1`](https://github.com/apache/spark/commit/f9b2fd1918ba2da860ed115abf8f9984d02ea16d).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86250 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86250/testReport)** for PR 20282 at commit [`70917a5`](https://github.com/apache/spark/commit/70917a59c052f4b9a3859b03528194b9d16d384e).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class KafkaContinuousReader(`
      * `case class KafkaContinuousReadTask(`
      * `class KafkaContinuousDataReader(`
      * `class KafkaContinuousWriter(`
      * `case class KafkaContinuousWriterFactory(`
      * `class KafkaContinuousDataWriter(`
      * `case class KafkaSourceOffset(partitionToOffsets: Map[TopicPartition, Long]) extends OffsetV2 `
      * `case class KafkaSourcePartitionOffset(topicPartition: TopicPartition, partitionOffset: Long)`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #20282: [SPARK-23093][SS] Don't change run id when reconf...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20282


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by jose-torres <gi...@git.apache.org>.
Github user jose-torres commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    The auto-merge in 70917a5 somehow reverted part of this PR, causing the newly added test to fail.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #20282: [SPARK-23093][SS] Don't change run id when reconf...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20282#discussion_r161884338
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala ---
    @@ -210,15 +214,17 @@ class ContinuousExecution(
           lastExecution.executedPlan // Force the lazy generation of execution plan
         }
     
    -    sparkSession.sparkContext.setLocalProperty(
    +    sparkSessionForQuery.sparkContext.setLocalProperty(
           ContinuousExecution.START_EPOCH_KEY, currentBatchId.toString)
    -    sparkSession.sparkContext.setLocalProperty(
    -      ContinuousExecution.RUN_ID_KEY, runId.toString)
    +    val epochCoordinatorId = UUID.randomUUID.toString
    --- End diff --
    
    could you add `run_id + random_uuid` so that it's easy to tell which query this epoch coordinator belongs to?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86203/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86250/testReport)** for PR 20282 at commit [`70917a5`](https://github.com/apache/spark/commit/70917a59c052f4b9a3859b03528194b9d16d384e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86236/testReport)** for PR 20282 at commit [`70917a5`](https://github.com/apache/spark/commit/70917a59c052f4b9a3859b03528194b9d16d384e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86198/testReport)** for PR 20282 at commit [`8092a4d`](https://github.com/apache/spark/commit/8092a4dfe07640ee473d76e4e88cd7e10f371be6).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86236 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86236/testReport)** for PR 20282 at commit [`70917a5`](https://github.com/apache/spark/commit/70917a59c052f4b9a3859b03528194b9d16d384e).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class KafkaContinuousReader(`
      * `case class KafkaContinuousReadTask(`
      * `class KafkaContinuousDataReader(`
      * `class KafkaContinuousWriter(`
      * `case class KafkaContinuousWriterFactory(`
      * `class KafkaContinuousDataWriter(`
      * `case class KafkaSourceOffset(partitionToOffsets: Map[TopicPartition, Long]) extends OffsetV2 `
      * `case class KafkaSourcePartitionOffset(topicPartition: TopicPartition, partitionOffset: Long)`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86285/testReport)** for PR 20282 at commit [`f9b2fd1`](https://github.com/apache/spark/commit/f9b2fd1918ba2da860ed115abf8f9984d02ea16d).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86203/testReport)** for PR 20282 at commit [`6bc61b0`](https://github.com/apache/spark/commit/6bc61b024c489d6fd7c1dd1b8a5478bbd6a4fa7e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86236/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86250/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86203/testReport)** for PR 20282 at commit [`6bc61b0`](https://github.com/apache/spark/commit/6bc61b024c489d6fd7c1dd1b8a5478bbd6a4fa7e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20282: [SPARK-23093][SS] Don't change run id when reconfiguring...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20282
  
    **[Test build #86198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86198/testReport)** for PR 20282 at commit [`8092a4d`](https://github.com/apache/spark/commit/8092a4dfe07640ee473d76e4e88cd7e10f371be6).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org