You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lw-lin <gi...@git.apache.org> on 2016/03/29 14:33:12 UTC

[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

GitHub user lw-lin opened a pull request:

    https://github.com/apache/spark/pull/12035

    [SPARK-13211][STREAMING] Deprecate certain constructors of StreamingContext's

    ## What changes were proposed in this pull request?
    
    These constructors, each of which takes a checkpoint path parameter, are deprecated by this PR:
    - `StreamingContext`'s
      - `this(path: String, hadoopConf: Configuration)`
      - `this(path: String)`
      - `this(path: String, sparkContext: SparkContext)`
    - `JavaStreamingContext`'s
      - `this(path: String)`
      - `this(path: String, hadoopConf: Configuration)`
    
    ## How was this patch tested?
    
    Existing test suits.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lw-lin/spark spark-13211

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/12035.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #12035
    
----
commit 1ab24e1f5fd1687d84f43eea1429d98867c888c9
Author: Liwei Lin <lw...@gmail.com>
Date:   2016-03-29T07:45:49Z

    fix java.util.NoSuchElementException: None.get

commit 8e5cf1ca5501345d82adcd80c781aa67a9beb17f
Author: Liwei Lin <lw...@gmail.com>
Date:   2016-03-29T09:03:36Z

    deprecates StreamingContext's constructors taking ck path as a parameter

commit e230ec06cae0750c68745f7148bce49fa695e298
Author: Liwei Lin <lw...@gmail.com>
Date:   2016-03-29T09:06:28Z

    revert read.getOrElse(null) to read.get

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-202908814
  
    This doesn't address the problem. The existing API you're purporting to deprecate is still entirely fine and necessary. We're talking about a better error message when, incorrectly, later, a checkpoint directory is required but none was specified.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin closed the pull request at:

    https://github.com/apache/spark/pull/12035


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-206123094
  
    @srowen sure, thank you for this review, :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-202874434
  
    **[Test build #54435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54435/consoleFull)** for PR 12035 at commit [`e230ec0`](https://github.com/apache/spark/commit/e230ec06cae0750c68745f7148bce49fa695e298).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-206096337
  
    @lw-lin  you can close this PR given https://github.com/apache/spark/pull/12174


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-202893189
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-202881741
  
    As reported in [\[Spark-13211\]](https://issues.apache.org/jira/browse/SPARK-13211), `StreamingContext` would throw `NoSuchElementException` when created from non-existent checkpoint directory.
    
    Actually, the `SparkContext.getOrCreate(path, ...)` was added in https://github.com/apache/spark/commit/a1b8dd53e3474dae2c49b30bc9719c7f6b98c7cc as the major API for users to create a `StreamingContext` from some checkpoint; see http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing. `SparkContext.getOrCreate(path, ...)` checks if the path exists, checks if any valid checkpoint file exists in the given path, and does or does not re-throw the exception when encounter one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-202893195
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54435/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13211][STREAMING] Deprecate certain con...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12035#issuecomment-202892844
  
    **[Test build #54435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54435/consoleFull)** for PR 12035 at commit [`e230ec0`](https://github.com/apache/spark/commit/e230ec06cae0750c68745f7148bce49fa695e298).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org