You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yssharma <gi...@git.apache.org> on 2017/05/23 11:58:31 UTC
[GitHub] spark pull request #18071: [SPARK-20855][WIP][DStream] Update the Spark kine...
GitHub user yssharma opened a pull request:
https://github.com/apache/spark/pull/18071
[SPARK-20855][WIP][DStream] Update the Spark kinesis docs to use the KinesisInputDStream builder instead of deprecated KinesisUtils
## What changes were proposed in this pull request?
The examples and docs for Spark-Kinesis integrations use the deprecated KinesisUtils. We should update the docs to use the KinesisInputDStream builder to create DStreams.
## How was this patch tested?
The patch primarily updates the documents. The patch will also need to make changes to the Spark-Kinesis examples. The examples need to be tested.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yssharma/spark ysharma/kinesis_docs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18071.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18071
----
commit 9c1800435ce718c8d32829417dfe8ea462ad9fca
Author: Yash Sharma <ys...@atlassian.com>
Date: 2017-05-22T12:42:40Z
initial changes to update kinesis docs to newer version of API
commit 1e7e8790bdba4373e9bc79b0d12773505367d4ca
Author: Yash Sharma <ys...@atlassian.com>
Date: 2017-05-23T11:56:55Z
fixing indentations
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18071
**[Test build #77250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77250/testReport)** for PR 18071 at commit [`1e7e879`](https://github.com/apache/spark/commit/1e7e8790bdba4373e9bc79b0d12773505367d4ca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18071
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77250/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...
Posted by yssharma <gi...@git.apache.org>.
Github user yssharma commented on the issue:
https://github.com/apache/spark/pull/18071
@HyukjinKwon The PR is ready. Just waiting for some 👍 from the reviewers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by yssharma <gi...@git.apache.org>.
Github user yssharma commented on the issue:
https://github.com/apache/spark/pull/18071
Thanks @budde @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18071
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18071
Hi @yssharma, is it still WIP? I think we should make the PR on the mergable state at least.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18071
**[Test build #79904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79904/testReport)** for PR 18071 at commit [`9d50bd9`](https://github.com/apache/spark/commit/9d50bd9146ad08fce269443a9aa6768f676d0430).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18071: [SPARK-20855][Docs][DStream] Update the Spark kin...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18071
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18071
**[Test build #77250 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77250/testReport)** for PR 18071 at commit [`1e7e879`](https://github.com/apache/spark/commit/1e7e8790bdba4373e9bc79b0d12773505367d4ca).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18071
I'm OK to merge something like this but don't know about the kinesis integration, not enough to review the content. Who could give it a second look, @budde ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by yssharma <gi...@git.apache.org>.
Github user yssharma commented on the issue:
https://github.com/apache/spark/pull/18071
Updated the patch @HyukjinKwon . Thanks for the note. 👍
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...
Posted by yssharma <gi...@git.apache.org>.
Github user yssharma commented on the issue:
https://github.com/apache/spark/pull/18071
Hey @budde , drawing your attention here please.
Just wanted to update the docs with the new Builder interface by you. Few questions on top of my mind -
- Is it compatible with the python API as well like KinesisUtils used to be ?
- How can I test that on the py spark-shell. Trying to use KinesisInputDStream gives me Builder not found execptions. I might be missing something obvious here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18071
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79904/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18071
It looks the last test was failed by style checking. Would you have some time to fix them up?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by yssharma <gi...@git.apache.org>.
Github user yssharma commented on the issue:
https://github.com/apache/spark/pull/18071
I noticed that now. Yes I will post an updated patch today. Thanks @HyukjinKwon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18071
**[Test build #79904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79904/testReport)** for PR 18071 at commit [`9d50bd9`](https://github.com/apache/spark/commit/9d50bd9146ad08fce269443a9aa6768f676d0430).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18071
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18071
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org