You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by helena <gi...@git.apache.org> on 2014/12/07 16:52:16 UTC
[GitHub] spark pull request: #2808 update kafka to version 0.8.2
GitHub user helena opened a pull request:
https://github.com/apache/spark/pull/3631
#2808 update kafka to version 0.8.2
#2808 update kafka to version 0.8.2. Kafka 0.8.2 is in beta still.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/helena/spark wip-2808-kafka-0.8.2-upgrade
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3631.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3631
----
commit e768164fca1c93ec0a99f7020e301368f798156c
Author: Helena Edelson <he...@datastax.com>
Date: 2014-12-07T15:50:44Z
#2808 update kafka to version 0.8.2
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73959340
@helena I updated it, pr is at https://github.com/apache/spark/pull/4537
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-72926976
All streaming-kafka sbt tests and /dev/run-tests passed in my initial push as beta, and with the GA update all still pass. Should I resubmit this PR against another branch?
I am getting cluster backwards compat docs..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73100148
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena closed the pull request at:
https://github.com/apache/spark/pull/3631
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-68013540
Hey @helena I am not sure whether we want to upgrade to a beta version. Especially, there have been subtle changes within Kafka between releases and I am afraid to make changes (even more so with beta versions) without fully understanding implications. So could you elaborate on why you want to update to 0.8.2 beta ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-72784310
Aah cool. However 0.8.1 and 0.8.2 have pretty big changes between them, so lets merge this for the next release. We are already doing a lot of experimental Kafka stuff in this release (feature merge window has closed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-72858529
@tdas what shall I do with this PR to complete it then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-68013974
Hi @tdas, +1. This PR was done as initial push, and as soon as 0.8.2 is out of beta I planned to do the final commit, then have it considered. I never expected it to be merged while in beta.
To your other point, it is important to allow users to use the latest version of kafka in their applications which integrate with spark streaming kafka.
You can find a high-level description here http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2 with full release notes here https://archive.apache.org/dist/kafka/0.8.2-beta/RELEASE_NOTES.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by rustyrazorblade <gi...@git.apache.org>.
Github user rustyrazorblade commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-72775510
Kafka 0.8.2 is now listed as latest stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73368921
@helena I too would like to update to Kafka 0.8.2 as it does have a number of good improvements. can you rebase? @tdas are you OK with merging this for only master / 1.4.0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73728226
This will need some changes to KafkaCluster and possibly other things related to the new api... let me know if you want a hand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-67836412
@JoshRosen Ticket name updated :) Sorry for the delay, I was away.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73101317
NP, I think manual may not be necessary, just the addition of tests that automate testing the version or testing all and handling the execution path properly. I can add that when merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73120485
[Test build #26849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26849/consoleFull) for PR 3631 at commit [`2e67c66`](https://github.com/apache/spark/commit/2e67c66b174bd1641ea8986edde3ce5598add612).
* This patch **passes all tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73938104
@koeninger this is a definite blocker for me, I'm upgrading the connector to scala 2.11 with a cross build. Let me know if you have time, otherwise I will get back to this tomorrow morning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73120496
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26849/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #2808 update kafka to version 0.8.2
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-65941807
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73100513
Then for the new API (which does not store offsets externally) may need to be updated to ensure compatibility. I am not sure whether the current unit tests will catch such issues. This will require a bit of manual testing. Will do after 1.3 release rush. :)
Thanks for the link!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73979757
Great I'll close this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73100788
[Test build #26849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26849/consoleFull) for PR 3631 at commit [`2e67c66`](https://github.com/apache/spark/commit/2e67c66b174bd1641ea8986edde3ce5598add612).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73728845
@koeninger Yes, there consumer offset and some other minor changes to make.
I'm finding it hard to carve out time for this, high ticket load this week. Go for it if you want, otherwise I'll do it later in the week.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-69206347
Waiting for kafka 0.8.2 to move to GA, still beta
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #2808 update kafka to version 0.8.2
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-67233594
Hi @helena,
Do you mind editing this PR's title to say "SPARK-2808" instead of #2808? This will ensure that this PR is properly linked to JIRA and [our PR review dashboard](http://spark-prs.appspot.com).
Does this PR introduce any backwards-incompatible changes? If a user is using an older version of Kafka, will they have to update any code or configurations as a result of this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-72903283
The first step would be to see whether upgrading to 0.8.2 passes the
inbuilt unit-tests or not (including the new Kafka stuff).
In addition, is there are documentation somewhere that talks about the
compatibility of 0.8.2 with 0.8.0 / 0.8.1.1 kafka clusters? We can figure
out the path accordingly. In any case, it will be in the next release cycle
(1.4).
On Wed, Feb 4, 2015 at 6:02 AM, Helena Edelson <no...@github.com>
wrote:
> @tdas <https://github.com/tdas> what shall I do with this PR to complete
> it then?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3631#issuecomment-72858529>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: #SPARK-2808 update kafka to version 0.8.2
Posted by helena <gi...@git.apache.org>.
Github user helena commented on the pull request:
https://github.com/apache/spark/pull/3631#issuecomment-73377050
Hi @srowen, will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org