You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by markgrover <gi...@git.apache.org> on 2016/02/09 23:40:53 UTC
[GitHub] spark pull request: SPARK-13252: Bump up Kafka to 0.9.0.0
GitHub user markgrover opened a pull request:
https://github.com/apache/spark/pull/11143
SPARK-13252: Bump up Kafka to 0.9.0.0
Apache Kafka release 0.9.0.0 came out some time and we should add support for it. This JIRA is related to SPARK-12177 which is related to adding support for new consumer API only available starting v0.9.0.0
However, we should upgrade Kafka to 0.9.0.0 regardless of when (and before) the support for the new consumer API gets added.
We also use some non-public APIs from Kafka which have changed in 0.9.0.0 release. So, this change should also take care of updating those usages.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markgrover/spark kafka09-vanilla
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11143.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11143
----
commit f46b5c8489a6bfd0197aa2b0a8bc61b0cb7135d8
Author: Mark Grover <ma...@apache.org>
Date: 2016-02-09T22:39:12Z
SPARK-13252: Bump up Kafka to 0.9.0.0
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182163032
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51004/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182162862
**[Test build #51004 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51004/consoleFull)** for PR 11143 at commit [`a1cf44e`](https://github.com/apache/spark/commit/a1cf44e928ff62b571b117f05688e17c7ccf5971).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182133935
Thanks for reviewing, @vanzin Much appreciated!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-190914907
I really don't think it makes sense to discuss this PR outside of the context of SPARK-12177 and the approach taken for supporting the new consumer. Merging it as is would break things for end users without giving them any new features as a benefit, and nothing in Spark 2.0 that I'm aware of otherwise requires the kafka version bump.
If SPARK-12177 ends up being implemented as a separate artifact, this PR isn't even necessary. If it ends up being implemented in the same artifact and breaking 0.8 compatibility, this PR is a small drop in the bucket compared to the overall code change and could be handled at that time.
Regarding the semi-private apis, those are either in testing code, or in the receiver-based implementation which (so far) no one has shown a lot of interest in updating for the new consumer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182153879
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51000/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182996634
Thanks Mario. Do let us know what you think @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189099110
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182153559
**[Test build #51000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51000/consoleFull)** for PR 11143 at commit [`f46b5c8`](https://github.com/apache/spark/commit/f46b5c8489a6bfd0197aa2b0a8bc61b0cb7135d8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182157957
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by mbonaci <gi...@git.apache.org>.
Github user mbonaci commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-190971237
@ijuma my bad, I was looking for `SimpleConsumer` in [the client api](https://home.apache.org/~junrao/kafka-0.9.0.1-candidate1/javadoc/), totally forgetting that it's part of the `core`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189098928
**[Test build #52012 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52012/consoleFull)** for PR 11143 at commit [`d3951e0`](https://github.com/apache/spark/commit/d3951e03e303b57ba2e5937ec4a8f8c9226c76aa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182157427
**[Test build #51001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51001/consoleFull)** for PR 11143 at commit [`f46b5c8`](https://github.com/apache/spark/commit/f46b5c8489a6bfd0197aa2b0a8bc61b0cb7135d8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182120485
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189259048
@koeninger may be a better reviewer at this point.
I suspect that 2.0.0 is the only opportunity in the near future to move to 0.9, so I think it's a good idea. Any objections?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189365359
Kafka isn't just a library dependency that a user can try out a new version of on a particular job and see if it works.
It's an infrastructure component that the entire business may depend on. Upgrading running brokers is not risk-free.
From a user point of view, I think it's better to have an artifact that works with spark 2.0 and kafka 0.8 brokers, even if that means moving the existing code to a spark-streaming-kafka08 subproject and doing new development on spark-streaming-kafka for kafka 0.9 (or, for that matter, kafka 0.10, which may be out by the time Spark 2.0 is ready)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182157962
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51001/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover closed the pull request at:
https://github.com/apache/spark/pull/11143
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182154267
Hi @tdas ! Thanks for reviewing.
I talk about the compatibility story at length in the related PR #10953 (in particular, [here](https://github.com/apache/spark/pull/10953#issue-129260890))
TLDR from that comment:
The status quo is only supporting 0.8 which will work with 0.8 or later brokers. There is definitely tons of interest in 0.9's new consumer API, so we will have to support that imminently (#10953). So, the question that really remains is, which version(s) do we support the Kafka old consumer API from?
We have two options:
1. Support Apache 0.9 and later ONLY.
Pros:
* Easy to manage and easy to support, since there is only one version of Kafka that's support.
* No changes required from users to their Spark apps.
* Kafka community is pushing all their users to move to 0.9.0.
Cons:
* Users will have to upgrade Kafka brokers to Kafka 0.9
2. Support both Apache Kafka 0.8 and 0.9+.
Pros:
* Support for both 0.8 and 0.9 brokers
Cons:
* Build management - we'll likely need a 2 maven profiles, 2 assemblies, 2 builds, actual code duplication, to publish two flavors of the kafka integration artifacts. Decide what will be our default, etc.
* We'll have to this around for at least another major release i.e. until Spark 3.0.
Given that we are doing Spark 2.0 where we have the liberty of breaking away from old versions of Kafka, I proposed that we go with Option 1 and only support Kafka 0.9 release. Does that sound reasonable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by ijuma <gi...@git.apache.org>.
Github user ijuma commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-190818071
"When Kafka went from 0.8 to 0.9, they did not leave the old high and low level consumer APIs around just because there are users that use them."
Kafka 0.9 still has the old high and low level consumers (and the old producers too).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by mbonaci <gi...@git.apache.org>.
Github user mbonaci commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-190808475
IMHO option 1 is cleaner.
Users who have to postpone upgrading Kafka brokers have the option of using older versions of Spark.
When Kafka went from 0.8 to 0.9, they did not leave the old high and low level consumer APIs around just because there are users that use them. If you want to upgrade to Kafka 0.9 you'll have to refactor your consumer clients to work with 0.9 API, simple as that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189383164
Thanks @srowen and @koeninger for your thoughts.
I did take a look at what other projects are doing. In particular, I looked at Storm and Flume since they both depend on Kafka in ways similar to Spark streaming.
Storm has [this PR](https://github.com/apache/storm/pull/986/files) for bumping to Kafka 0.9 which is doing the same thing I am proposing in this PR - bumping Kafka from 0.8 to 0.9 without keeping support for 0.8.
Flume is doing the same thing in [FLUME-2855](https://issues.apache.org/jira/browse/FLUME-2855) - moving to support 0.9 without keeping 0.8 support.
For both Storm and Flume, it's not even a major release, like it is the case for us.
Long story short, I want to do the right thing for Spark, and I am more than happy to go implement the 2 subproject approach with maven profiles - one supporting Kafka 0.8 and other supporting Kafka 0.9. I also realize that Kafka is not a simple library dependency and there is a good chunk of operational overhead involved in folks upgrading to Kafka. However, the way things are looking right now, perhaps, that's the way Kafka community (cc @jkreps @gwenshap) wanted them to, folks essentially would have to move to Kafka 0.9 because of one of their other Kafka apps may need it. The reason I raised this discussion of getting rid of Kafka 0.8 support is because we will have carry the burden of supporting Kafka 0.8.x line for Spark 2.x line and now is our chance to think through this and drop support for Kafka 0.8, if we decide to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-190845440
Kafka 0.9 still has the old high and low level consumers (and the old producers too).
Correct but they are not compatible when using the 0.9 client with 0.8 brokers. So, your Kafka client app may need no code changes if you are using the old consumer API but you'd still have to upgrade your broker first before you upgrade your client.
Anyways, since @koeninger and @tdas don't feel comfortable about my proposal upgrading to Kafka 0.9 (and dropping support for 0.8), I am working on updating this PR to support 0.8 and 0.9. It's not just a simple maven profile, because we use some non-private APIs (in AdminUtils, ZKClient, etc.) which have changed between 0.8 and 0.9, so a) I have to either do a shim layer, or b) change the code so it doesn't depend on those APIs. Will keep you all posted here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182153877
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182136615
**[Test build #51004 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51004/consoleFull)** for PR 11143 at commit [`a1cf44e`](https://github.com/apache/spark/commit/a1cf44e928ff62b571b117f05688e17c7ccf5971).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by gwenshap <gi...@git.apache.org>.
Github user gwenshap commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-190909149
Agreeing with @mbonaci. Users who delay upgrading their Kafka brokers can continue using older versions of Spark.
@markgrover If you have code that depends on AdminUtils, perhaps you will want to delay the version upgrade until Grant Henke's patches adding Admin APIs are in, and then at least you'll have a path for going off the semi-private APIs :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189099111
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52012/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by mariobriggs <gi...@git.apache.org>.
Github user mariobriggs commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182769749
FWIW, the [IBM Cloud Message Hub service](https://www.ng.bluemix.net/docs/services/MessageHub/index.html#messagehub050) which is Kafka, has already moved to 0.9.0 , so i support option 1 that @markgrover suggests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182131389
Code LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-191376232
This discussion is very closely tied to SPARK-12177 and so I agree with @koeninger that it makes sense for this to decided when we resolved SPARK-12177. So, for those interested, please follow that JIRA and its related PR #10953
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182136221
@markgrover What is the compatibility story of Kafka 0.8 and 0.9 with existing Kafka installations? If Kafka 0.9 client API is not supposed to back compatible with Kafka 0.8 installations, then upgrading SS to use 0.9 is not a good idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189072415
**[Test build #52012 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52012/consoleFull)** for PR 11143 at commit [`d3951e0`](https://github.com/apache/spark/commit/d3951e03e303b57ba2e5937ec4a8f8c9226c76aa).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182163030
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182127663
**[Test build #51001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51001/consoleFull)** for PR 11143 at commit [`f46b5c8`](https://github.com/apache/spark/commit/f46b5c8489a6bfd0197aa2b0a8bc61b0cb7135d8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by ijuma <gi...@git.apache.org>.
Github user ijuma commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-190883372
That's right @markgrover, the current approach used by Kafka preserves compatibility for users, but makes it a bit complicated for libraries/systems that want to support newer client features while still supporting older Kafka brokers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-189110299
Hi @tdas, I'd still appreciate if you could review this. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13252] [KAFKA] Bump up Kafka to 0.9.0.0
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11143#issuecomment-182122561
**[Test build #51000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51000/consoleFull)** for PR 11143 at commit [`f46b5c8`](https://github.com/apache/spark/commit/f46b5c8489a6bfd0197aa2b0a8bc61b0cb7135d8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org