You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by abhishekagarwal87 <gi...@git.apache.org> on 2016/06/10 17:28:23 UTC
[GitHub] storm pull request #1482: STORM-1876: Option to build storm-kafka and storm-...
GitHub user abhishekagarwal87 opened a pull request:
https://github.com/apache/storm/pull/1482
STORM-1876: Option to build storm-kafka and storm-kafka-client with different kafka client version
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/abhishekagarwal87/storm STORM-1876
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1482.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1482
----
commit be877a02466208aa9c469fea4d1826b48c06839f
Author: Abhishek Agarwal <ab...@inmobi.com>
Date: 2016-06-10T17:27:18Z
STORM-1876: Option to build storm-kafka and storm-kafka-client with different kafka client version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1482: STORM-1876: Option to build storm-kafka and storm-...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/1482#discussion_r70818175
--- Diff: pom.xml ---
@@ -258,6 +258,12 @@
<calcite.version>1.4.0-incubating</calcite.version>
<jackson.version>2.6.3</jackson.version>
<maven-surefire.version>2.18.1</maven-surefire.version>
+
+ <!-- Kafka version used by old kafka spout code -->
+ <storm.kafka.version>0.8.2.1</storm.kafka.version>
+ <storm.kafka.artifact.id>kafka_2.10</storm.kafka.artifact.id>
+
+ <!-- kafka version used by new kafka spout code -->
--- End diff --
@abhishekagarwal87 can the property names be a bit more consistent?
A suggestion would be to prefix the properties used by `storm.kafka.client`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/1482
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1482: STORM-1876: Option to build storm-kafka and storm-...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1482
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1482
Great @unsleepy22 you got the point.
@abhishekagarwal87 I guess you want to address also doc since you mark component of issue to also 'documentation'. Could you address this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1482
+1 Nice addressing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1482
+1 besides the neat pick which would be to have the new spout property names prefixed by `storm.kafka.client` property names for consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by unsleepy22 <gi...@git.apache.org>.
Github user unsleepy22 commented on the issue:
https://github.com/apache/storm/pull/1482
Should we change the README.md accordingly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1482
I guess what we would want is
- We describe default version of storm-kafka and storm-kafka-client clearly so that users can use it directly when version is matched to what users want.
- If it's not, we encourage users to build their own storm-kafka / storm-kafka-client with kafka version which they wanted.
So I think documentation is more important on this change.
Regarding `provided`, I also think so. Do we want to remove `provided` from kafka-client and kafka artifact from storm-kafka and storm-kafka-client after this change? @harshach @hmcl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/1482
I am still +1 on merging this. As @ptgoetz stated lets document this in README
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on the issue:
https://github.com/apache/storm/pull/1482
@harshach @hmcl - can you guys take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on the issue:
https://github.com/apache/storm/pull/1482
Thanks @harshach and @unsleepy22 for comments. Even I would like to understand more on how these options are to be used :)
@harshach @hmcl - can you add some details here?
Another thing I noticed is that org.apache.kafka:kafka-clients is longer marked provided. should we revert this since we mark org.apache.kafka:kafka as provided? If the dependencies are marked provided, then compiling with different versions is more like a no-op.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/1482
@abhishekagarwal87 @HeartSaVioR @unsleepy22 currently we are hard-coding the version here https://github.com/apache/storm/blob/master/external/storm-kafka/pom.xml#L34 . Without having this kafka.version we won't be able to override during the build times if I want to build storm-kafka with version 0.9. Hence the reason for this patch where we are introducing 2 versions
1. storm.kafka.version -> storm-kafka
2. kafka.version -> to build the storm-kafka-client(new kafka api)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1482: STORM-1876: Option to build storm-kafka and storm-...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1482#discussion_r71076464
--- Diff: external/storm-kafka/README.md ---
@@ -242,6 +242,16 @@ use Kafka 0.8.1.1 built against Scala 2.10, you would use the following dependen
Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.
--- End diff --
It is already there on line 222.
**When building a project with storm-kafka, you must explicitly add the Kafka dependency**
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1482: STORM-1876: Option to build storm-kafka and storm-...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/1482#discussion_r71072896
--- Diff: external/storm-kafka/README.md ---
@@ -242,6 +242,16 @@ use Kafka 0.8.1.1 built against Scala 2.10, you would use the following dependen
Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.
--- End diff --
Lets make a note that users have to explicitly state the dependency on kafka-core libs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the issue:
https://github.com/apache/storm/pull/1482
@abhishekagarwal87 Any update on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the issue:
https://github.com/apache/storm/pull/1482
The README needs to be updated for this change with instructions on how to use different Kafka client and scala versions.
I'm okay with either leaving or removing the `provided` scope for the Kafka dependency. It was originally there in order to force the user to make a choice of scala version (mainly directed at users using Scala in their topology code). Without it, users have to explicitly exclude the Kafka dependency and include the one they want. It's kind of a 6 vs. one-half dozen tradeoff.
I'd be +1 once the documentation is updated to reflect this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1482: STORM-1876: Option to build storm-kafka and storm-kafka-c...
Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on the issue:
https://github.com/apache/storm/pull/1482
Hey guys, can you take a look now?
1. changed the scopes as discussed in #1556
2. Updated the readme with information regarding maven parameters as well as kafka api compatibility with storm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---