You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Scott C. Cote" <sc...@gmail.com> on 2014/04/20 05:11:27 UTC
kafka/storm integration help needed
Is storm-kafka integration still supported?
Am trying to setup a dev environment that encompasses storm + kafka
To that goal, I have:
1. kafka 2.9.2-0.8.1 installed on my machine and independently running (all
local host).
2. Successfully configured via log4j a simple java app to write into kafka
(verified with output showing from the kafka-console-consumer.sh).
3. Downloaded the binaries apache-storm-0.9.1-incubating and placed its bin
in the path (verified that storm is now on the path with ³which storm²)
4. Created an eclipse project with the maven dependency on Storm and also
Kafka (didn¹t really know if I need Kafka pom dependency assumed I would):
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.9.2</artifactId>
<version>0.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.1-incubating</version>
</dependency>
5. Want to use storm-contrib/storm-kafka , but there wasn¹t any pom.xml
there, no maven repo, and work seems to be frozen at version .7 kafka. The
KafkaSpout seems like the natural way to plug all of this together.
Other sub-projects in contrib have pom.xml files, but not storm-kafka .
Am I going about this the wrong way? Should I ditch Kafka and use JMS? (and
give up on the supposed easy lo4j integration via kafka)?
I took up the approach of Storm/Kafka integration because it works for
twitter am starting to have doubts .
While I wait for an answer, I¹m going to try to build storm-kafka and add to
my local maven repo .
SCott
Re: kafka/storm integration help needed
Posted by "Scott C. Cote" <sc...@gmail.com>.
Thanks :)
I was busy rewiring all of the FetchRequests to be a benefactor of
FetchRequestBuilder .
Glad that I didn¹t have to finish that exercise .
SCott
[INFO] Building jar:
/Users/scottccote/git/storm-kafka-0.8-plus/target/storm-kafka-0.8-plus-0.5.0
-SNAPSHOT.jar
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 44.210s
[INFO] Finished at: Sun Apr 20 01:10:02 CDT 2014
[INFO] Final Memory: 33M/117M
[INFO] -------------------------
From: Kiran Nagasubramanian <nk...@gmail.com>
Reply-To: "user@storm.incubator.apache.org"
<us...@storm.incubator.apache.org>
Date: Saturday, April 19, 2014 at 10:37 PM
To: "user@storm.incubator.apache.org" <us...@storm.incubator.apache.org>
Subject: Re: kafka/storm integration help needed
This may be of some help to you
https://github.com/wurstmeister/storm-kafka-0.8-plus
Kiran
On Sat, Apr 19, 2014 at 8:11 PM, Scott C. Cote <sc...@gmail.com> wrote:
> Is storm-kafka integration still supported?
>
> Am trying to setup a dev environment that encompasses storm + kafka
>
> To that goal, I have:
>
> 1. kafka 2.9.2-0.8.1 installed on my machine and independently running (all
> local host).
> 2. Successfully configured via log4j a simple java app to write into kafka
> (verified with output showing from the kafka-console-consumer.sh).
> 3. Downloaded the binaries apache-storm-0.9.1-incubating and placed its bin in
> the path (verified that storm is now on the path with ³which storm²)
> 4. Created an eclipse project with the maven dependency on Storm and also
> Kafka (didn¹t really know if I need Kafka pom dependency assumed I would):
>
> <dependency>
>
> <groupId>org.apache.kafka</groupId>
>
> <artifactId>kafka_2.9.2</artifactId>
>
> <version>0.8.1</version>
>
> </dependency>
>
>
>
> <dependency>
>
> <groupId>org.apache.storm</groupId>
>
> <artifactId>storm-core</artifactId>
>
> <version>0.9.1-incubating</version>
>
> </dependency>
>
>
>
> 5. Want to use storm-contrib/storm-kafka , but there wasn¹t any pom.xml there,
> no maven repo, and work seems to be frozen at version .7 kafka. The KafkaSpout
> seems like the natural way to plug all of this together.
> Other sub-projects in contrib have pom.xml files, but not storm-kafka .
>
> Am I going about this the wrong way? Should I ditch Kafka and use JMS? (and
> give up on the supposed easy lo4j integration via kafka)?
>
> I took up the approach of Storm/Kafka integration because it works for twitter
> am starting to have doubts .
>
> While I wait for an answer, I¹m going to try to build storm-kafka and add to
> my local maven repo .
>
> SCott
>
Re: kafka/storm integration help needed
Posted by Kiran Nagasubramanian <nk...@gmail.com>.
This may be of some help to you
https://github.com/wurstmeister/storm-kafka-0.8-plus
Kiran
On Sat, Apr 19, 2014 at 8:11 PM, Scott C. Cote <sc...@gmail.com> wrote:
> Is storm-kafka integration still supported?
>
> Am trying to setup a dev environment that encompasses storm + kafka
>
> To that goal, I have:
>
> 1. kafka 2.9.2-0.8.1 installed on my machine and independently running
> (all local host).
> 2. Successfully configured via log4j a simple java app to write into kafka
> (verified with output showing from the kafka-console-consumer.sh).
> 3. Downloaded the binaries apache-storm-0.9.1-incubating and placed its
> bin in the path (verified that storm is now on the path with “which storm”)
> 4. Created an eclipse project with the maven dependency on Storm and also
> Kafka (didn’t really know if I need Kafka pom dependency – assumed I would):
>
> <dependency>
>
> <groupId>org.apache.kafka</groupId>
>
> <artifactId>kafka_2.9.2</artifactId>
>
> <version>0.8.1</version>
>
> </dependency>
>
>
> <dependency>
>
> <groupId>org.apache.storm</groupId>
>
> <artifactId>storm-core</artifactId>
>
> <version>0.9.1-incubating</version>
>
> </dependency>
>
>
> 5. Want to use storm-contrib/storm-kafka , but there wasn’t any pom.xml
> there, no maven repo, and work seems to be frozen at version .7 kafka. The
> KafkaSpout seems like the natural way to plug all of this together.
> Other sub-projects in contrib have pom.xml files, but not storm-kafka .
>
> Am I going about this the wrong way? Should I ditch Kafka and use JMS?
> (and give up on the supposed easy lo4j integration via kafka)?
>
> I took up the approach of Storm/Kafka integration because it works for
> twitter – am starting to have doubts ….
>
> While I wait for an answer, I’m going to try to build storm-kafka and add
> to my local maven repo ….
>
> SCott
>
>