You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Abhi Basu <90...@gmail.com> on 2013/11/07 20:56:40 UTC
Kafka Hadoop Consumer issues
Let me describe my environment. Working on two nodes currently:
1.Single-node hadoop cluster (will refer as Node1)
2.Single node Kafka cluster (will refer as Node2)
Node 2 has 1 broker started with a topic (iot.test.stream) and one command
line producer and one command line consumer to test the kafka install.
Producer can send messages and the Consumer is receiving it.
Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have edited
the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with the
following:
kafka.etl.topic=iot.test.stream
hdfs.default.classpath.dir=/tmp/kafka/lib
hadoop.job.ugi=kafka,hadoop
kafka.server.uri=tcp://idh251-kafka:9095
input=/tmp/kafka/data
output=/tmp/kafka/output
kafka.request.limit=-1
...........
I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
Next I run the following on Node 1:
./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties from
the /kafka-0.8/contrib/hadoop-consumer folder and get a
classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
What am I missing? I was thinking that running the sh file would allow me
to retrieve messages with the same topic name to HDFS from Node 2 to Node
1. I just want to do an end to end test to see that messages coming into
Kafka are being stored in HDFS with the minimal amount of code change
required.
Thanks,
Abhi
--
Abhi Basu
Re: Kafka Hadoop Consumer issues
Posted by Abhi Basu <90...@gmail.com>.
Copy-jars.sh did not copy the hadoop consumer and kafka jars. I have copied
them to HDFS manually, but still getting the same error.
Looks like I have all the reqd jars now:
[root@idh251-0 test]# hadoop fs -ls /tmp/kafka/lib
Warning: $HADOOP_HOME is deprecated.
Found 7 items
-rw-r--r-- 3 root hadoop 29614 2013-11-08 10:11
/tmp/kafka/lib/hadoop-consumer-0.8-SNAPSHOT.jar
-rw-r--r-- 3 root hadoop 2559966 2013-11-08 10:05
/tmp/kafka/lib/kafka_2.8.0-0.8-SNAPSHOT.jar
-rw-r--r-- 3 root hadoop 4502 2013-11-08 10:03
/tmp/kafka/lib/metrics-annotation-3.0.0-c0c8be71.jar
-rw-r--r-- 3 root hadoop 80316 2013-11-08 10:03
/tmp/kafka/lib/metrics-core-3.0.0-c0c8be71.jar
-rw-r--r-- 3 root hadoop 307858 2013-11-08 10:03
/tmp/kafka/lib/piggybank.jar
-rw-r--r-- 3 root hadoop 8807888 2013-11-08 10:07
/tmp/kafka/lib/scala-library.jar
-rw-r--r-- 3 root hadoop 98961 2013-11-08 10:03
/tmp/kafka/lib/zkclient-20120522.jar
On Fri, Nov 8, 2013 at 9:17 AM, Abhi Basu <90...@gmail.com> wrote:
> Still get the same error:
> [root@idh251-0 hadoop-consumer]# ./run-class.sh
> kafka.etl.impl.SimpleKafkaETLJob test/test.properties
>
> :./../../core/target/scala_2.8.0/kafka-*.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/*.jar:./../../contrib/hadoop-consumer/target/scala_2.8.0/*.jar:./../../contrib/hadoop-consumer/lib/piggybank.jar:./../../project/boot/scala-2.8.0/lib/scala-library.jar
> Exception in thread "main" java.lang.NoClassDefFoundError:
> kafka/etl/impl/SimpleKafkaETLJob
> Caused by: java.lang.ClassNotFoundException:
> kafka.etl.impl.SimpleKafkaETLJob
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: kafka.etl.impl.SimpleKafkaETLJob. Program
> will exit.
> [root@idh251-0 hadoop-consumer]# hadoop fs -ls /tmp/kafka/lib
> Warning: $HADOOP_HOME is deprecated.
>
> Here is what is in the HDFS /tmp/kafka/lib folder after running the
> copy-jars.sh:
>
> Found 4 items
> -rw-r--r-- 3 root hadoop 4502 2013-11-08 09:04
> /tmp/kafka/lib/metrics-annotation-3.0.0-c0c8be71.jar
> -rw-r--r-- 3 root hadoop 80316 2013-11-08 09:04
> /tmp/kafka/lib/metrics-core-3.0.0-c0c8be71.jar
> -rw-r--r-- 3 root hadoop 307858 2013-11-08 09:04
> /tmp/kafka/lib/piggybank.jar
> -rw-r--r-- 3 root hadoop 98961 2013-11-08 09:04
> /tmp/kafka/lib/zkclient-20120522.jar
>
>
> On Fri, Nov 8, 2013 at 9:06 AM, Abhi Basu <90...@gmail.com> wrote:
>
>> Ok, sorry, missed a step, ran the copy_jars.sh and now retrying ....
>>
>>
>> On Fri, Nov 8, 2013 at 8:55 AM, Abhi Basu <90...@gmail.com> wrote:
>>
>>> Hi Neha:
>>>
>>> I was following the directions outlined here -
>>> https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It
>>> does not mention anything about registering jars. Can you please provide
>>> more details?
>>>
>>> Thanks,
>>>
>>> Abhi
>>>
>>>
>>> On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede <ne...@gmail.com>wrote:
>>>
>>>> ClassNotFound means the Hadoop job is not able to find the related jar.
>>>> Have you made sure the related jars are registered in the distributed
>>>> cache?
>>>>
>>>>
>>>> On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <90...@gmail.com> wrote:
>>>>
>>>> > Can anyone help me with this issue? I feel like I am very close and am
>>>> > probably making some silly config error.
>>>> >
>>>> > Kafka team, please provide more detailed notes on how to make this
>>>> > component work.
>>>> >
>>>> > Thanks.
>>>> >
>>>> >
>>>> > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <90...@gmail.com> wrote:
>>>> >
>>>> > > Simplekafkaetljob class, as mentioned in the post.
>>>> > >
>>>> > > Thanks
>>>> > >
>>>> > > Abhi
>>>> > >
>>>> > > From Samsung Galaxy S4
>>>> > > On Nov 7, 2013 8:34 PM, "Jun Rao" <ju...@gmail.com> wrote:
>>>> > >
>>>> > >> Which class is not found?
>>>> > >>
>>>> > >> Thanks,
>>>> > >>
>>>> > >> Jun
>>>> > >>
>>>> > >>
>>>> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com>
>>>> wrote:
>>>> > >>
>>>> > >> > Let me describe my environment. Working on two nodes currently:
>>>> > >> > 1.Single-node hadoop cluster (will refer as Node1)
>>>> > >> > 2.Single node Kafka cluster (will refer as Node2)
>>>> > >> >
>>>> > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and
>>>> one
>>>> > >> command
>>>> > >> > line producer and one command line consumer to test the kafka
>>>> install.
>>>> > >> > Producer can send messages and the Consumer is receiving it.
>>>> > >> >
>>>> > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built.
>>>> Have
>>>> > >> edited
>>>> > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file
>>>> with
>>>> > >> the
>>>> > >> > following:
>>>> > >> >
>>>> > >> > kafka.etl.topic=iot.test.stream
>>>> > >> > hdfs.default.classpath.dir=/tmp/kafka/lib
>>>> > >> > hadoop.job.ugi=kafka,hadoop
>>>> > >> > kafka.server.uri=tcp://idh251-kafka:9095
>>>> > >> > input=/tmp/kafka/data
>>>> > >> > output=/tmp/kafka/output
>>>> > >> > kafka.request.limit=-1
>>>> > >> > ...........
>>>> > >> >
>>>> > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
>>>> > >> >
>>>> > >> > Next I run the following on Node 1:
>>>> > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob
>>>> test/test.properties
>>>> > >> from
>>>> > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
>>>> > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob
>>>> class.
>>>> > >> >
>>>> > >> > What am I missing? I was thinking that running the sh file would
>>>> allow
>>>> > >> me
>>>> > >> > to retrieve messages with the same topic name to HDFS from Node
>>>> 2 to
>>>> > >> Node
>>>> > >> > 1. I just want to do an end to end test to see that messages
>>>> coming
>>>> > into
>>>> > >> > Kafka are being stored in HDFS with the minimal amount of code
>>>> change
>>>> > >> > required.
>>>> > >> >
>>>> > >> > Thanks,
>>>> > >> >
>>>> > >> > Abhi
>>>> > >> >
>>>> > >> > --
>>>> > >> > Abhi Basu
>>>> > >> >
>>>> > >>
>>>> > >
>>>> >
>>>> >
>>>> > --
>>>> > Abhi Basu
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Abhi Basu
>>>
>>
>>
>>
>> --
>> Abhi Basu
>>
>
>
>
> --
> Abhi Basu
>
--
Abhi Basu
Re: Kafka Hadoop Consumer issues
Posted by Abhi Basu <90...@gmail.com>.
Still get the same error:
[root@idh251-0 hadoop-consumer]# ./run-class.sh
kafka.etl.impl.SimpleKafkaETLJob test/test.properties
:./../../core/target/scala_2.8.0/kafka-*.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/*.jar:./../../contrib/hadoop-consumer/target/scala_2.8.0/*.jar:./../../contrib/hadoop-consumer/lib/piggybank.jar:./../../project/boot/scala-2.8.0/lib/scala-library.jar
Exception in thread "main" java.lang.NoClassDefFoundError:
kafka/etl/impl/SimpleKafkaETLJob
Caused by: java.lang.ClassNotFoundException:
kafka.etl.impl.SimpleKafkaETLJob
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: kafka.etl.impl.SimpleKafkaETLJob. Program
will exit.
[root@idh251-0 hadoop-consumer]# hadoop fs -ls /tmp/kafka/lib
Warning: $HADOOP_HOME is deprecated.
Here is what is in the HDFS /tmp/kafka/lib folder after running the
copy-jars.sh:
Found 4 items
-rw-r--r-- 3 root hadoop 4502 2013-11-08 09:04
/tmp/kafka/lib/metrics-annotation-3.0.0-c0c8be71.jar
-rw-r--r-- 3 root hadoop 80316 2013-11-08 09:04
/tmp/kafka/lib/metrics-core-3.0.0-c0c8be71.jar
-rw-r--r-- 3 root hadoop 307858 2013-11-08 09:04
/tmp/kafka/lib/piggybank.jar
-rw-r--r-- 3 root hadoop 98961 2013-11-08 09:04
/tmp/kafka/lib/zkclient-20120522.jar
On Fri, Nov 8, 2013 at 9:06 AM, Abhi Basu <90...@gmail.com> wrote:
> Ok, sorry, missed a step, ran the copy_jars.sh and now retrying ....
>
>
> On Fri, Nov 8, 2013 at 8:55 AM, Abhi Basu <90...@gmail.com> wrote:
>
>> Hi Neha:
>>
>> I was following the directions outlined here -
>> https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It
>> does not mention anything about registering jars. Can you please provide
>> more details?
>>
>> Thanks,
>>
>> Abhi
>>
>>
>> On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede <ne...@gmail.com>wrote:
>>
>>> ClassNotFound means the Hadoop job is not able to find the related jar.
>>> Have you made sure the related jars are registered in the distributed
>>> cache?
>>>
>>>
>>> On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <90...@gmail.com> wrote:
>>>
>>> > Can anyone help me with this issue? I feel like I am very close and am
>>> > probably making some silly config error.
>>> >
>>> > Kafka team, please provide more detailed notes on how to make this
>>> > component work.
>>> >
>>> > Thanks.
>>> >
>>> >
>>> > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <90...@gmail.com> wrote:
>>> >
>>> > > Simplekafkaetljob class, as mentioned in the post.
>>> > >
>>> > > Thanks
>>> > >
>>> > > Abhi
>>> > >
>>> > > From Samsung Galaxy S4
>>> > > On Nov 7, 2013 8:34 PM, "Jun Rao" <ju...@gmail.com> wrote:
>>> > >
>>> > >> Which class is not found?
>>> > >>
>>> > >> Thanks,
>>> > >>
>>> > >> Jun
>>> > >>
>>> > >>
>>> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com>
>>> wrote:
>>> > >>
>>> > >> > Let me describe my environment. Working on two nodes currently:
>>> > >> > 1.Single-node hadoop cluster (will refer as Node1)
>>> > >> > 2.Single node Kafka cluster (will refer as Node2)
>>> > >> >
>>> > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
>>> > >> command
>>> > >> > line producer and one command line consumer to test the kafka
>>> install.
>>> > >> > Producer can send messages and the Consumer is receiving it.
>>> > >> >
>>> > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have
>>> > >> edited
>>> > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file
>>> with
>>> > >> the
>>> > >> > following:
>>> > >> >
>>> > >> > kafka.etl.topic=iot.test.stream
>>> > >> > hdfs.default.classpath.dir=/tmp/kafka/lib
>>> > >> > hadoop.job.ugi=kafka,hadoop
>>> > >> > kafka.server.uri=tcp://idh251-kafka:9095
>>> > >> > input=/tmp/kafka/data
>>> > >> > output=/tmp/kafka/output
>>> > >> > kafka.request.limit=-1
>>> > >> > ...........
>>> > >> >
>>> > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
>>> > >> >
>>> > >> > Next I run the following on Node 1:
>>> > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob
>>> test/test.properties
>>> > >> from
>>> > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
>>> > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
>>> > >> >
>>> > >> > What am I missing? I was thinking that running the sh file would
>>> allow
>>> > >> me
>>> > >> > to retrieve messages with the same topic name to HDFS from Node 2
>>> to
>>> > >> Node
>>> > >> > 1. I just want to do an end to end test to see that messages
>>> coming
>>> > into
>>> > >> > Kafka are being stored in HDFS with the minimal amount of code
>>> change
>>> > >> > required.
>>> > >> >
>>> > >> > Thanks,
>>> > >> >
>>> > >> > Abhi
>>> > >> >
>>> > >> > --
>>> > >> > Abhi Basu
>>> > >> >
>>> > >>
>>> > >
>>> >
>>> >
>>> > --
>>> > Abhi Basu
>>> >
>>>
>>
>>
>>
>> --
>> Abhi Basu
>>
>
>
>
> --
> Abhi Basu
>
--
Abhi Basu
Re: Kafka Hadoop Consumer issues
Posted by Abhi Basu <90...@gmail.com>.
Ok, sorry, missed a step, ran the copy_jars.sh and now retrying ....
On Fri, Nov 8, 2013 at 8:55 AM, Abhi Basu <90...@gmail.com> wrote:
> Hi Neha:
>
> I was following the directions outlined here -
> https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It does
> not mention anything about registering jars. Can you please provide more
> details?
>
> Thanks,
>
> Abhi
>
>
> On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> ClassNotFound means the Hadoop job is not able to find the related jar.
>> Have you made sure the related jars are registered in the distributed
>> cache?
>>
>>
>> On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <90...@gmail.com> wrote:
>>
>> > Can anyone help me with this issue? I feel like I am very close and am
>> > probably making some silly config error.
>> >
>> > Kafka team, please provide more detailed notes on how to make this
>> > component work.
>> >
>> > Thanks.
>> >
>> >
>> > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <90...@gmail.com> wrote:
>> >
>> > > Simplekafkaetljob class, as mentioned in the post.
>> > >
>> > > Thanks
>> > >
>> > > Abhi
>> > >
>> > > From Samsung Galaxy S4
>> > > On Nov 7, 2013 8:34 PM, "Jun Rao" <ju...@gmail.com> wrote:
>> > >
>> > >> Which class is not found?
>> > >>
>> > >> Thanks,
>> > >>
>> > >> Jun
>> > >>
>> > >>
>> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com>
>> wrote:
>> > >>
>> > >> > Let me describe my environment. Working on two nodes currently:
>> > >> > 1.Single-node hadoop cluster (will refer as Node1)
>> > >> > 2.Single node Kafka cluster (will refer as Node2)
>> > >> >
>> > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
>> > >> command
>> > >> > line producer and one command line consumer to test the kafka
>> install.
>> > >> > Producer can send messages and the Consumer is receiving it.
>> > >> >
>> > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have
>> > >> edited
>> > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file
>> with
>> > >> the
>> > >> > following:
>> > >> >
>> > >> > kafka.etl.topic=iot.test.stream
>> > >> > hdfs.default.classpath.dir=/tmp/kafka/lib
>> > >> > hadoop.job.ugi=kafka,hadoop
>> > >> > kafka.server.uri=tcp://idh251-kafka:9095
>> > >> > input=/tmp/kafka/data
>> > >> > output=/tmp/kafka/output
>> > >> > kafka.request.limit=-1
>> > >> > ...........
>> > >> >
>> > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
>> > >> >
>> > >> > Next I run the following on Node 1:
>> > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob
>> test/test.properties
>> > >> from
>> > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
>> > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
>> > >> >
>> > >> > What am I missing? I was thinking that running the sh file would
>> allow
>> > >> me
>> > >> > to retrieve messages with the same topic name to HDFS from Node 2
>> to
>> > >> Node
>> > >> > 1. I just want to do an end to end test to see that messages coming
>> > into
>> > >> > Kafka are being stored in HDFS with the minimal amount of code
>> change
>> > >> > required.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Abhi
>> > >> >
>> > >> > --
>> > >> > Abhi Basu
>> > >> >
>> > >>
>> > >
>> >
>> >
>> > --
>> > Abhi Basu
>> >
>>
>
>
>
> --
> Abhi Basu
>
--
Abhi Basu
Re: Kafka Hadoop Consumer issues
Posted by Abhi Basu <90...@gmail.com>.
Hi Neha:
I was following the directions outlined here -
https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It does
not mention anything about registering jars. Can you please provide more
details?
Thanks,
Abhi
On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede <ne...@gmail.com>wrote:
> ClassNotFound means the Hadoop job is not able to find the related jar.
> Have you made sure the related jars are registered in the distributed
> cache?
>
>
> On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <90...@gmail.com> wrote:
>
> > Can anyone help me with this issue? I feel like I am very close and am
> > probably making some silly config error.
> >
> > Kafka team, please provide more detailed notes on how to make this
> > component work.
> >
> > Thanks.
> >
> >
> > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <90...@gmail.com> wrote:
> >
> > > Simplekafkaetljob class, as mentioned in the post.
> > >
> > > Thanks
> > >
> > > Abhi
> > >
> > > From Samsung Galaxy S4
> > > On Nov 7, 2013 8:34 PM, "Jun Rao" <ju...@gmail.com> wrote:
> > >
> > >> Which class is not found?
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com>
> wrote:
> > >>
> > >> > Let me describe my environment. Working on two nodes currently:
> > >> > 1.Single-node hadoop cluster (will refer as Node1)
> > >> > 2.Single node Kafka cluster (will refer as Node2)
> > >> >
> > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
> > >> command
> > >> > line producer and one command line consumer to test the kafka
> install.
> > >> > Producer can send messages and the Consumer is receiving it.
> > >> >
> > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have
> > >> edited
> > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file
> with
> > >> the
> > >> > following:
> > >> >
> > >> > kafka.etl.topic=iot.test.stream
> > >> > hdfs.default.classpath.dir=/tmp/kafka/lib
> > >> > hadoop.job.ugi=kafka,hadoop
> > >> > kafka.server.uri=tcp://idh251-kafka:9095
> > >> > input=/tmp/kafka/data
> > >> > output=/tmp/kafka/output
> > >> > kafka.request.limit=-1
> > >> > ...........
> > >> >
> > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
> > >> >
> > >> > Next I run the following on Node 1:
> > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties
> > >> from
> > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
> > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
> > >> >
> > >> > What am I missing? I was thinking that running the sh file would
> allow
> > >> me
> > >> > to retrieve messages with the same topic name to HDFS from Node 2 to
> > >> Node
> > >> > 1. I just want to do an end to end test to see that messages coming
> > into
> > >> > Kafka are being stored in HDFS with the minimal amount of code
> change
> > >> > required.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Abhi
> > >> >
> > >> > --
> > >> > Abhi Basu
> > >> >
> > >>
> > >
> >
> >
> > --
> > Abhi Basu
> >
>
--
Abhi Basu
Re: Kafka Hadoop Consumer issues
Posted by Neha Narkhede <ne...@gmail.com>.
ClassNotFound means the Hadoop job is not able to find the related jar.
Have you made sure the related jars are registered in the distributed cache?
On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <90...@gmail.com> wrote:
> Can anyone help me with this issue? I feel like I am very close and am
> probably making some silly config error.
>
> Kafka team, please provide more detailed notes on how to make this
> component work.
>
> Thanks.
>
>
> On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <90...@gmail.com> wrote:
>
> > Simplekafkaetljob class, as mentioned in the post.
> >
> > Thanks
> >
> > Abhi
> >
> > From Samsung Galaxy S4
> > On Nov 7, 2013 8:34 PM, "Jun Rao" <ju...@gmail.com> wrote:
> >
> >> Which class is not found?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com> wrote:
> >>
> >> > Let me describe my environment. Working on two nodes currently:
> >> > 1.Single-node hadoop cluster (will refer as Node1)
> >> > 2.Single node Kafka cluster (will refer as Node2)
> >> >
> >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
> >> command
> >> > line producer and one command line consumer to test the kafka install.
> >> > Producer can send messages and the Consumer is receiving it.
> >> >
> >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have
> >> edited
> >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with
> >> the
> >> > following:
> >> >
> >> > kafka.etl.topic=iot.test.stream
> >> > hdfs.default.classpath.dir=/tmp/kafka/lib
> >> > hadoop.job.ugi=kafka,hadoop
> >> > kafka.server.uri=tcp://idh251-kafka:9095
> >> > input=/tmp/kafka/data
> >> > output=/tmp/kafka/output
> >> > kafka.request.limit=-1
> >> > ...........
> >> >
> >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
> >> >
> >> > Next I run the following on Node 1:
> >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties
> >> from
> >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
> >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
> >> >
> >> > What am I missing? I was thinking that running the sh file would allow
> >> me
> >> > to retrieve messages with the same topic name to HDFS from Node 2 to
> >> Node
> >> > 1. I just want to do an end to end test to see that messages coming
> into
> >> > Kafka are being stored in HDFS with the minimal amount of code change
> >> > required.
> >> >
> >> > Thanks,
> >> >
> >> > Abhi
> >> >
> >> > --
> >> > Abhi Basu
> >> >
> >>
> >
>
>
> --
> Abhi Basu
>
Re: Kafka Hadoop Consumer issues
Posted by Abhi Basu <90...@gmail.com>.
Can anyone help me with this issue? I feel like I am very close and am
probably making some silly config error.
Kafka team, please provide more detailed notes on how to make this
component work.
Thanks.
On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <90...@gmail.com> wrote:
> Simplekafkaetljob class, as mentioned in the post.
>
> Thanks
>
> Abhi
>
> From Samsung Galaxy S4
> On Nov 7, 2013 8:34 PM, "Jun Rao" <ju...@gmail.com> wrote:
>
>> Which class is not found?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com> wrote:
>>
>> > Let me describe my environment. Working on two nodes currently:
>> > 1.Single-node hadoop cluster (will refer as Node1)
>> > 2.Single node Kafka cluster (will refer as Node2)
>> >
>> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
>> command
>> > line producer and one command line consumer to test the kafka install.
>> > Producer can send messages and the Consumer is receiving it.
>> >
>> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have
>> edited
>> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with
>> the
>> > following:
>> >
>> > kafka.etl.topic=iot.test.stream
>> > hdfs.default.classpath.dir=/tmp/kafka/lib
>> > hadoop.job.ugi=kafka,hadoop
>> > kafka.server.uri=tcp://idh251-kafka:9095
>> > input=/tmp/kafka/data
>> > output=/tmp/kafka/output
>> > kafka.request.limit=-1
>> > ...........
>> >
>> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
>> >
>> > Next I run the following on Node 1:
>> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties
>> from
>> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
>> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
>> >
>> > What am I missing? I was thinking that running the sh file would allow
>> me
>> > to retrieve messages with the same topic name to HDFS from Node 2 to
>> Node
>> > 1. I just want to do an end to end test to see that messages coming into
>> > Kafka are being stored in HDFS with the minimal amount of code change
>> > required.
>> >
>> > Thanks,
>> >
>> > Abhi
>> >
>> > --
>> > Abhi Basu
>> >
>>
>
--
Abhi Basu
Re: Kafka Hadoop Consumer issues
Posted by Abhi Basu <90...@gmail.com>.
Simplekafkaetljob class, as mentioned in the post.
Thanks
Abhi
>From Samsung Galaxy S4
On Nov 7, 2013 8:34 PM, "Jun Rao" <ju...@gmail.com> wrote:
> Which class is not found?
>
> Thanks,
>
> Jun
>
>
> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com> wrote:
>
> > Let me describe my environment. Working on two nodes currently:
> > 1.Single-node hadoop cluster (will refer as Node1)
> > 2.Single node Kafka cluster (will refer as Node2)
> >
> > Node 2 has 1 broker started with a topic (iot.test.stream) and one
> command
> > line producer and one command line consumer to test the kafka install.
> > Producer can send messages and the Consumer is receiving it.
> >
> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have edited
> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with the
> > following:
> >
> > kafka.etl.topic=iot.test.stream
> > hdfs.default.classpath.dir=/tmp/kafka/lib
> > hadoop.job.ugi=kafka,hadoop
> > kafka.server.uri=tcp://idh251-kafka:9095
> > input=/tmp/kafka/data
> > output=/tmp/kafka/output
> > kafka.request.limit=-1
> > ...........
> >
> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
> >
> > Next I run the following on Node 1:
> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties from
> > the /kafka-0.8/contrib/hadoop-consumer folder and get a
> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
> >
> > What am I missing? I was thinking that running the sh file would allow me
> > to retrieve messages with the same topic name to HDFS from Node 2 to Node
> > 1. I just want to do an end to end test to see that messages coming into
> > Kafka are being stored in HDFS with the minimal amount of code change
> > required.
> >
> > Thanks,
> >
> > Abhi
> >
> > --
> > Abhi Basu
> >
>
Re: Kafka Hadoop Consumer issues
Posted by Jun Rao <ju...@gmail.com>.
Which class is not found?
Thanks,
Jun
On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <90...@gmail.com> wrote:
> Let me describe my environment. Working on two nodes currently:
> 1.Single-node hadoop cluster (will refer as Node1)
> 2.Single node Kafka cluster (will refer as Node2)
>
> Node 2 has 1 broker started with a topic (iot.test.stream) and one command
> line producer and one command line consumer to test the kafka install.
> Producer can send messages and the Consumer is receiving it.
>
> Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have edited
> the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with the
> following:
>
> kafka.etl.topic=iot.test.stream
> hdfs.default.classpath.dir=/tmp/kafka/lib
> hadoop.job.ugi=kafka,hadoop
> kafka.server.uri=tcp://idh251-kafka:9095
> input=/tmp/kafka/data
> output=/tmp/kafka/output
> kafka.request.limit=-1
> ...........
>
> I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS)
>
> Next I run the following on Node 1:
> ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties from
> the /kafka-0.8/contrib/hadoop-consumer folder and get a
> classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class.
>
> What am I missing? I was thinking that running the sh file would allow me
> to retrieve messages with the same topic name to HDFS from Node 2 to Node
> 1. I just want to do an end to end test to see that messages coming into
> Kafka are being stored in HDFS with the minimal amount of code change
> required.
>
> Thanks,
>
> Abhi
>
> --
> Abhi Basu
>