You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Neha Narkhede <ne...@gmail.com> on 2011/08/10 01:29:47 UTC
Re: Kafka Sample Hadoop consumer.
Sam,
I tried this on a downloaded copy of kafka
v0.6<http://sna-projects.com/kafka/downloads/kafka-0.6.zip>-
nnarkhed-md:kafka nnarkhed$ jar tvf kafka-0.6.jar | grep
"kafka.javaapi.consumer.SimpleConsumer"
3501 Tue May 24 10:23:24 PDT 2011
kafka/javaapi/consumer/SimpleConsumer.class
I suspect that the kafka-0.6.jar is not registered correctly with the
DistributedCache.
Thanks,
Neha
On Tue, Aug 9, 2011 at 3:32 PM, Sam William <sa...@stumbleupon.com> wrote:
> Im trying to run this sample hadoop consumer in the 0.6 version . I see
> that the jar files (including kafka-0.6.jar) are proper when being copied to
> the DistributedCache , but I get the exception
>
> Error: java.lang.ClassNotFoundException:
> kafka.javaapi.consumer.SimpleConsumer
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at kafka.etl.KafkaETLContext.(KafkaETLContext.java:93)
> at
> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
> at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
>
> Is there something Im missing here ?
>
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>
Re: Kafka Sample Hadoop consumer.
Posted by Sam William <sa...@stumbleupon.com>.
I was using the kafka-v0.6 branch.. Seems like this Null pointer issue has been fixed on the master.
However, I see that , _response.iterator() method is being invoked multiple times on the same response object. This was what was causing the loops. Id have the iterator object as the class attribute instead of the _response object.
Sam
On Aug 11, 2011, at 2:07 PM, Sam William wrote:
> Neha,
> I followed the procedure described in
>
> https://github.com/kafka-dev/kafka/tree/kafka-v0.6/contrib/hadoop-consumer . I could nt get the distributed cache working . So, for the time being I went ahead with copying the jars to the tasknodes to get it to run.
>
>
> Im not sure , if this particular portion of the examples is completely tested or if I have a wrong version of the file ..
>
> For example .
>
> The file KafkaETLContext.java
>
>
> public boolean getNext(KafkaETLKey key, BytesWritable value) throws IOException {
> if ( !hasMore() ) return false;
>
> boolean gotNext = get(key, value);
>
> Iterator<ByteBufferMessageSet> iter = _response.iterator();
>
> while ( !gotNext && _response != null && iter.hasNext()) {
> ByteBufferMessageSet msgSet = iter.next();
> if ( hasError(msgSet)) return false;
> _messageIt = (Iterator<Message>) msgSet.iterator();
> gotNext = get(key, value);
> }
>
> return gotNext;
> }
>
>
>
>
>
> The null check for _response is comes after Iterator<ByteBufferMessageSet> iter = _response.iterator() and the first time this function is called , the _response object is null.
>
> Another question ,
> If i just generate say 2000 events . and set kafka.fetch.limit = 3000 or (-1) , Id expect the the job to stop after reading the 2000 events, but it seems to keep looping from the beginning. Am I missing something here ?
>
> Sam
>
>
>
>
>
>
>
>
>
>
> On Aug 9, 2011, at 4:29 PM, Neha Narkhede wrote:
>
>> Sam,
>>
>> I tried this on a downloaded copy of kafka
>> v0.6<http://sna-projects.com/kafka/downloads/kafka-0.6.zip>-
>>
>> nnarkhed-md:kafka nnarkhed$ jar tvf kafka-0.6.jar | grep
>> "kafka.javaapi.consumer.SimpleConsumer"
>> 3501 Tue May 24 10:23:24 PDT 2011
>> kafka/javaapi/consumer/SimpleConsumer.class
>>
>> I suspect that the kafka-0.6.jar is not registered correctly with the
>> DistributedCache.
>>
>> Thanks,
>> Neha
>>
>> On Tue, Aug 9, 2011 at 3:32 PM, Sam William <sa...@stumbleupon.com> wrote:
>>
>>> Im trying to run this sample hadoop consumer in the 0.6 version . I see
>>> that the jar files (including kafka-0.6.jar) are proper when being copied to
>>> the DistributedCache , but I get the exception
>>>
>>> Error: java.lang.ClassNotFoundException:
>>> kafka.javaapi.consumer.SimpleConsumer
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at kafka.etl.KafkaETLContext.(KafkaETLContext.java:93)
>>> at
>>> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
>>> at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
>>> at
>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>>> at
>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>
>>>
>>>
>>> Is there something Im missing here ?
>>>
>>>
>>> Sam William
>>> sampd@stumbleupon.com
>>>
>>>
>>>
>>>
>
> Sam William
> sampd@stumbleupon.com
>
>
>
Sam William
sampd@stumbleupon.com
Re: Kafka Sample Hadoop consumer.
Posted by Sam William <sa...@stumbleupon.com>.
Neha,
I followed the procedure described in
https://github.com/kafka-dev/kafka/tree/kafka-v0.6/contrib/hadoop-consumer . I could nt get the distributed cache working . So, for the time being I went ahead with copying the jars to the tasknodes to get it to run.
Im not sure , if this particular portion of the examples is completely tested or if I have a wrong version of the file ..
For example .
The file KafkaETLContext.java
public boolean getNext(KafkaETLKey key, BytesWritable value) throws IOException {
if ( !hasMore() ) return false;
boolean gotNext = get(key, value);
Iterator<ByteBufferMessageSet> iter = _response.iterator();
while ( !gotNext && _response != null && iter.hasNext()) {
ByteBufferMessageSet msgSet = iter.next();
if ( hasError(msgSet)) return false;
_messageIt = (Iterator<Message>) msgSet.iterator();
gotNext = get(key, value);
}
return gotNext;
}
The null check for _response is comes after Iterator<ByteBufferMessageSet> iter = _response.iterator() and the first time this function is called , the _response object is null.
Another question ,
If i just generate say 2000 events . and set kafka.fetch.limit = 3000 or (-1) , Id expect the the job to stop after reading the 2000 events, but it seems to keep looping from the beginning. Am I missing something here ?
Sam
On Aug 9, 2011, at 4:29 PM, Neha Narkhede wrote:
> Sam,
>
> I tried this on a downloaded copy of kafka
> v0.6<http://sna-projects.com/kafka/downloads/kafka-0.6.zip>-
>
> nnarkhed-md:kafka nnarkhed$ jar tvf kafka-0.6.jar | grep
> "kafka.javaapi.consumer.SimpleConsumer"
> 3501 Tue May 24 10:23:24 PDT 2011
> kafka/javaapi/consumer/SimpleConsumer.class
>
> I suspect that the kafka-0.6.jar is not registered correctly with the
> DistributedCache.
>
> Thanks,
> Neha
>
> On Tue, Aug 9, 2011 at 3:32 PM, Sam William <sa...@stumbleupon.com> wrote:
>
>> Im trying to run this sample hadoop consumer in the 0.6 version . I see
>> that the jar files (including kafka-0.6.jar) are proper when being copied to
>> the DistributedCache , but I get the exception
>>
>> Error: java.lang.ClassNotFoundException:
>> kafka.javaapi.consumer.SimpleConsumer
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> at kafka.etl.KafkaETLContext.(KafkaETLContext.java:93)
>> at
>> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
>> at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
>> at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>> at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>>
>>
>> Is there something Im missing here ?
>>
>>
>> Sam William
>> sampd@stumbleupon.com
>>
>>
>>
>>
Sam William
sampd@stumbleupon.com