You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ben Ciceron <be...@triggit.com> on 2011/08/26 01:31:35 UTC
jobtracker / hadoop comsumer
Hello,
does kafka hadoop consumer expect the jobtracker to run locally only ?
it seems it expect it locally (localhost/127.0.0.1:9001) .
Is it a requirement or there is a way to change it to a remote uri ?
Cheers,
Ben-
Re: jobtracker / hadoop comsumer
Posted by Ben Ciceron <be...@triggit.com>.
yes!
On Thu, Sep 1, 2011 at 10:56 AM, Sam William <sa...@stumbleupon.com> wrote:
> Ben,
> There is a bug in KafkaETLContext.java in where the iterator is created multiple times. You ll have this issue only when kafka.request.limit is set to -1 in your test.properties.
>
> Line 120 Iterator<ByteBufferMessageSet> iter = _response.iterator();
>
>
> This is what is causing the loops . In my setup , I fixed it by having a _respIterator attribute in the class and assigning
> it to _response.iterator() just once in the fetchMore() method. I use the class attribute in the rest of the places.
>
> Hope this helps.
>
> Sam
>
>
> On Sep 1, 2011, at 9:18 AM, Ben Ciceron wrote:
>
>> hello,
>>
>> i am able to run the job now but it loops indefinitely.
>>
>> can someone send a valid sample of 1.dat as generated by
>> kafka.etl.impl.DataGenerator in the README, please ?
>> mine seems awfully short (see below).
>>
>>>> 3. 1.dat is a sequence file, so if it isn't compressed, you should be
>>>> able to see its contents in plain text. The contents will essentially list
>>>> the kafka server url, the partition number and the topic as well as the
>>>> offset.
>>>
>>>
>>> mine has only 1 line (some encrypted in the middle of it is shown as is):
>>> SEQkafka.etl.KafkaETLKey"org.apache.hadoop.io.BytesWritable$?ړ??tl?3aC+tcp://<ip_of_my_hostA>:9092 SimpleTestEvent 0 -1
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>
Re: jobtracker / hadoop comsumer
Posted by Sam William <sa...@stumbleupon.com>.
Sure
On Sep 1, 2011, at 1:02 PM, Jun Rao wrote:
> Sam,
>
> Could you provide a patch for that? Thanks,
>
> Jun
>
> On Thu, Sep 1, 2011 at 10:56 AM, Sam William <sa...@stumbleupon.com> wrote:
>
>> Ben,
>> There is a bug in KafkaETLContext.java in where the iterator is created
>> multiple times. You ll have this issue only when kafka.request.limit is set
>> to -1 in your test.properties.
>>
>> Line 120 Iterator<ByteBufferMessageSet> iter = _response.iterator();
>>
>>
>> This is what is causing the loops . In my setup , I fixed it by having a
>> _respIterator attribute in the class and assigning
>> it to _response.iterator() just once in the fetchMore() method. I use the
>> class attribute in the rest of the places.
>>
>> Hope this helps.
>>
>> Sam
>>
>>
>> On Sep 1, 2011, at 9:18 AM, Ben Ciceron wrote:
>>
>>> hello,
>>>
>>> i am able to run the job now but it loops indefinitely.
>>>
>>> can someone send a valid sample of 1.dat as generated by
>>> kafka.etl.impl.DataGenerator in the README, please ?
>>> mine seems awfully short (see below).
>>>
>>>>> 3. 1.dat is a sequence file, so if it isn't compressed, you should be
>>>>> able to see its contents in plain text. The contents will essentially
>> list
>>>>> the kafka server url, the partition number and the topic as well as
>> the
>>>>> offset.
>>>>
>>>>
>>>> mine has only 1 line (some encrypted in the middle of it is shown as
>> is):
>>>>
>> SEQkafka.etl.KafkaETLKey"org.apache.hadoop.io.BytesWritable$?ړ??tl?3aC+tcp://<ip_of_my_hostA>:9092
>> SimpleTestEvent 0 -1
>>
>> Sam William
>> sampd@stumbleupon.com
>>
>>
>>
>>
Sam William
sampd@stumbleupon.com
Re: jobtracker / hadoop comsumer
Posted by Jun Rao <ju...@gmail.com>.
Sam,
Could you provide a patch for that? Thanks,
Jun
On Thu, Sep 1, 2011 at 10:56 AM, Sam William <sa...@stumbleupon.com> wrote:
> Ben,
> There is a bug in KafkaETLContext.java in where the iterator is created
> multiple times. You ll have this issue only when kafka.request.limit is set
> to -1 in your test.properties.
>
> Line 120 Iterator<ByteBufferMessageSet> iter = _response.iterator();
>
>
> This is what is causing the loops . In my setup , I fixed it by having a
> _respIterator attribute in the class and assigning
> it to _response.iterator() just once in the fetchMore() method. I use the
> class attribute in the rest of the places.
>
> Hope this helps.
>
> Sam
>
>
> On Sep 1, 2011, at 9:18 AM, Ben Ciceron wrote:
>
> > hello,
> >
> > i am able to run the job now but it loops indefinitely.
> >
> > can someone send a valid sample of 1.dat as generated by
> > kafka.etl.impl.DataGenerator in the README, please ?
> > mine seems awfully short (see below).
> >
> >>> 3. 1.dat is a sequence file, so if it isn't compressed, you should be
> >>> able to see its contents in plain text. The contents will essentially
> list
> >>> the kafka server url, the partition number and the topic as well as
> the
> >>> offset.
> >>
> >>
> >> mine has only 1 line (some encrypted in the middle of it is shown as
> is):
> >>
> SEQkafka.etl.KafkaETLKey"org.apache.hadoop.io.BytesWritable$?ړ??tl?3aC+tcp://<ip_of_my_hostA>:9092
> SimpleTestEvent 0 -1
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>
Re: jobtracker / hadoop comsumer
Posted by Sam William <sa...@stumbleupon.com>.
Ben,
There is a bug in KafkaETLContext.java in where the iterator is created multiple times. You ll have this issue only when kafka.request.limit is set to -1 in your test.properties.
Line 120 Iterator<ByteBufferMessageSet> iter = _response.iterator();
This is what is causing the loops . In my setup , I fixed it by having a _respIterator attribute in the class and assigning
it to _response.iterator() just once in the fetchMore() method. I use the class attribute in the rest of the places.
Hope this helps.
Sam
On Sep 1, 2011, at 9:18 AM, Ben Ciceron wrote:
> hello,
>
> i am able to run the job now but it loops indefinitely.
>
> can someone send a valid sample of 1.dat as generated by
> kafka.etl.impl.DataGenerator in the README, please ?
> mine seems awfully short (see below).
>
>>> 3. 1.dat is a sequence file, so if it isn't compressed, you should be
>>> able to see its contents in plain text. The contents will essentially list
>>> the kafka server url, the partition number and the topic as well as the
>>> offset.
>>
>>
>> mine has only 1 line (some encrypted in the middle of it is shown as is):
>> SEQkafka.etl.KafkaETLKey"org.apache.hadoop.io.BytesWritable$?ړ??tl?3aC+tcp://<ip_of_my_hostA>:9092 SimpleTestEvent 0 -1
Sam William
sampd@stumbleupon.com
Re: jobtracker / hadoop comsumer
Posted by Ben Ciceron <be...@triggit.com>.
hello,
i am able to run the job now but it loops indefinitely.
can someone send a valid sample of 1.dat as generated by
kafka.etl.impl.DataGenerator in the README, please ?
mine seems awfully short (see below).
>> 3. 1.dat is a sequence file, so if it isn't compressed, you should be
>> able to see its contents in plain text. The contents will essentially list
>> the kafka server url, the partition number and the topic as well as the
>> offset.
>
>
> mine has only 1 line (some encrypted in the middle of it is shown as is):
> SEQkafka.etl.KafkaETLKey"org.apache.hadoop.io.BytesWritable$?ړ??tl?3aC+tcp://<ip_of_my_hostA>:9092 SimpleTestEvent 0 -1
Re: jobtracker / hadoop comsumer
Posted by Ben Ciceron <be...@triggit.com>.
> It really looks like your mapper tasks may be failing to connect to your
> kafka server.
ok
> Here's a brief overview of what that demo job is doing so you can understand
> where the example may have gone wrong.
> DataGenerator:
>
> 1. When DataGenerator is run, it needs the property 'kafka.etl.topic',
> and 'kafka.server.uri' set in the properties file. When you run
> ./run-class.sh
> kafka.etl.impl.DataGenerator test/test.properties, you can tell that
> they're properly set by the output 'topic=<blah>' and 'server uri=<kafka
> server url>.
seems ok :
topics=SimpleTestEvent
server uri:tcp://<ip_of_my_hostA>:9092
send 1000 SimpleTestEvent count events to tcp://<ip_of_my_hostA>:9092
11/09/01 05:04:05 INFO producer.SyncProducer: Connected to
<ip_of_my_hostA>:9092 for producing
11/09/01 05:04:05 INFO producer.SyncProducer: Disconnecting from
<ip_of_my_hostA>:9092
Dump tcp://<ip_of_my_hostA>:9092 SimpleTestEvent 0 -1 to /tmp/ben6/data/1.dat
> 2. The DataGenerator will create a bunch of dummy messages and pump it to
> that kafka server. Afterwards, it will write a file to HDFS at path 'input'
> which you also set in the properties file. The file that is created will be
> named something like 1.dat.
yes , i see it under hadoop directory as specified as 'input' in
test.properties.
> 3. 1.dat is a sequence file, so if it isn't compressed, you should be
> able to see its contents in plain text. The contents will essentially list
> the kafka server url, the partition number and the topic as well as the
> offset.
mine has only 1 line (some encrypted in the middle of it is shown as is):
SEQkafka.etl.KafkaETLKey"org.apache.hadoop.io.BytesWritable$?ړ??tl?3aC+tcp://<ip_of_my_hostA>:9092 SimpleTestEvent 0 -1
> 4. In a real scenario, you'll probably create several of these files for
> each broker and possibly partition, but for this example, you only need one
> file. Each file will spawn a mapper during the mapred step.
>
> CopyJars:
>
> 1. This should copy the necessary jars for kafka hadoop, and push them
> into HDFS for the distributed cache. If the jars are copied locally instead
> of to a remote cluster, most likely HADOOP_CONF_DIR hasn't been set up
> correctly. The environment should probably be set by the script, so someone
> can change that.
yes i've got the proper jars under the hadoop directory now.
> SimpleKafkaETLJob
>
> 1. This job will then setup the distributed classpath, and the input path
> should be the directory that 1.dat was written to.
> 2. Internally, the mappers will then load 1.dat and use the connection
> data contained in it to connect to kafka. If it's trying to connect to
> anything but your kafka server, than this file was incorrectly written.
can we see a sample of a valid 1.dat file , please ?
> 3. The RecordReader wraps all of this and hides all the connection stuff
> so that your Mapper should see a stream of Kafka messages rather than the
> contents of 1.dat.
>
> So please see if you can figure out what is wrong with your example and feel
> free beef up the README instructions to take in account your pitfalls.
ok , yes i plan to do this once i got all the steps right.
thx for all your support so far.
Re: jobtracker / hadoop comsumer
Posted by Richard Park <ri...@gmail.com>.
That part might not be written in the simple example.
You may need to manually move that offset file to an empty input directory
and have your map reduce job use that as your input path. That offset file
is similar format to the 1.dat file that was created by the DataGenerator
and should be compatible as input for subsequent runs.
If you look inside the file, you may see that the only difference between
the output file and the 1.dat is the offset value (it'll be greater than
-1).
This is essentially what I did for a fully functional version of this kafka
hadoop consumer.
1. Read previous offset files if they exist.
2. Write/copy offset files to new directory and rename if necessary.
3. Run job against new directory. I usually have the kafka data output to
the same directory as input. The reason is that I can just delete the
directory and have the hadoop job repull the data from previous hadoop job
without having to do complex offset management.
4. Repeat 1.
Hope this clarifies things.
-Richard
On Wed, Aug 31, 2011 at 4:40 PM, Sam William <sa...@stumbleupon.com> wrote:
> Richard,
> I read somewhere that the mappers write out the offset to the output
> dir , so that further attempts (after a task failure) can start from the
> right offset.
> I see that the offset is generated . But where is the logic to read this
> and adjust the offset for the next read ? . I wasnt able to find it.
>
> Sam
>
> On Aug 31, 2011, at 12:48 PM, Richard Park wrote:
>
> > It really looks like your mapper tasks may be failing to connect to your
> > kafka server.
> >
> > Here's a brief overview of what that demo job is doing so you can
> understand
> > where the example may have gone wrong.
> > DataGenerator:
> >
> > 1. When DataGenerator is run, it needs the property 'kafka.etl.topic',
> > and 'kafka.server.uri' set in the properties file. When you run
> > ./run-class.sh
> > kafka.etl.impl.DataGenerator test/test.properties, you can tell that
> > they're properly set by the output 'topic=<blah>' and 'server
> uri=<kafka
> > server url>.
> > 2. The DataGenerator will create a bunch of dummy messages and pump it
> to
> > that kafka server. Afterwards, it will write a file to HDFS at path
> 'input'
> > which you also set in the properties file. The file that is created
> will be
> > named something like 1.dat.
> > 3. 1.dat is a sequence file, so if it isn't compressed, you should be
> > able to see its contents in plain text. The contents will essentially
> list
> > the kafka server url, the partition number and the topic as well as the
> > offset.
> > 4. In a real scenario, you'll probably create several of these files
> for
> > each broker and possibly partition, but for this example, you only need
> one
> > file. Each file will spawn a mapper during the mapred step.
> >
> > CopyJars:
> >
> > 1. This should copy the necessary jars for kafka hadoop, and push them
> > into HDFS for the distributed cache. If the jars are copied locally
> instead
> > of to a remote cluster, most likely HADOOP_CONF_DIR hasn't been set up
> > correctly. The environment should probably be set by the script, so
> someone
> > can change that.
> >
> > SimpleKafkaETLJob
> >
> > 1. This job will then setup the distributed classpath, and the input
> path
> > should be the directory that 1.dat was written to.
> > 2. Internally, the mappers will then load 1.dat and use the connection
> > data contained in it to connect to kafka. If it's trying to connect to
> > anything but your kafka server, than this file was incorrectly written.
> > 3. The RecordReader wraps all of this and hides all the connection
> stuff
> > so that your Mapper should see a stream of Kafka messages rather than
> the
> > contents of 1.dat.
> >
> > So please see if you can figure out what is wrong with your example and
> feel
> > free beef up the README instructions to take in account your pitfalls.
> >
> > Thanks,
> > -Richard
> >
> >
> >
> > On Wed, Aug 31, 2011 at 12:02 PM, Ben Ciceron <be...@triggit.com> wrote:
> >
> >> ok i could live with setting mapred.job.tracker manually for the code
> for
> >> now.
> >> This way it can connect now to the proper jobtracker.
> >>
> >>> The hadoop map tasks will need to connect to the kafka server port (the
> >>> broker uri/port).
> >>
> >> i run the hadoop soncumer on the same hostA where the kafka-server is
> >> running.
> >>
> >> each of the host in the hadoop cluster can telnet/nmap to port 9092 on
> >> hostA where the kafka-server is running.
> >> also HostA can connect to port 5181 on any host in the cluster.
> >>
> >> but each task fails with a similar connection issue :
> >> java.io.IOException: java.net.ConnectException: Connection refused
> >> at
> >> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:155)
> >> at
> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> >> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:396)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> >> Caused by: java.net.ConnectException: Connection refused
> >> at sun.nio.ch.Net.connect(Native Method)
> >> at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
> >> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:54)
> >> at
> >>
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:193)
> >> at
> >> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:156)
> >> at
> >>
> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:65)
> >> at
> >> kafka.etl.KafkaETLContext.getOffsetRange(KafkaETLContext.java:209)
> >> at kafka.etl.KafkaETLContext.<init>(KafkaETLContext.java:97)
> >> at
> >> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
> >> ... 11 more
> >>
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>
Re: jobtracker / hadoop comsumer
Posted by Sam William <sa...@stumbleupon.com>.
Richard,
I read somewhere that the mappers write out the offset to the output dir , so that further attempts (after a task failure) can start from the right offset.
I see that the offset is generated . But where is the logic to read this and adjust the offset for the next read ? . I wasnt able to find it.
Sam
On Aug 31, 2011, at 12:48 PM, Richard Park wrote:
> It really looks like your mapper tasks may be failing to connect to your
> kafka server.
>
> Here's a brief overview of what that demo job is doing so you can understand
> where the example may have gone wrong.
> DataGenerator:
>
> 1. When DataGenerator is run, it needs the property 'kafka.etl.topic',
> and 'kafka.server.uri' set in the properties file. When you run
> ./run-class.sh
> kafka.etl.impl.DataGenerator test/test.properties, you can tell that
> they're properly set by the output 'topic=<blah>' and 'server uri=<kafka
> server url>.
> 2. The DataGenerator will create a bunch of dummy messages and pump it to
> that kafka server. Afterwards, it will write a file to HDFS at path 'input'
> which you also set in the properties file. The file that is created will be
> named something like 1.dat.
> 3. 1.dat is a sequence file, so if it isn't compressed, you should be
> able to see its contents in plain text. The contents will essentially list
> the kafka server url, the partition number and the topic as well as the
> offset.
> 4. In a real scenario, you'll probably create several of these files for
> each broker and possibly partition, but for this example, you only need one
> file. Each file will spawn a mapper during the mapred step.
>
> CopyJars:
>
> 1. This should copy the necessary jars for kafka hadoop, and push them
> into HDFS for the distributed cache. If the jars are copied locally instead
> of to a remote cluster, most likely HADOOP_CONF_DIR hasn't been set up
> correctly. The environment should probably be set by the script, so someone
> can change that.
>
> SimpleKafkaETLJob
>
> 1. This job will then setup the distributed classpath, and the input path
> should be the directory that 1.dat was written to.
> 2. Internally, the mappers will then load 1.dat and use the connection
> data contained in it to connect to kafka. If it's trying to connect to
> anything but your kafka server, than this file was incorrectly written.
> 3. The RecordReader wraps all of this and hides all the connection stuff
> so that your Mapper should see a stream of Kafka messages rather than the
> contents of 1.dat.
>
> So please see if you can figure out what is wrong with your example and feel
> free beef up the README instructions to take in account your pitfalls.
>
> Thanks,
> -Richard
>
>
>
> On Wed, Aug 31, 2011 at 12:02 PM, Ben Ciceron <be...@triggit.com> wrote:
>
>> ok i could live with setting mapred.job.tracker manually for the code for
>> now.
>> This way it can connect now to the proper jobtracker.
>>
>>> The hadoop map tasks will need to connect to the kafka server port (the
>>> broker uri/port).
>>
>> i run the hadoop soncumer on the same hostA where the kafka-server is
>> running.
>>
>> each of the host in the hadoop cluster can telnet/nmap to port 9092 on
>> hostA where the kafka-server is running.
>> also HostA can connect to port 5181 on any host in the cluster.
>>
>> but each task fails with a similar connection issue :
>> java.io.IOException: java.net.ConnectException: Connection refused
>> at
>> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:155)
>> at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
>> at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
>> at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> Caused by: java.net.ConnectException: Connection refused
>> at sun.nio.ch.Net.connect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
>> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:54)
>> at
>> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:193)
>> at
>> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:156)
>> at
>> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:65)
>> at
>> kafka.etl.KafkaETLContext.getOffsetRange(KafkaETLContext.java:209)
>> at kafka.etl.KafkaETLContext.<init>(KafkaETLContext.java:97)
>> at
>> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
>> ... 11 more
>>
Sam William
sampd@stumbleupon.com
Re: jobtracker / hadoop comsumer
Posted by Richard Park <ri...@gmail.com>.
It really looks like your mapper tasks may be failing to connect to your
kafka server.
Here's a brief overview of what that demo job is doing so you can understand
where the example may have gone wrong.
DataGenerator:
1. When DataGenerator is run, it needs the property 'kafka.etl.topic',
and 'kafka.server.uri' set in the properties file. When you run
./run-class.sh
kafka.etl.impl.DataGenerator test/test.properties, you can tell that
they're properly set by the output 'topic=<blah>' and 'server uri=<kafka
server url>.
2. The DataGenerator will create a bunch of dummy messages and pump it to
that kafka server. Afterwards, it will write a file to HDFS at path 'input'
which you also set in the properties file. The file that is created will be
named something like 1.dat.
3. 1.dat is a sequence file, so if it isn't compressed, you should be
able to see its contents in plain text. The contents will essentially list
the kafka server url, the partition number and the topic as well as the
offset.
4. In a real scenario, you'll probably create several of these files for
each broker and possibly partition, but for this example, you only need one
file. Each file will spawn a mapper during the mapred step.
CopyJars:
1. This should copy the necessary jars for kafka hadoop, and push them
into HDFS for the distributed cache. If the jars are copied locally instead
of to a remote cluster, most likely HADOOP_CONF_DIR hasn't been set up
correctly. The environment should probably be set by the script, so someone
can change that.
SimpleKafkaETLJob
1. This job will then setup the distributed classpath, and the input path
should be the directory that 1.dat was written to.
2. Internally, the mappers will then load 1.dat and use the connection
data contained in it to connect to kafka. If it's trying to connect to
anything but your kafka server, than this file was incorrectly written.
3. The RecordReader wraps all of this and hides all the connection stuff
so that your Mapper should see a stream of Kafka messages rather than the
contents of 1.dat.
So please see if you can figure out what is wrong with your example and feel
free beef up the README instructions to take in account your pitfalls.
Thanks,
-Richard
On Wed, Aug 31, 2011 at 12:02 PM, Ben Ciceron <be...@triggit.com> wrote:
> ok i could live with setting mapred.job.tracker manually for the code for
> now.
> This way it can connect now to the proper jobtracker.
>
> > The hadoop map tasks will need to connect to the kafka server port (the
> > broker uri/port).
>
> i run the hadoop soncumer on the same hostA where the kafka-server is
> running.
>
> each of the host in the hadoop cluster can telnet/nmap to port 9092 on
> hostA where the kafka-server is running.
> also HostA can connect to port 5181 on any host in the cluster.
>
> but each task fails with a similar connection issue :
> java.io.IOException: java.net.ConnectException: Connection refused
> at
> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:155)
> at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:54)
> at
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:193)
> at
> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:156)
> at
> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:65)
> at
> kafka.etl.KafkaETLContext.getOffsetRange(KafkaETLContext.java:209)
> at kafka.etl.KafkaETLContext.<init>(KafkaETLContext.java:97)
> at
> kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
> ... 11 more
>
Re: jobtracker / hadoop comsumer
Posted by Ben Ciceron <be...@triggit.com>.
ok i could live with setting mapred.job.tracker manually for the code for now.
This way it can connect now to the proper jobtracker.
> The hadoop map tasks will need to connect to the kafka server port (the
> broker uri/port).
i run the hadoop soncumer on the same hostA where the kafka-server is running.
each of the host in the hadoop cluster can telnet/nmap to port 9092 on
hostA where the kafka-server is running.
also HostA can connect to port 5181 on any host in the cluster.
but each task fails with a similar connection issue :
java.io.IOException: java.net.ConnectException: Connection refused
at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:155)
at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:54)
at kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:193)
at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:156)
at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:65)
at kafka.etl.KafkaETLContext.getOffsetRange(KafkaETLContext.java:209)
at kafka.etl.KafkaETLContext.<init>(KafkaETLContext.java:97)
at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
... 11 more
Re: jobtracker / hadoop comsumer
Posted by Ben Ciceron <be...@triggit.com>.
> I think some changes will be needed with the instructions.
thx Richard, i apreciate your effort.
yes as a newbie i struggle with those but i'd be happy to participate
in your effort to make them better .
>
> If you have a local Hadoop installed then it goes to the local job tracker.
> You'll have to set up your Hadoop to point to the correct cluster. There are
> several ways to do it, but you can set your HADOOP_CONF_DIR to point to the
> correct hadoop conf xml (ie. location of mapred-site.xml) for the remote
> clusters.
ok , can we see an example ? i just tried it with several way to
define mapred.job.tracker inside
/opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml as explained in
the embeded comment then pointing HADOOP_CONF_DIR to
/opt/mapr/hadoop/hadoop-0.20.2/conf in the environment but calling the
hadoop consumer still call the default local jobtracker and fails:
...
11/08/31 12:33:48 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9001. Already tried 0 time(s).
11/08/31 12:33:49 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9001. Already tried 1 time(s).
...
>
> The hadoop map tasks will need to connect to the kafka server port (the
> broker uri/port).
>
>
> On Tue, Aug 30, 2011 at 4:50 PM, Ben Ciceron <be...@triggit.com> wrote:
>
>> thx for confirming.
>>
>> so when follow the instructions to run the hadoop consumer
>> (https://github.com/kafka-dev/kafka/tree/master/contrib/hadoop-consumer)
>> i see my mapred job being submitted properly on hostB (jobtracker) but
>> it always fails with :
>>
>> console output on hostA
>>
>> 11/08/31 07:32:22 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the
>> same.
>> 11/08/31 07:32:35 INFO mapred.FileInputFormat: Total input paths to process
>> : 1
>> Hadoop job id=job_201108291829_0041
>> Exception in thread "main" java.lang.Exception: Hadoop ETL job failed!
>> Please check status on
>> http://localhost:9001/jobdetails.jsp?jobid=job_201108291829_0041
>> at
>> kafka.etl.impl.SimpleKafkaETLJob.execute(SimpleKafkaETLJob.java:82)
>> at kafka.etl.impl.SimpleKafkaETLJob.main(SimpleKafkaETLJob.java:100)
>>
>> in mapred log, for each tasktracker host i see :
>>
>> Meta VERSION="1" .
>> Job JOBID="job_201108291829_0041" JOBNAME="SimpleKafakETL" USER="root"
>> SUBMIT_TIME="1314747158794"
>>
>> JOBCONF="maprfs://10\.18\.125\.176:7222/var/mapr/cluster/mapred/jobTracker/staging/root/\.staging/job_201108291829_0041/job\.xml"
>> VIEW_JOB="*" MODIFY_JOB="*" JOB_QUEUE="default" .
>> Job JOBID="job_201108291829_0041" JOB_PRIORITY="NORMAL" .
>> Job JOBID="job_201108291829_0041" JOB_STATUS="RUNNING" .
>> Job JOBID="job_201108291829_0041" LAUNCH_TIME="1314747158885"
>> TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
>> Task TASKID="task_201108291829_0041_m_000000" TASK_TYPE="MAP"
>> START_TIME="1314747160010"
>> SPLITS="/default-rack/hadoop2,/default-rack/hadoop9,/default-rack/hadoop6"
>> .
>> MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
>> TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
>> START_TIME="1314747160121"
>> TRACKER_NAME="tracker_hadoop9:localhost/127\.0\.0\.1:59411"
>> HTTP_PORT="50060" .
>> MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
>> TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
>> TASK_STATUS="FAILED" FINISH_TIME="1314747164349" HOSTNAME="hadoop9"
>> ERROR="java\.io\.IOException: java\.net\.ConnectException: Connection
>> refused
>> at
>> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:155)
>> at
>> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:14)
>> at
>> org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.moveToNext(MapTask\.java:210)
>> at
>> org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.next(MapTask\.java:195)
>> at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:48)
>> at
>> org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:393)
>> at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:326)
>> at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:268)
>> at java\.security\.AccessController\.doPrivileged(Native Method)
>> at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
>> at
>> org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1074)
>> at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:262)
>> Caused by: java\.net\.ConnectException: Connection refused
>> at sun\.nio\.ch\.Net\.connect(Native Method)
>> at
>> sun\.nio\.ch\.SocketChannelImpl\.connect(SocketChannelImpl\.java:500)
>> at
>> kafka\.consumer\.SimpleConsumer\.connect(SimpleConsumer\.scala:54)
>> at
>> kafka\.consumer\.SimpleConsumer\.getOrMakeConnection(SimpleConsumer\.scala:193)
>> at
>> kafka\.consumer\.SimpleConsumer\.getOffsetsBefore(SimpleConsumer\.scala:156)
>> at
>> kafka\.javaapi\.consumer\.SimpleConsumer\.getOffsetsBefore(SimpleConsumer\.scala:65)
>> at
>> kafka\.etl\.KafkaETLContext\.getOffsetRange(KafkaETLContext\.java:209)
>> at kafka\.etl\.KafkaETLContext\.<init>(KafkaETLContext\.java:97)
>> at
>> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:115)
>> \.\.\. 11 more
>>
>> which other port do i need to open between hostA and the tasktrackers ?
>> please note i can send a simple non kafka job from the same hostA to
>> hadoop and it completes successfully.
>>
>> Cheers,
>> Ben-
>>
>>
>>
>>
>> On Tue, Aug 30, 2011 at 12:04 PM, Richard Park <ri...@gmail.com>
>> wrote:
>> > The answer should be yes, each process should be able to run on different
>> > hosts. We are currently doing this.
>> >
>> > Host A submits kafka hadoop job to the job tracker on Host B,
>> > Host B then then connects to Host C (or many host C's)
>> >
>> > I planned on having a look at the example again to see if there are steps
>> > there are missing, or if the examples need to be beefed up.
>> >
>> > Thanks,
>> > -Richard
>> >
>> >
>> > On Tue, Aug 30, 2011 at 11:39 AM, Ben Ciceron <be...@triggit.com> wrote:
>> >
>> >> let me rephrase this:
>> >>
>> >> can any of the kafka process run outside the hadoop cluster as long as
>> >> it can connect to the hadoop process from that host ?
>> >> e.g :
>> >>
>> >> hostA (NOT in th hadoop cluster) : runs kafka hadoop consumer
>> >> hostB (in th hadoop cluster) : runs jobtracker
>> >>
>> >>
>> >> Cheers,
>> >> Ben-
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Aug 29, 2011 at 4:59 PM, Jun Rao <ju...@gmail.com> wrote:
>> >> > My understanding is that it's not tied to localhost. You just need to
>> >> change
>> >> > the jobtracker setting in you Hadoop config.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Jun
>> >> >
>> >> > On Thu, Aug 25, 2011 at 4:31 PM, Ben Ciceron <be...@triggit.com> wrote:
>> >> >
>> >> >> Hello,
>> >> >>
>> >> >> does kafka hadoop consumer expect the jobtracker to run locally only
>> ?
>> >> >> it seems it expect it locally (localhost/127.0.0.1:9001) .
>> >> >> Is it a requirement or there is a way to change it to a remote uri ?
>> >> >>
>> >> >> Cheers,
>> >> >> Ben-
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: jobtracker / hadoop comsumer
Posted by Richard Park <ri...@gmail.com>.
I think some changes will be needed with the instructions.
If you have a local Hadoop installed then it goes to the local job tracker.
You'll have to set up your Hadoop to point to the correct cluster. There are
several ways to do it, but you can set your HADOOP_CONF_DIR to point to the
correct hadoop conf xml (ie. location of mapred-site.xml) for the remote
clusters.
The hadoop map tasks will need to connect to the kafka server port (the
broker uri/port).
On Tue, Aug 30, 2011 at 4:50 PM, Ben Ciceron <be...@triggit.com> wrote:
> thx for confirming.
>
> so when follow the instructions to run the hadoop consumer
> (https://github.com/kafka-dev/kafka/tree/master/contrib/hadoop-consumer)
> i see my mapred job being submitted properly on hostB (jobtracker) but
> it always fails with :
>
> console output on hostA
>
> 11/08/31 07:32:22 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> 11/08/31 07:32:35 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> Hadoop job id=job_201108291829_0041
> Exception in thread "main" java.lang.Exception: Hadoop ETL job failed!
> Please check status on
> http://localhost:9001/jobdetails.jsp?jobid=job_201108291829_0041
> at
> kafka.etl.impl.SimpleKafkaETLJob.execute(SimpleKafkaETLJob.java:82)
> at kafka.etl.impl.SimpleKafkaETLJob.main(SimpleKafkaETLJob.java:100)
>
> in mapred log, for each tasktracker host i see :
>
> Meta VERSION="1" .
> Job JOBID="job_201108291829_0041" JOBNAME="SimpleKafakETL" USER="root"
> SUBMIT_TIME="1314747158794"
>
> JOBCONF="maprfs://10\.18\.125\.176:7222/var/mapr/cluster/mapred/jobTracker/staging/root/\.staging/job_201108291829_0041/job\.xml"
> VIEW_JOB="*" MODIFY_JOB="*" JOB_QUEUE="default" .
> Job JOBID="job_201108291829_0041" JOB_PRIORITY="NORMAL" .
> Job JOBID="job_201108291829_0041" JOB_STATUS="RUNNING" .
> Job JOBID="job_201108291829_0041" LAUNCH_TIME="1314747158885"
> TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
> Task TASKID="task_201108291829_0041_m_000000" TASK_TYPE="MAP"
> START_TIME="1314747160010"
> SPLITS="/default-rack/hadoop2,/default-rack/hadoop9,/default-rack/hadoop6"
> .
> MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
> TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
> START_TIME="1314747160121"
> TRACKER_NAME="tracker_hadoop9:localhost/127\.0\.0\.1:59411"
> HTTP_PORT="50060" .
> MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
> TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
> TASK_STATUS="FAILED" FINISH_TIME="1314747164349" HOSTNAME="hadoop9"
> ERROR="java\.io\.IOException: java\.net\.ConnectException: Connection
> refused
> at
> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:155)
> at
> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:14)
> at
> org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.moveToNext(MapTask\.java:210)
> at
> org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.next(MapTask\.java:195)
> at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:48)
> at
> org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:393)
> at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:326)
> at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:268)
> at java\.security\.AccessController\.doPrivileged(Native Method)
> at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
> at
> org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1074)
> at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:262)
> Caused by: java\.net\.ConnectException: Connection refused
> at sun\.nio\.ch\.Net\.connect(Native Method)
> at
> sun\.nio\.ch\.SocketChannelImpl\.connect(SocketChannelImpl\.java:500)
> at
> kafka\.consumer\.SimpleConsumer\.connect(SimpleConsumer\.scala:54)
> at
> kafka\.consumer\.SimpleConsumer\.getOrMakeConnection(SimpleConsumer\.scala:193)
> at
> kafka\.consumer\.SimpleConsumer\.getOffsetsBefore(SimpleConsumer\.scala:156)
> at
> kafka\.javaapi\.consumer\.SimpleConsumer\.getOffsetsBefore(SimpleConsumer\.scala:65)
> at
> kafka\.etl\.KafkaETLContext\.getOffsetRange(KafkaETLContext\.java:209)
> at kafka\.etl\.KafkaETLContext\.<init>(KafkaETLContext\.java:97)
> at
> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:115)
> \.\.\. 11 more
>
> which other port do i need to open between hostA and the tasktrackers ?
> please note i can send a simple non kafka job from the same hostA to
> hadoop and it completes successfully.
>
> Cheers,
> Ben-
>
>
>
>
> On Tue, Aug 30, 2011 at 12:04 PM, Richard Park <ri...@gmail.com>
> wrote:
> > The answer should be yes, each process should be able to run on different
> > hosts. We are currently doing this.
> >
> > Host A submits kafka hadoop job to the job tracker on Host B,
> > Host B then then connects to Host C (or many host C's)
> >
> > I planned on having a look at the example again to see if there are steps
> > there are missing, or if the examples need to be beefed up.
> >
> > Thanks,
> > -Richard
> >
> >
> > On Tue, Aug 30, 2011 at 11:39 AM, Ben Ciceron <be...@triggit.com> wrote:
> >
> >> let me rephrase this:
> >>
> >> can any of the kafka process run outside the hadoop cluster as long as
> >> it can connect to the hadoop process from that host ?
> >> e.g :
> >>
> >> hostA (NOT in th hadoop cluster) : runs kafka hadoop consumer
> >> hostB (in th hadoop cluster) : runs jobtracker
> >>
> >>
> >> Cheers,
> >> Ben-
> >>
> >>
> >>
> >>
> >> On Mon, Aug 29, 2011 at 4:59 PM, Jun Rao <ju...@gmail.com> wrote:
> >> > My understanding is that it's not tied to localhost. You just need to
> >> change
> >> > the jobtracker setting in you Hadoop config.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Thu, Aug 25, 2011 at 4:31 PM, Ben Ciceron <be...@triggit.com> wrote:
> >> >
> >> >> Hello,
> >> >>
> >> >> does kafka hadoop consumer expect the jobtracker to run locally only
> ?
> >> >> it seems it expect it locally (localhost/127.0.0.1:9001) .
> >> >> Is it a requirement or there is a way to change it to a remote uri ?
> >> >>
> >> >> Cheers,
> >> >> Ben-
> >> >>
> >> >
> >>
> >
>
Re: jobtracker / hadoop comsumer
Posted by Ben Ciceron <be...@triggit.com>.
thx for confirming.
so when follow the instructions to run the hadoop consumer
(https://github.com/kafka-dev/kafka/tree/master/contrib/hadoop-consumer)
i see my mapred job being submitted properly on hostB (jobtracker) but
it always fails with :
console output on hostA
11/08/31 07:32:22 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
11/08/31 07:32:35 INFO mapred.FileInputFormat: Total input paths to process : 1
Hadoop job id=job_201108291829_0041
Exception in thread "main" java.lang.Exception: Hadoop ETL job failed!
Please check status on
http://localhost:9001/jobdetails.jsp?jobid=job_201108291829_0041
at kafka.etl.impl.SimpleKafkaETLJob.execute(SimpleKafkaETLJob.java:82)
at kafka.etl.impl.SimpleKafkaETLJob.main(SimpleKafkaETLJob.java:100)
in mapred log, for each tasktracker host i see :
Meta VERSION="1" .
Job JOBID="job_201108291829_0041" JOBNAME="SimpleKafakETL" USER="root"
SUBMIT_TIME="1314747158794"
JOBCONF="maprfs://10\.18\.125\.176:7222/var/mapr/cluster/mapred/jobTracker/staging/root/\.staging/job_201108291829_0041/job\.xml"
VIEW_JOB="*" MODIFY_JOB="*" JOB_QUEUE="default" .
Job JOBID="job_201108291829_0041" JOB_PRIORITY="NORMAL" .
Job JOBID="job_201108291829_0041" JOB_STATUS="RUNNING" .
Job JOBID="job_201108291829_0041" LAUNCH_TIME="1314747158885"
TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
Task TASKID="task_201108291829_0041_m_000000" TASK_TYPE="MAP"
START_TIME="1314747160010"
SPLITS="/default-rack/hadoop2,/default-rack/hadoop9,/default-rack/hadoop6"
.
MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
START_TIME="1314747160121"
TRACKER_NAME="tracker_hadoop9:localhost/127\.0\.0\.1:59411"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
TASK_STATUS="FAILED" FINISH_TIME="1314747164349" HOSTNAME="hadoop9"
ERROR="java\.io\.IOException: java\.net\.ConnectException: Connection
refused
at kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:155)
at kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:14)
at org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.moveToNext(MapTask\.java:210)
at org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.next(MapTask\.java:195)
at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:48)
at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:393)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:326)
at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:268)
at java\.security\.AccessController\.doPrivileged(Native Method)
at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
at org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1074)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:262)
Caused by: java\.net\.ConnectException: Connection refused
at sun\.nio\.ch\.Net\.connect(Native Method)
at sun\.nio\.ch\.SocketChannelImpl\.connect(SocketChannelImpl\.java:500)
at kafka\.consumer\.SimpleConsumer\.connect(SimpleConsumer\.scala:54)
at kafka\.consumer\.SimpleConsumer\.getOrMakeConnection(SimpleConsumer\.scala:193)
at kafka\.consumer\.SimpleConsumer\.getOffsetsBefore(SimpleConsumer\.scala:156)
at kafka\.javaapi\.consumer\.SimpleConsumer\.getOffsetsBefore(SimpleConsumer\.scala:65)
at kafka\.etl\.KafkaETLContext\.getOffsetRange(KafkaETLContext\.java:209)
at kafka\.etl\.KafkaETLContext\.<init>(KafkaETLContext\.java:97)
at kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:115)
\.\.\. 11 more
which other port do i need to open between hostA and the tasktrackers ?
please note i can send a simple non kafka job from the same hostA to
hadoop and it completes successfully.
Cheers,
Ben-
On Tue, Aug 30, 2011 at 12:04 PM, Richard Park <ri...@gmail.com> wrote:
> The answer should be yes, each process should be able to run on different
> hosts. We are currently doing this.
>
> Host A submits kafka hadoop job to the job tracker on Host B,
> Host B then then connects to Host C (or many host C's)
>
> I planned on having a look at the example again to see if there are steps
> there are missing, or if the examples need to be beefed up.
>
> Thanks,
> -Richard
>
>
> On Tue, Aug 30, 2011 at 11:39 AM, Ben Ciceron <be...@triggit.com> wrote:
>
>> let me rephrase this:
>>
>> can any of the kafka process run outside the hadoop cluster as long as
>> it can connect to the hadoop process from that host ?
>> e.g :
>>
>> hostA (NOT in th hadoop cluster) : runs kafka hadoop consumer
>> hostB (in th hadoop cluster) : runs jobtracker
>>
>>
>> Cheers,
>> Ben-
>>
>>
>>
>>
>> On Mon, Aug 29, 2011 at 4:59 PM, Jun Rao <ju...@gmail.com> wrote:
>> > My understanding is that it's not tied to localhost. You just need to
>> change
>> > the jobtracker setting in you Hadoop config.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Thu, Aug 25, 2011 at 4:31 PM, Ben Ciceron <be...@triggit.com> wrote:
>> >
>> >> Hello,
>> >>
>> >> does kafka hadoop consumer expect the jobtracker to run locally only ?
>> >> it seems it expect it locally (localhost/127.0.0.1:9001) .
>> >> Is it a requirement or there is a way to change it to a remote uri ?
>> >>
>> >> Cheers,
>> >> Ben-
>> >>
>> >
>>
>
Re: jobtracker / hadoop comsumer
Posted by Richard Park <ri...@gmail.com>.
The answer should be yes, each process should be able to run on different
hosts. We are currently doing this.
Host A submits kafka hadoop job to the job tracker on Host B,
Host B then then connects to Host C (or many host C's)
I planned on having a look at the example again to see if there are steps
there are missing, or if the examples need to be beefed up.
Thanks,
-Richard
On Tue, Aug 30, 2011 at 11:39 AM, Ben Ciceron <be...@triggit.com> wrote:
> let me rephrase this:
>
> can any of the kafka process run outside the hadoop cluster as long as
> it can connect to the hadoop process from that host ?
> e.g :
>
> hostA (NOT in th hadoop cluster) : runs kafka hadoop consumer
> hostB (in th hadoop cluster) : runs jobtracker
>
>
> Cheers,
> Ben-
>
>
>
>
> On Mon, Aug 29, 2011 at 4:59 PM, Jun Rao <ju...@gmail.com> wrote:
> > My understanding is that it's not tied to localhost. You just need to
> change
> > the jobtracker setting in you Hadoop config.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Aug 25, 2011 at 4:31 PM, Ben Ciceron <be...@triggit.com> wrote:
> >
> >> Hello,
> >>
> >> does kafka hadoop consumer expect the jobtracker to run locally only ?
> >> it seems it expect it locally (localhost/127.0.0.1:9001) .
> >> Is it a requirement or there is a way to change it to a remote uri ?
> >>
> >> Cheers,
> >> Ben-
> >>
> >
>
Re: jobtracker / hadoop comsumer
Posted by Ben Ciceron <be...@triggit.com>.
let me rephrase this:
can any of the kafka process run outside the hadoop cluster as long as
it can connect to the hadoop process from that host ?
e.g :
hostA (NOT in th hadoop cluster) : runs kafka hadoop consumer
hostB (in th hadoop cluster) : runs jobtracker
Cheers,
Ben-
On Mon, Aug 29, 2011 at 4:59 PM, Jun Rao <ju...@gmail.com> wrote:
> My understanding is that it's not tied to localhost. You just need to change
> the jobtracker setting in you Hadoop config.
>
> Thanks,
>
> Jun
>
> On Thu, Aug 25, 2011 at 4:31 PM, Ben Ciceron <be...@triggit.com> wrote:
>
>> Hello,
>>
>> does kafka hadoop consumer expect the jobtracker to run locally only ?
>> it seems it expect it locally (localhost/127.0.0.1:9001) .
>> Is it a requirement or there is a way to change it to a remote uri ?
>>
>> Cheers,
>> Ben-
>>
>
Re: jobtracker / hadoop comsumer
Posted by Jun Rao <ju...@gmail.com>.
My understanding is that it's not tied to localhost. You just need to change
the jobtracker setting in you Hadoop config.
Thanks,
Jun
On Thu, Aug 25, 2011 at 4:31 PM, Ben Ciceron <be...@triggit.com> wrote:
> Hello,
>
> does kafka hadoop consumer expect the jobtracker to run locally only ?
> it seems it expect it locally (localhost/127.0.0.1:9001) .
> Is it a requirement or there is a way to change it to a remote uri ?
>
> Cheers,
> Ben-
>