You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by srinath <co...@gmail.com> on 2007/01/04 17:13:52 UTC
Issues Starting Hadoop Process in Nutch0.9l.1
Hi,
While starting hadoop process we are getting the following error in logs
tasktracker in datanode is not able to connect back to jobtracker (but
jobtracker on the other machine started successfully and listening on port
9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on 50050:
starting
2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on 50050:
starting
2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
tracker_websearch3:50050
2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on 50050:
starting
2007-01-04 23:57:35,566 INFO ipc.Client -
org.apache.hadoop.io.ObjectWritableConnection culler maxidletime= 1000ms
2007-01-04 23:57:35,567 INFO ipc.Client -
org.apache.hadoop.io.ObjectWritable Connection Culler: starting
2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 1 time(s).
2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 2 time(s).
2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 3 time(s).
2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 4 time(s).
2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 5 time(s).
2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 6 time(s).
2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 7 time(s).
2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 8 time(s).
2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 9 time(s).
2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 10 time(s).
2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
websearch1/10.50.12.220:9001 not available yet, Zzzzz...
2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 1 time(s).
2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 2 time(s).
2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 3 time(s).
2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to server:
websearch1/10.50.12.220:9001. Already tried 4 time(s).
can any one help? regarding this????? does something to do with hadoop
configuration?
--
View this message in context: http://www.nabble.com/Issues-Starting-Hadoop-Process-in-Nutch0.9l.1-tf2920348.html#a8161938
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Issues Starting Hadoop Process in Nutch0.9l.1
Posted by srinath <co...@gmail.com>.
Hi,
Thx For Your Reply .. But namenode was started successfully on masternode
.... and datanode where we started on the machine where masternode is
running is able to connect but datanode on the other machine is not able to
connect back!!!!!!!!!! if u like to see i will post configuration params
what we set????
Dennis Kubes wrote:
>
> I would take a look at the processes on the namenode server and see if
> the namenode has started up. It doesn't look like it did. If this is a
> new install, did you format the namenode?
>
> Dennis
>
> srinath wrote:
>> Hi,
>> While starting hadoop process we are getting the following error in
>> logs
>> tasktracker in datanode is not able to connect back to jobtracker (but
>> jobtracker on the other machine started successfully and listening on
>> port
>> 9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
>>
>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on
>> 50050:
>> starting
>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on
>> 50050:
>> starting
>> 2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
>> tracker_websearch3:50050
>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on
>> 50050:
>> starting
>> 2007-01-04 23:57:35,566 INFO ipc.Client -
>> org.apache.hadoop.io.ObjectWritableConnection culler maxidletime= 1000ms
>> 2007-01-04 23:57:35,567 INFO ipc.Client -
>> org.apache.hadoop.io.ObjectWritable Connection Culler: starting
>> 2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>> 2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>> 2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>> 2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>> 2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 5 time(s).
>> 2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 6 time(s).
>> 2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 7 time(s).
>> 2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 8 time(s).
>> 2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 9 time(s).
>> 2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 10 time(s).
>> 2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
>> websearch1/10.50.12.220:9001 not available yet, Zzzzz...
>> 2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>> 2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>> 2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>> 2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>
>>
>> can any one help? regarding this????? does something to do with hadoop
>> configuration?
>
>
--
View this message in context: http://www.nabble.com/Issues-Starting-Hadoop-Process-in-Nutch0.9l.1-tf2920348.html#a8173263
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Issues Starting Hadoop Process in Nutch0.9l.1
Posted by srinath <co...@gmail.com>.
Hi dennis,
Ur's intitution was spot on!!!!!!!!!!!!!!!!!!!!!!!!!! After we
changed to fully qualified domain name it's working perfect........Thanks
alot for your help!!!!!!!!!!!!!! We require more help from u in coming
days!!!!!!!!!!!!!
THANK YOU!!!!!
Dennis Kubes wrote:
>
> I may be wrong but my intuition keeps telling me it is a dns issue.
> Below you have websearch1. Have you tried fully qualified domain names
> in both the hadoop-site.xml and slaves files. For example
> websearch1.internal.com:9000 where internal is your internal domain. I
> have seen some computers have a hard time connecting when not using
> fully qualified names. Try that and let me know the result.
>
> The other thing I see, although I don't think it would be causing any
> problems, is on some of your properties below there are spaces in the
> names. I would remove the spaces.
>
> Dennis
>
> srinath wrote:
>> Hi Dennis,
>>
>> Yes i can ping from slavenode (there is no problems with
>> network connections and i have successfully cofingured Nutch0.8.1 and
>> crawled some pages without any issues) ...and below i'm attaching the
>> configurations which i'm using in hadoop-site.xml file and name of
>> systems i
>> mentioned in slaves file
>>
>> The content of hadoop-site.xml is as follows :
>>
>> <configuration>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>websearch1:9000</value>
>> </property>
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>websearch1:9001</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktracker.tasks.maximum </name>
>> <value>20</value>
>> </property>
>>
>> <property>
>> <name>mapred.map.tasks</name>
>> <value>60</value>
>> </property>
>>
>> <property>
>> <name>mapred.reduce.tasks</name>
>> <value>6</value>
>> </property>
>>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/data/nutch/nutch0.9/filesystem0.9/name</value>
>> </property>
>>
>> <property>
>> <name>dfs.data.dir</name>
>> <value>/data/nutch/nutch0.9/filesystem0.9/data</value>
>> </property>
>>
>> <property>
>> <name> dfs.replication</name>
>> <value>2</value>
>> </property>
>> </configuration>
>>
>> and slaves.txt
>>
>> websearch1
>> websearch2
>>
>> on Websearch1 datanode is starting without any
>> problems..but
>> on websearch3 it says it's started task tracker and datanode but they
>> both
>> can't send heartbeat to masternode(i mean they can't able to register
>> themselves with masternode ) ..... i'm continuosuly getting the same
>> error
>> as metioned before...
>> One more thing i tried yesterday ...was to change the
>> hadoop
>> version... to 0.5 from 0.9.1 then namenode and datanode started
>> successfully
>> but ...jobtracker failed to start because nutch version(0.9.1) is using
>> some
>> classes which are not available on 0.5.....and i tried 0.7.1 and 0.8.2
>> ....same problems....and also i tried with 0.9.2 version i can't succeed
>> ..then i feel there is something to do with configurations?
>>
>>
>> Dennis Kubes wrote:
>>> Can you ping the master computer (name node) from the slave (data node)
>>> computers. Also is your namenode configuration fs.default.name variable
>>> pointing to 127.0.0.1 or is it pointing to the fully qualified domain
>>> name of the master computer?
>>>
>>> Dennis Kubes
>>>
>>> srinath wrote:
>>>> Hi,
>>>> Thx For Your Reply .. But namenode was started successfully on
>>>> masternode
>>>> .... and datanode where we started on the machine where masternode is
>>>> running is able to connect but datanode on the other machine is not
>>>> able
>>>> to
>>>> connect back!!!!!!!!!! if u like to see i will post configuration
>>>> params
>>>> what we set????
>>>>
>>>>
>>>> Dennis Kubes wrote:
>>>>> I would take a look at the processes on the namenode server and see if
>>>>> the namenode has started up. It doesn't look like it did. If this is
>>>>> a
>>>>> new install, did you format the namenode?
>>>>>
>>>>> Dennis
>>>>>
>>>>> srinath wrote:
>>>>>> Hi,
>>>>>> While starting hadoop process we are getting the following error
>>>>>> in
>>>>>> logs
>>>>>> tasktracker in datanode is not able to connect back to jobtracker
>>>>>> (but
>>>>>> jobtracker on the other machine started successfully and listening on
>>>>>> port
>>>>>> 9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
>>>>>>
>>>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on
>>>>>> 50050:
>>>>>> starting
>>>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on
>>>>>> 50050:
>>>>>> starting
>>>>>> 2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
>>>>>> tracker_websearch3:50050
>>>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on
>>>>>> 50050:
>>>>>> starting
>>>>>> 2007-01-04 23:57:35,566 INFO ipc.Client -
>>>>>> org.apache.hadoop.io.ObjectWritableConnection culler maxidletime=
>>>>>> 1000ms
>>>>>> 2007-01-04 23:57:35,567 INFO ipc.Client -
>>>>>> org.apache.hadoop.io.ObjectWritable Connection Culler: starting
>>>>>> 2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>>>>> 2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>>>>> 2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>>>>> 2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>>>>> 2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 5 time(s).
>>>>>> 2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 6 time(s).
>>>>>> 2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 7 time(s).
>>>>>> 2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 8 time(s).
>>>>>> 2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 9 time(s).
>>>>>> 2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 10 time(s).
>>>>>> 2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
>>>>>> websearch1/10.50.12.220:9001 not available yet, Zzzzz...
>>>>>> 2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>>>>> 2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>>>>> 2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>>>>> 2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to
>>>>>> server:
>>>>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>>>>>
>>>>>>
>>>>>> can any one help? regarding this????? does something to do with
>>>>>> hadoop
>>>>>> configuration?
>>>
>>
>
>
--
View this message in context: http://www.nabble.com/Issues-Starting-Hadoop-Process-in-Nutch0.9l.1-tf2920348.html#a8213165
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Issues Starting Hadoop Process in Nutch0.9l.1
Posted by Dennis Kubes <nu...@dragonflymc.com>.
I may be wrong but my intuition keeps telling me it is a dns issue.
Below you have websearch1. Have you tried fully qualified domain names
in both the hadoop-site.xml and slaves files. For example
websearch1.internal.com:9000 where internal is your internal domain. I
have seen some computers have a hard time connecting when not using
fully qualified names. Try that and let me know the result.
The other thing I see, although I don't think it would be causing any
problems, is on some of your properties below there are spaces in the
names. I would remove the spaces.
Dennis
srinath wrote:
> Hi Dennis,
>
> Yes i can ping from slavenode (there is no problems with
> network connections and i have successfully cofingured Nutch0.8.1 and
> crawled some pages without any issues) ...and below i'm attaching the
> configurations which i'm using in hadoop-site.xml file and name of systems i
> mentioned in slaves file
>
> The content of hadoop-site.xml is as follows :
>
> <configuration>
>
> <property>
> <name>fs.default.name</name>
> <value>websearch1:9000</value>
> </property>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>websearch1:9001</value>
> </property>
>
> <property>
> <name>mapred.tasktracker.tasks.maximum </name>
> <value>20</value>
> </property>
>
> <property>
> <name>mapred.map.tasks</name>
> <value>60</value>
> </property>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>6</value>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>/data/nutch/nutch0.9/filesystem0.9/name</value>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/data/nutch/nutch0.9/filesystem0.9/data</value>
> </property>
>
> <property>
> <name> dfs.replication</name>
> <value>2</value>
> </property>
> </configuration>
>
> and slaves.txt
>
> websearch1
> websearch2
>
> on Websearch1 datanode is starting without any problems..but
> on websearch3 it says it's started task tracker and datanode but they both
> can't send heartbeat to masternode(i mean they can't able to register
> themselves with masternode ) ..... i'm continuosuly getting the same error
> as metioned before...
> One more thing i tried yesterday ...was to change the hadoop
> version... to 0.5 from 0.9.1 then namenode and datanode started successfully
> but ...jobtracker failed to start because nutch version(0.9.1) is using some
> classes which are not available on 0.5.....and i tried 0.7.1 and 0.8.2
> ....same problems....and also i tried with 0.9.2 version i can't succeed
> ..then i feel there is something to do with configurations?
>
>
> Dennis Kubes wrote:
>> Can you ping the master computer (name node) from the slave (data node)
>> computers. Also is your namenode configuration fs.default.name variable
>> pointing to 127.0.0.1 or is it pointing to the fully qualified domain
>> name of the master computer?
>>
>> Dennis Kubes
>>
>> srinath wrote:
>>> Hi,
>>> Thx For Your Reply .. But namenode was started successfully on
>>> masternode
>>> .... and datanode where we started on the machine where masternode is
>>> running is able to connect but datanode on the other machine is not able
>>> to
>>> connect back!!!!!!!!!! if u like to see i will post configuration params
>>> what we set????
>>>
>>>
>>> Dennis Kubes wrote:
>>>> I would take a look at the processes on the namenode server and see if
>>>> the namenode has started up. It doesn't look like it did. If this is a
>>>> new install, did you format the namenode?
>>>>
>>>> Dennis
>>>>
>>>> srinath wrote:
>>>>> Hi,
>>>>> While starting hadoop process we are getting the following error in
>>>>> logs
>>>>> tasktracker in datanode is not able to connect back to jobtracker (but
>>>>> jobtracker on the other machine started successfully and listening on
>>>>> port
>>>>> 9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
>>>>>
>>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on
>>>>> 50050:
>>>>> starting
>>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on
>>>>> 50050:
>>>>> starting
>>>>> 2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
>>>>> tracker_websearch3:50050
>>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on
>>>>> 50050:
>>>>> starting
>>>>> 2007-01-04 23:57:35,566 INFO ipc.Client -
>>>>> org.apache.hadoop.io.ObjectWritableConnection culler maxidletime=
>>>>> 1000ms
>>>>> 2007-01-04 23:57:35,567 INFO ipc.Client -
>>>>> org.apache.hadoop.io.ObjectWritable Connection Culler: starting
>>>>> 2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>>>> 2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>>>> 2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>>>> 2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>>>> 2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 5 time(s).
>>>>> 2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 6 time(s).
>>>>> 2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 7 time(s).
>>>>> 2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 8 time(s).
>>>>> 2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 9 time(s).
>>>>> 2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 10 time(s).
>>>>> 2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
>>>>> websearch1/10.50.12.220:9001 not available yet, Zzzzz...
>>>>> 2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>>>> 2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>>>> 2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>>>> 2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to server:
>>>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>>>>
>>>>>
>>>>> can any one help? regarding this????? does something to do with hadoop
>>>>> configuration?
>>
>
Re: Issues Starting Hadoop Process in Nutch0.9l.1
Posted by srinath <co...@gmail.com>.
Hi Dennis,
Yes i can ping from slavenode (there is no problems with
network connections and i have successfully cofingured Nutch0.8.1 and
crawled some pages without any issues) ...and below i'm attaching the
configurations which i'm using in hadoop-site.xml file and name of systems i
mentioned in slaves file
The content of hadoop-site.xml is as follows :
<configuration>
<property>
<name>fs.default.name</name>
<value>websearch1:9000</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>websearch1:9001</value>
</property>
<property>
<name>mapred.tasktracker.tasks.maximum </name>
<value>20</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>60</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>6</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/data/nutch/nutch0.9/filesystem0.9/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/data/nutch/nutch0.9/filesystem0.9/data</value>
</property>
<property>
<name> dfs.replication</name>
<value>2</value>
</property>
</configuration>
and slaves.txt
websearch1
websearch2
on Websearch1 datanode is starting without any problems..but
on websearch3 it says it's started task tracker and datanode but they both
can't send heartbeat to masternode(i mean they can't able to register
themselves with masternode ) ..... i'm continuosuly getting the same error
as metioned before...
One more thing i tried yesterday ...was to change the hadoop
version... to 0.5 from 0.9.1 then namenode and datanode started successfully
but ...jobtracker failed to start because nutch version(0.9.1) is using some
classes which are not available on 0.5.....and i tried 0.7.1 and 0.8.2
....same problems....and also i tried with 0.9.2 version i can't succeed
..then i feel there is something to do with configurations?
Dennis Kubes wrote:
>
> Can you ping the master computer (name node) from the slave (data node)
> computers. Also is your namenode configuration fs.default.name variable
> pointing to 127.0.0.1 or is it pointing to the fully qualified domain
> name of the master computer?
>
> Dennis Kubes
>
> srinath wrote:
>> Hi,
>> Thx For Your Reply .. But namenode was started successfully on
>> masternode
>> .... and datanode where we started on the machine where masternode is
>> running is able to connect but datanode on the other machine is not able
>> to
>> connect back!!!!!!!!!! if u like to see i will post configuration params
>> what we set????
>>
>>
>> Dennis Kubes wrote:
>>> I would take a look at the processes on the namenode server and see if
>>> the namenode has started up. It doesn't look like it did. If this is a
>>> new install, did you format the namenode?
>>>
>>> Dennis
>>>
>>> srinath wrote:
>>>> Hi,
>>>> While starting hadoop process we are getting the following error in
>>>> logs
>>>> tasktracker in datanode is not able to connect back to jobtracker (but
>>>> jobtracker on the other machine started successfully and listening on
>>>> port
>>>> 9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
>>>>
>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on
>>>> 50050:
>>>> starting
>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on
>>>> 50050:
>>>> starting
>>>> 2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
>>>> tracker_websearch3:50050
>>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on
>>>> 50050:
>>>> starting
>>>> 2007-01-04 23:57:35,566 INFO ipc.Client -
>>>> org.apache.hadoop.io.ObjectWritableConnection culler maxidletime=
>>>> 1000ms
>>>> 2007-01-04 23:57:35,567 INFO ipc.Client -
>>>> org.apache.hadoop.io.ObjectWritable Connection Culler: starting
>>>> 2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>>> 2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>>> 2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>>> 2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>>> 2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 5 time(s).
>>>> 2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 6 time(s).
>>>> 2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 7 time(s).
>>>> 2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 8 time(s).
>>>> 2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 9 time(s).
>>>> 2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 10 time(s).
>>>> 2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
>>>> websearch1/10.50.12.220:9001 not available yet, Zzzzz...
>>>> 2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>>> 2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>>> 2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>>> 2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to server:
>>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>>>
>>>>
>>>> can any one help? regarding this????? does something to do with hadoop
>>>> configuration?
>>>
>>
>
>
--
View this message in context: http://www.nabble.com/Issues-Starting-Hadoop-Process-in-Nutch0.9l.1-tf2920348.html#a8202161
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Issues Starting Hadoop Process in Nutch0.9l.1
Posted by Dennis Kubes <nu...@dragonflymc.com>.
Can you ping the master computer (name node) from the slave (data node)
computers. Also is your namenode configuration fs.default.name variable
pointing to 127.0.0.1 or is it pointing to the fully qualified domain
name of the master computer?
Dennis Kubes
srinath wrote:
> Hi,
> Thx For Your Reply .. But namenode was started successfully on masternode
> .... and datanode where we started on the machine where masternode is
> running is able to connect but datanode on the other machine is not able to
> connect back!!!!!!!!!! if u like to see i will post configuration params
> what we set????
>
>
> Dennis Kubes wrote:
>> I would take a look at the processes on the namenode server and see if
>> the namenode has started up. It doesn't look like it did. If this is a
>> new install, did you format the namenode?
>>
>> Dennis
>>
>> srinath wrote:
>>> Hi,
>>> While starting hadoop process we are getting the following error in
>>> logs
>>> tasktracker in datanode is not able to connect back to jobtracker (but
>>> jobtracker on the other machine started successfully and listening on
>>> port
>>> 9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
>>>
>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on
>>> 50050:
>>> starting
>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on
>>> 50050:
>>> starting
>>> 2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
>>> tracker_websearch3:50050
>>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on
>>> 50050:
>>> starting
>>> 2007-01-04 23:57:35,566 INFO ipc.Client -
>>> org.apache.hadoop.io.ObjectWritableConnection culler maxidletime= 1000ms
>>> 2007-01-04 23:57:35,567 INFO ipc.Client -
>>> org.apache.hadoop.io.ObjectWritable Connection Culler: starting
>>> 2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>> 2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>> 2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>> 2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>> 2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 5 time(s).
>>> 2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 6 time(s).
>>> 2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 7 time(s).
>>> 2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 8 time(s).
>>> 2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 9 time(s).
>>> 2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 10 time(s).
>>> 2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
>>> websearch1/10.50.12.220:9001 not available yet, Zzzzz...
>>> 2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>>> 2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>>> 2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>>> 2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to server:
>>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>>
>>>
>>> can any one help? regarding this????? does something to do with hadoop
>>> configuration?
>>
>
Re: Issues Starting Hadoop Process in Nutch0.9l.1
Posted by srinath <co...@gmail.com>.
Hi,
Thx For Your Reply .. But namenode was started successfully on masternode
.... and datanode where we started on the machine where masternode is
running is able to connect but datanode on the other machine is not able to
connect back!!!!!!!!!! if u like to see i will post configuration params
what we set????
Dennis Kubes wrote:
>
> I would take a look at the processes on the namenode server and see if
> the namenode has started up. It doesn't look like it did. If this is a
> new install, did you format the namenode?
>
> Dennis
>
> srinath wrote:
>> Hi,
>> While starting hadoop process we are getting the following error in
>> logs
>> tasktracker in datanode is not able to connect back to jobtracker (but
>> jobtracker on the other machine started successfully and listening on
>> port
>> 9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
>>
>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on
>> 50050:
>> starting
>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on
>> 50050:
>> starting
>> 2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
>> tracker_websearch3:50050
>> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on
>> 50050:
>> starting
>> 2007-01-04 23:57:35,566 INFO ipc.Client -
>> org.apache.hadoop.io.ObjectWritableConnection culler maxidletime= 1000ms
>> 2007-01-04 23:57:35,567 INFO ipc.Client -
>> org.apache.hadoop.io.ObjectWritable Connection Culler: starting
>> 2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>> 2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>> 2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>> 2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>> 2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 5 time(s).
>> 2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 6 time(s).
>> 2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 7 time(s).
>> 2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 8 time(s).
>> 2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 9 time(s).
>> 2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 10 time(s).
>> 2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
>> websearch1/10.50.12.220:9001 not available yet, Zzzzz...
>> 2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 1 time(s).
>> 2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 2 time(s).
>> 2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 3 time(s).
>> 2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to server:
>> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>>
>>
>> can any one help? regarding this????? does something to do with hadoop
>> configuration?
>
>
--
View this message in context: http://www.nabble.com/Issues-Starting-Hadoop-Process-in-Nutch0.9l.1-tf2920348.html#a8173265
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Issues Starting Hadoop Process in Nutch0.9l.1
Posted by Dennis Kubes <nu...@dragonflymc.com>.
I would take a look at the processes on the namenode server and see if
the namenode has started up. It doesn't look like it did. If this is a
new install, did you format the namenode?
Dennis
srinath wrote:
> Hi,
> While starting hadoop process we are getting the following error in logs
> tasktracker in datanode is not able to connect back to jobtracker (but
> jobtracker on the other machine started successfully and listening on port
> 9001) ... i'm using Nutch0.9.1 version and Hadoop0.9.1 ..
>
> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 17 on 50050:
> starting
> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 18 on 50050:
> starting
> 2007-01-04 23:57:35,559 INFO mapred.TaskTracker - Starting tracker
> tracker_websearch3:50050
> 2007-01-04 23:57:35,559 INFO ipc.Server - IPC Server handler 19 on 50050:
> starting
> 2007-01-04 23:57:35,566 INFO ipc.Client -
> org.apache.hadoop.io.ObjectWritableConnection culler maxidletime= 1000ms
> 2007-01-04 23:57:35,567 INFO ipc.Client -
> org.apache.hadoop.io.ObjectWritable Connection Culler: starting
> 2007-01-04 23:57:35,589 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 1 time(s).
> 2007-01-04 23:57:36,590 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 2 time(s).
> 2007-01-04 23:57:37,600 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 3 time(s).
> 2007-01-04 23:57:38,610 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 4 time(s).
> 2007-01-04 23:57:39,620 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 5 time(s).
> 2007-01-04 23:57:40,630 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 6 time(s).
> 2007-01-04 23:57:41,640 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 7 time(s).
> 2007-01-04 23:57:42,650 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 8 time(s).
> 2007-01-04 23:57:43,660 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 9 time(s).
> 2007-01-04 23:57:44,670 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 10 time(s).
> 2007-01-04 23:57:45,680 INFO ipc.RPC - Server at
> websearch1/10.50.12.220:9001 not available yet, Zzzzz...
> 2007-01-04 23:57:46,690 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 1 time(s).
> 2007-01-04 23:57:47,700 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 2 time(s).
> 2007-01-04 23:57:48,710 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 3 time(s).
> 2007-01-04 23:57:49,720 INFO ipc.Client - Retrying connect to server:
> websearch1/10.50.12.220:9001. Already tried 4 time(s).
>
>
> can any one help? regarding this????? does something to do with hadoop
> configuration?