You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by kaveh minooie <ka...@plutoz.com> on 2012/02/05 00:09:16 UTC
nutch logs when run over hadoop
Hi everyone
Anybody knows how I can see the nutch logs when it is run on top of
hadoop in a multi nodes environment? in hadoop log directories, I only
have datanode and tasktracker log files but they don't have any nutch
entries in them.
Thanks,
--
Kaveh Minooie
www.plutoz.com
Thread spinWaiting, utilizing bandwidth and connection time out error
Posted by kaveh minooie <ka...@plutoz.com>.
so I found the exception:
2012-02-06 12:20:36,614 ERROR org.apache.nutch.protocol.httpclient.Http:
org.apache.commons.httpclient.ConnectTimeoutException: The host did not
accept the connection within timeout of 50000 ms
which would have been understandable except for the fact that I am
barley using 10% of my bandwidth ( i understand that it could be busy on
the other end but I don't think this is the case here). I don't seem to
be able to utilize my bandwidth correctly or completely. I am defining 4
available slot per datanode for map (and reduce for that matter) and
during tasks like updatedb or generate I can see that my entire cpu
power available on all the nodes are being used (cpu idle time goes to
0.0) so I don't think increasing the available slots would give me
anything more. now the only thing that I can think of to increase the
fetcher performance is to increase the number of threads. when i use
anything more than 16 (I tried 32 and I have quad core cpus) for almost
95% of pages I get the above mentioned exception. when I run it with 16
treads I am able to fetch most of the pages (more than 90%) but when I
look at the log file most of the time 15 out of 16 threads are spinwaiting:
2012-02-06 12:20:24,686 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=13, fetchQueues.totalSize=800
2012-02-06 12:20:50,555 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=16, fetchQueues.totalSize=800
2012-02-06 12:20:52,612 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=15, fetchQueues.totalSize=799
2012-02-06 12:21:07,040 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=14, fetchQueues.totalSize=799
2012-02-06 12:21:08,156 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=14, fetchQueues.totalSize=800
2012-02-06 12:21:09,220 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=16, fetchQueues.totalSize=800
2012-02-06 12:21:10,248 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=16, fetchQueues.totalSize=800
2012-02-06 12:21:11,289 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=16, spinWaiting=16, fetchQueues.totalSize=800
these are excerpts from the log files. I am trying to crawl something
between 3000 to 4000 sites in each run so I don't think they are waiting
due to politeness factor. and when I check the firewall I see that I am
only using a fraction of the available bandwidth and even that is only
true in the beginning of the fetching process and it sharply goes down
to something between 1/3 to 1/4 of the initial value. I want to be able
to use my entire bandwidth. what am I doing wrong? what should I do?
Thanks,
On 02/06/2012 12:36 PM, Markus Jelsma wrote:
>
>> yes I think i did get the right mapper since this is what i could see in
>> the 50030/taskdetails.jsp:
>>
>> attempt_201202041349_0021_m_000004_0
>> and i looked at the same folder (same name) in that datanode machine.
>
> Odd. But without the stack trace we cannot figure out what's going on. But
> usually when throughput drops and errors rise then you crossed the limits or
> your hardware, network and/or settings. Always check syslog as well.
>
> anyway, did you view the log file entirely? via the gui or shell? it must be
> there or some magic is happening.
>
>>
>> and the number of exceptions do not increase linearly but they do
>> exponentially so for example if I run the same thing with 32 threads the
>> number of exception would be 69 and the number of success would be only
>> 1 or 2. it seems like a race scenario to me that causes them all fail,
>> but I don't know...
>>
>> On 02/06/2012 12:08 PM, Markus Jelsma wrote:
>>>> Thanks Markus.
>>>>
>>>> when I am running the fetcher, some times in the job detail I see
>>>>
>>>> something like this: (this is on 50030:jobtasks.jsp btw)
>>>>
>>>> 16 threads, 1 queues, 15 URLs queued, 55 pages, 1 errors, 0.0 (1)
>>>>
>>>> pages/s, 19.0 (11815) kbits/s,
>>>>
>>>> when i click on the counters I see this:
>>>>
>>>> FetcherStatus
>>>>
>>>> exception 2
>>>> success 69
>>>>
>>>> and that was why I was looking for the nutch log files which Thanks to
>>>> Markus I found, but when I looked at the syslog file corresponding to
>>>> the job Id and attempt that generate above message there was no
>>>> exception in the log files.
>>>
>>> Did you get the right mapper? Each mapper has its own log file.
>>>
>>>> so does any one know what this means? btw,
>>>> the number of "exception" increases exponentially when I increase the
>>>> number of threads for fetcher (like using 32 instead of 16).
>>>
>>> Well, that makes sense. If for example one out of 10 fetches fail per
>>> thread then you get twice as many failures for doubling the number of
>>> threads. There are always exceptions unless you are in an ideal world.
>>>
>>>> If some one
>>>> could explain to me what is the mechanism here that causes this I would
>>>> very much appreciate it.
>>>>
>>>> Thanks,
>>>>
>>>> On 02/06/2012 07:46 AM, Markus Jelsma wrote:
>>>>> there's a userlog directory on your datanodes. You can also view them
>>>>> through your web gui.
>>>>>
>>>>> On Sunday 05 February 2012 00:09:16 kaveh minooie wrote:
>>>>>> Hi everyone
>>>>>>
>>>>>> Anybody knows how I can see the nutch logs when it is run on top
>>>>>> of
>>>>>>
>>>>>> hadoop in a multi nodes environment? in hadoop log directories, I
>>>>>> only have datanode and tasktracker log files but they don't have any
>>>>>> nutch entries in them.
>>>>>>
>>>>>> Thanks,
--
Kaveh Minooie
www.plutoz.com
Re: nutch logs when run over hadoop
Posted by Markus Jelsma <ma...@openindex.io>.
> yes I think i did get the right mapper since this is what i could see in
> the 50030/taskdetails.jsp:
>
> attempt_201202041349_0021_m_000004_0
> and i looked at the same folder (same name) in that datanode machine.
Odd. But without the stack trace we cannot figure out what's going on. But
usually when throughput drops and errors rise then you crossed the limits or
your hardware, network and/or settings. Always check syslog as well.
anyway, did you view the log file entirely? via the gui or shell? it must be
there or some magic is happening.
>
> and the number of exceptions do not increase linearly but they do
> exponentially so for example if I run the same thing with 32 threads the
> number of exception would be 69 and the number of success would be only
> 1 or 2. it seems like a race scenario to me that causes them all fail,
> but I don't know...
>
> On 02/06/2012 12:08 PM, Markus Jelsma wrote:
> >> Thanks Markus.
> >>
> >> when I am running the fetcher, some times in the job detail I see
> >>
> >> something like this: (this is on 50030:jobtasks.jsp btw)
> >>
> >> 16 threads, 1 queues, 15 URLs queued, 55 pages, 1 errors, 0.0 (1)
> >>
> >> pages/s, 19.0 (11815) kbits/s,
> >>
> >> when i click on the counters I see this:
> >>
> >> FetcherStatus
> >>
> >> exception 2
> >> success 69
> >>
> >> and that was why I was looking for the nutch log files which Thanks to
> >> Markus I found, but when I looked at the syslog file corresponding to
> >> the job Id and attempt that generate above message there was no
> >> exception in the log files.
> >
> > Did you get the right mapper? Each mapper has its own log file.
> >
> >> so does any one know what this means? btw,
> >> the number of "exception" increases exponentially when I increase the
> >> number of threads for fetcher (like using 32 instead of 16).
> >
> > Well, that makes sense. If for example one out of 10 fetches fail per
> > thread then you get twice as many failures for doubling the number of
> > threads. There are always exceptions unless you are in an ideal world.
> >
> >> If some one
> >> could explain to me what is the mechanism here that causes this I would
> >> very much appreciate it.
> >>
> >> Thanks,
> >>
> >> On 02/06/2012 07:46 AM, Markus Jelsma wrote:
> >>> there's a userlog directory on your datanodes. You can also view them
> >>> through your web gui.
> >>>
> >>> On Sunday 05 February 2012 00:09:16 kaveh minooie wrote:
> >>>> Hi everyone
> >>>>
> >>>> Anybody knows how I can see the nutch logs when it is run on top
> >>>> of
> >>>>
> >>>> hadoop in a multi nodes environment? in hadoop log directories, I
> >>>> only have datanode and tasktracker log files but they don't have any
> >>>> nutch entries in them.
> >>>>
> >>>> Thanks,
Re: nutch logs when run over hadoop
Posted by kaveh minooie <ka...@plutoz.com>.
yes I think i did get the right mapper since this is what i could see in
the 50030/taskdetails.jsp:
attempt_201202041349_0021_m_000004_0
and i looked at the same folder (same name) in that datanode machine.
and the number of exceptions do not increase linearly but they do
exponentially so for example if I run the same thing with 32 threads the
number of exception would be 69 and the number of success would be only
1 or 2. it seems like a race scenario to me that causes them all fail,
but I don't know...
On 02/06/2012 12:08 PM, Markus Jelsma wrote:
>
>> Thanks Markus.
>> when I am running the fetcher, some times in the job detail I see
>> something like this: (this is on 50030:jobtasks.jsp btw)
>>
>> 16 threads, 1 queues, 15 URLs queued, 55 pages, 1 errors, 0.0 (1)
>> pages/s, 19.0 (11815) kbits/s,
>>
>> when i click on the counters I see this:
>>
>> FetcherStatus
>> exception 2
>> success 69
>>
>> and that was why I was looking for the nutch log files which Thanks to
>> Markus I found, but when I looked at the syslog file corresponding to
>> the job Id and attempt that generate above message there was no
>> exception in the log files.
>
> Did you get the right mapper? Each mapper has its own log file.
>
>> so does any one know what this means? btw,
>> the number of "exception" increases exponentially when I increase the
>> number of threads for fetcher (like using 32 instead of 16).
>
> Well, that makes sense. If for example one out of 10 fetches fail per thread
> then you get twice as many failures for doubling the number of threads. There
> are always exceptions unless you are in an ideal world.
>
>> If some one
>> could explain to me what is the mechanism here that causes this I would
>> very much appreciate it.
>>
>> Thanks,
>>
>> On 02/06/2012 07:46 AM, Markus Jelsma wrote:
>>> there's a userlog directory on your datanodes. You can also view them
>>> through your web gui.
>>>
>>> On Sunday 05 February 2012 00:09:16 kaveh minooie wrote:
>>>> Hi everyone
>>>>
>>>> Anybody knows how I can see the nutch logs when it is run on top of
>>>>
>>>> hadoop in a multi nodes environment? in hadoop log directories, I only
>>>> have datanode and tasktracker log files but they don't have any nutch
>>>> entries in them.
>>>>
>>>> Thanks,
--
Kaveh Minooie
www.plutoz.com
Re: nutch logs when run over hadoop
Posted by Markus Jelsma <ma...@openindex.io>.
> Thanks Markus.
> when I am running the fetcher, some times in the job detail I see
> something like this: (this is on 50030:jobtasks.jsp btw)
>
> 16 threads, 1 queues, 15 URLs queued, 55 pages, 1 errors, 0.0 (1)
> pages/s, 19.0 (11815) kbits/s,
>
> when i click on the counters I see this:
>
> FetcherStatus
> exception 2
> success 69
>
> and that was why I was looking for the nutch log files which Thanks to
> Markus I found, but when I looked at the syslog file corresponding to
> the job Id and attempt that generate above message there was no
> exception in the log files.
Did you get the right mapper? Each mapper has its own log file.
> so does any one know what this means? btw,
> the number of "exception" increases exponentially when I increase the
> number of threads for fetcher (like using 32 instead of 16).
Well, that makes sense. If for example one out of 10 fetches fail per thread
then you get twice as many failures for doubling the number of threads. There
are always exceptions unless you are in an ideal world.
> If some one
> could explain to me what is the mechanism here that causes this I would
> very much appreciate it.
>
> Thanks,
>
> On 02/06/2012 07:46 AM, Markus Jelsma wrote:
> > there's a userlog directory on your datanodes. You can also view them
> > through your web gui.
> >
> > On Sunday 05 February 2012 00:09:16 kaveh minooie wrote:
> >> Hi everyone
> >>
> >> Anybody knows how I can see the nutch logs when it is run on top of
> >>
> >> hadoop in a multi nodes environment? in hadoop log directories, I only
> >> have datanode and tasktracker log files but they don't have any nutch
> >> entries in them.
> >>
> >> Thanks,
Re: nutch logs when run over hadoop
Posted by kaveh minooie <ka...@plutoz.com>.
Thanks Markus.
when I am running the fetcher, some times in the job detail I see
something like this: (this is on 50030:jobtasks.jsp btw)
16 threads, 1 queues, 15 URLs queued, 55 pages, 1 errors, 0.0 (1)
pages/s, 19.0 (11815) kbits/s,
when i click on the counters I see this:
FetcherStatus
exception 2
success 69
and that was why I was looking for the nutch log files which Thanks to
Markus I found, but when I looked at the syslog file corresponding to
the job Id and attempt that generate above message there was no
exception in the log files. so does any one know what this means? btw,
the number of "exception" increases exponentially when I increase the
number of threads for fetcher (like using 32 instead of 16). If some one
could explain to me what is the mechanism here that causes this I would
very much appreciate it.
Thanks,
On 02/06/2012 07:46 AM, Markus Jelsma wrote:
> there's a userlog directory on your datanodes. You can also view them through
> your web gui.
>
> On Sunday 05 February 2012 00:09:16 kaveh minooie wrote:
>> Hi everyone
>> Anybody knows how I can see the nutch logs when it is run on top of
>> hadoop in a multi nodes environment? in hadoop log directories, I only
>> have datanode and tasktracker log files but they don't have any nutch
>> entries in them.
>>
>> Thanks,
>
--
Kaveh Minooie
www.plutoz.com
Re: nutch logs when run over hadoop
Posted by Markus Jelsma <ma...@openindex.io>.
there's a userlog directory on your datanodes. You can also view them through
your web gui.
On Sunday 05 February 2012 00:09:16 kaveh minooie wrote:
> Hi everyone
> Anybody knows how I can see the nutch logs when it is run on top of
> hadoop in a multi nodes environment? in hadoop log directories, I only
> have datanode and tasktracker log files but they don't have any nutch
> entries in them.
>
> Thanks,
--
Markus Jelsma - CTO - Openindex