You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Amit Sela <am...@infolinks.com> on 2013/03/05 12:16:55 UTC
Understanding fetch MapReduce job counters and logs
Hi all,
I am trying to better understand the counters and logging of the fetch
MapReduce executed when crawling.
When looking at the job counters in the MapReduce web UI, I note the
following counters and values:
*Map input records 162,080*
moved 345
robots_denied 4,441
robots_denied_maxcrawldelay 259
*hitByTimeLimit 7,493*
exception 3,801
notmodified 2
gone 48
access_denied 1
*success 93,583*
temp_moved 3,068
notfound 1,490
And summing all counters does not equal the total map input...
But, when I go to the map task logs, at the end of each log there is a line
stating:
QueueFeeder finished: total *36651* records + hit by time limit :*20975*
QueueFeeder finished: total *30248* records + hit by time limit :*25492*
QueueFeeder finished: total *44257* records + hit by time limit :*4460*
*
*
Summing all of theses numbers does equal the total map input. I also note
that the total hit by time limit here is 50927 but the job counters show
7493.
Anyone can elaborate ?
Thanks,
Amit.
Re: Understanding fetch MapReduce job counters and logs
Posted by Amit Sela <am...@infolinks.com>.
I am using bin/crawl - I'll change the timeLimitFetch to something a bit
higher.
Thanks!
On Sun, Mar 17, 2013 at 5:07 PM, feng lu <am...@gmail.com> wrote:
> yes, the property is fetcher.timelimit.mins. if you not set this property,
> the QueueFeeder will not filter the url and log output may like this
>
> QueueFeeder finished: total 36651 records + hit by time limit :0
>
> Do you use bin/crawl command script. it will set the time limit for
> fetching to 180. the code like this
>
> # time limit for fetching
> timeLimitFetch=180
>
> # fetching the segment
> echo "Fetching : $SEGMENT"
> $bin/nutch fetch $commonOptions -D fetcher.timelimit.mins=$timeLimitFetch
> $CRAWL_PATH/segments/$SEGMENT -noParsing -threads $numThreads
>
>
>
>
>
> On Sun, Mar 17, 2013 at 7:47 PM, Amit Sela <am...@infolinks.com> wrote:
>
> > By fetcher.limit property,do you mean fetcher.timelimit.mins ? because I
> > have it set on default (-1) - no time limit.
> >
> >
> > On Sat, Mar 16, 2013 at 5:12 PM, feng lu <am...@gmail.com> wrote:
> >
> > > Hi Amit
> > >
> > > <<
> > > I also note
> > > that the total hit by time limit here is 50927 but the job counters
> show
> > > 7493.
> > > >>
> > >
> > > This two time limits are all set bye fetcher.limit property. One is
> used
> > in
> > > QueueFeeder class, indicate that the QueueFeeder should finish load
> data
> > if
> > > current time is larger than time limit. So the total hit by time limit
> is
> > > 50927. Another is used in FetchItemQueues class, indicate that check
> the
> > > time if current time is larger than time limit and feeder has stopped ,
> > > emptying the queues, So here job counters of time limit is 7493. There
> > are
> > > not equal.
> > >
> > >
> > > <<
> > > Summing all of theses numbers does equal the total map input.
> > > >>
> > >
> > > do you set the property of "fetcher.follow.outlinks.depth", when
> > > fetcher.parse is true and this value is greater than 0 the fetcher will
> > > extract outlinks
> > > and follow until the desired depth is reached.
> > >
> > > Another reason is that when this page is redirect to another page ,
> fetch
> > > will add new redirect page to fetch queues, so you can see that map
> input
> > > is not equal to numbers of all sum.
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Mar 16, 2013 at 8:03 AM, Lewis John Mcgibbney <
> > > lewis.mcgibbney@gmail.com> wrote:
> > >
> > > > Hi Amit,
> > > >
> > > > I know this thread is a bit old now, however it is also something
> which
> > > > bugged me when I was looking into something else (InjectorJob
> > counters).
> > > >
> > > > On Tue, Mar 5, 2013 at 3:16 AM, Amit Sela <am...@infolinks.com>
> wrote:
> > > >
> > > > >
> > > > > And summing all counters does not equal the total map input...
> > > > >
> > > > > Summing all of theses numbers does equal the total map input. I
> also
> > > note
> > > > > that the total hit by time limit here is 50927 but the job counters
> > > show
> > > > > 7493.
> > > > >
> > > > >
> > > > Basically, the easiest way to see and generally understand counters
> is
> > to
> > > > run the Nutch application within your Hadoop cluster (if no cluster
> > > > available then use psudo mode) and use the web application interface
> to
> > > > Hadoop. You will clearly see all counters associated with the job and
> > you
> > > > can take it from there.
> > > > I like the notion of creating custom counters to obtain specific
> > metrics
> > > > but this is solely driven by user requirements.
> > > > Do you want to learn more about counters? Look into the code.
> > > > Do you want to know more about Nutch counters, or make the counters
> > more
> > > > explicit? Then consider opening a Jira issue and we can discuss this
> in
> > > > more detail.
> > > > With regards to the Fetcher, there are many possible areas where
> > counters
> > > > are (and could be) really useful... as I said though this s only
> driven
> > > by
> > > > user requirements.
> > > >
> > >
> > >
> > >
> > > --
> > > Don't Grow Old, Grow Up... :-)
> > >
> >
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>
Re: Understanding fetch MapReduce job counters and logs
Posted by feng lu <am...@gmail.com>.
yes, the property is fetcher.timelimit.mins. if you not set this property,
the QueueFeeder will not filter the url and log output may like this
QueueFeeder finished: total 36651 records + hit by time limit :0
Do you use bin/crawl command script. it will set the time limit for
fetching to 180. the code like this
# time limit for fetching
timeLimitFetch=180
# fetching the segment
echo "Fetching : $SEGMENT"
$bin/nutch fetch $commonOptions -D fetcher.timelimit.mins=$timeLimitFetch
$CRAWL_PATH/segments/$SEGMENT -noParsing -threads $numThreads
On Sun, Mar 17, 2013 at 7:47 PM, Amit Sela <am...@infolinks.com> wrote:
> By fetcher.limit property,do you mean fetcher.timelimit.mins ? because I
> have it set on default (-1) - no time limit.
>
>
> On Sat, Mar 16, 2013 at 5:12 PM, feng lu <am...@gmail.com> wrote:
>
> > Hi Amit
> >
> > <<
> > I also note
> > that the total hit by time limit here is 50927 but the job counters show
> > 7493.
> > >>
> >
> > This two time limits are all set bye fetcher.limit property. One is used
> in
> > QueueFeeder class, indicate that the QueueFeeder should finish load data
> if
> > current time is larger than time limit. So the total hit by time limit is
> > 50927. Another is used in FetchItemQueues class, indicate that check the
> > time if current time is larger than time limit and feeder has stopped ,
> > emptying the queues, So here job counters of time limit is 7493. There
> are
> > not equal.
> >
> >
> > <<
> > Summing all of theses numbers does equal the total map input.
> > >>
> >
> > do you set the property of "fetcher.follow.outlinks.depth", when
> > fetcher.parse is true and this value is greater than 0 the fetcher will
> > extract outlinks
> > and follow until the desired depth is reached.
> >
> > Another reason is that when this page is redirect to another page , fetch
> > will add new redirect page to fetch queues, so you can see that map input
> > is not equal to numbers of all sum.
> >
> >
> >
> >
> >
> >
> > On Sat, Mar 16, 2013 at 8:03 AM, Lewis John Mcgibbney <
> > lewis.mcgibbney@gmail.com> wrote:
> >
> > > Hi Amit,
> > >
> > > I know this thread is a bit old now, however it is also something which
> > > bugged me when I was looking into something else (InjectorJob
> counters).
> > >
> > > On Tue, Mar 5, 2013 at 3:16 AM, Amit Sela <am...@infolinks.com> wrote:
> > >
> > > >
> > > > And summing all counters does not equal the total map input...
> > > >
> > > > Summing all of theses numbers does equal the total map input. I also
> > note
> > > > that the total hit by time limit here is 50927 but the job counters
> > show
> > > > 7493.
> > > >
> > > >
> > > Basically, the easiest way to see and generally understand counters is
> to
> > > run the Nutch application within your Hadoop cluster (if no cluster
> > > available then use psudo mode) and use the web application interface to
> > > Hadoop. You will clearly see all counters associated with the job and
> you
> > > can take it from there.
> > > I like the notion of creating custom counters to obtain specific
> metrics
> > > but this is solely driven by user requirements.
> > > Do you want to learn more about counters? Look into the code.
> > > Do you want to know more about Nutch counters, or make the counters
> more
> > > explicit? Then consider opening a Jira issue and we can discuss this in
> > > more detail.
> > > With regards to the Fetcher, there are many possible areas where
> counters
> > > are (and could be) really useful... as I said though this s only driven
> > by
> > > user requirements.
> > >
> >
> >
> >
> > --
> > Don't Grow Old, Grow Up... :-)
> >
>
--
Don't Grow Old, Grow Up... :-)
Re: Understanding fetch MapReduce job counters and logs
Posted by Amit Sela <am...@infolinks.com>.
By fetcher.limit property,do you mean fetcher.timelimit.mins ? because I
have it set on default (-1) - no time limit.
On Sat, Mar 16, 2013 at 5:12 PM, feng lu <am...@gmail.com> wrote:
> Hi Amit
>
> <<
> I also note
> that the total hit by time limit here is 50927 but the job counters show
> 7493.
> >>
>
> This two time limits are all set bye fetcher.limit property. One is used in
> QueueFeeder class, indicate that the QueueFeeder should finish load data if
> current time is larger than time limit. So the total hit by time limit is
> 50927. Another is used in FetchItemQueues class, indicate that check the
> time if current time is larger than time limit and feeder has stopped ,
> emptying the queues, So here job counters of time limit is 7493. There are
> not equal.
>
>
> <<
> Summing all of theses numbers does equal the total map input.
> >>
>
> do you set the property of "fetcher.follow.outlinks.depth", when
> fetcher.parse is true and this value is greater than 0 the fetcher will
> extract outlinks
> and follow until the desired depth is reached.
>
> Another reason is that when this page is redirect to another page , fetch
> will add new redirect page to fetch queues, so you can see that map input
> is not equal to numbers of all sum.
>
>
>
>
>
>
> On Sat, Mar 16, 2013 at 8:03 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi Amit,
> >
> > I know this thread is a bit old now, however it is also something which
> > bugged me when I was looking into something else (InjectorJob counters).
> >
> > On Tue, Mar 5, 2013 at 3:16 AM, Amit Sela <am...@infolinks.com> wrote:
> >
> > >
> > > And summing all counters does not equal the total map input...
> > >
> > > Summing all of theses numbers does equal the total map input. I also
> note
> > > that the total hit by time limit here is 50927 but the job counters
> show
> > > 7493.
> > >
> > >
> > Basically, the easiest way to see and generally understand counters is to
> > run the Nutch application within your Hadoop cluster (if no cluster
> > available then use psudo mode) and use the web application interface to
> > Hadoop. You will clearly see all counters associated with the job and you
> > can take it from there.
> > I like the notion of creating custom counters to obtain specific metrics
> > but this is solely driven by user requirements.
> > Do you want to learn more about counters? Look into the code.
> > Do you want to know more about Nutch counters, or make the counters more
> > explicit? Then consider opening a Jira issue and we can discuss this in
> > more detail.
> > With regards to the Fetcher, there are many possible areas where counters
> > are (and could be) really useful... as I said though this s only driven
> by
> > user requirements.
> >
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>
Re: Understanding fetch MapReduce job counters and logs
Posted by feng lu <am...@gmail.com>.
Hi Amit
<<
I also note
that the total hit by time limit here is 50927 but the job counters show
7493.
>>
This two time limits are all set bye fetcher.limit property. One is used in
QueueFeeder class, indicate that the QueueFeeder should finish load data if
current time is larger than time limit. So the total hit by time limit is
50927. Another is used in FetchItemQueues class, indicate that check the
time if current time is larger than time limit and feeder has stopped ,
emptying the queues, So here job counters of time limit is 7493. There are
not equal.
<<
Summing all of theses numbers does equal the total map input.
>>
do you set the property of "fetcher.follow.outlinks.depth", when
fetcher.parse is true and this value is greater than 0 the fetcher will
extract outlinks
and follow until the desired depth is reached.
Another reason is that when this page is redirect to another page , fetch
will add new redirect page to fetch queues, so you can see that map input
is not equal to numbers of all sum.
On Sat, Mar 16, 2013 at 8:03 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi Amit,
>
> I know this thread is a bit old now, however it is also something which
> bugged me when I was looking into something else (InjectorJob counters).
>
> On Tue, Mar 5, 2013 at 3:16 AM, Amit Sela <am...@infolinks.com> wrote:
>
> >
> > And summing all counters does not equal the total map input...
> >
> > Summing all of theses numbers does equal the total map input. I also note
> > that the total hit by time limit here is 50927 but the job counters show
> > 7493.
> >
> >
> Basically, the easiest way to see and generally understand counters is to
> run the Nutch application within your Hadoop cluster (if no cluster
> available then use psudo mode) and use the web application interface to
> Hadoop. You will clearly see all counters associated with the job and you
> can take it from there.
> I like the notion of creating custom counters to obtain specific metrics
> but this is solely driven by user requirements.
> Do you want to learn more about counters? Look into the code.
> Do you want to know more about Nutch counters, or make the counters more
> explicit? Then consider opening a Jira issue and we can discuss this in
> more detail.
> With regards to the Fetcher, there are many possible areas where counters
> are (and could be) really useful... as I said though this s only driven by
> user requirements.
>
--
Don't Grow Old, Grow Up... :-)
Re: Understanding fetch MapReduce job counters and logs
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Amit,
I know this thread is a bit old now, however it is also something which
bugged me when I was looking into something else (InjectorJob counters).
On Tue, Mar 5, 2013 at 3:16 AM, Amit Sela <am...@infolinks.com> wrote:
>
> And summing all counters does not equal the total map input...
>
> Summing all of theses numbers does equal the total map input. I also note
> that the total hit by time limit here is 50927 but the job counters show
> 7493.
>
>
Basically, the easiest way to see and generally understand counters is to
run the Nutch application within your Hadoop cluster (if no cluster
available then use psudo mode) and use the web application interface to
Hadoop. You will clearly see all counters associated with the job and you
can take it from there.
I like the notion of creating custom counters to obtain specific metrics
but this is solely driven by user requirements.
Do you want to learn more about counters? Look into the code.
Do you want to know more about Nutch counters, or make the counters more
explicit? Then consider opening a Jira issue and we can discuss this in
more detail.
With regards to the Fetcher, there are many possible areas where counters
are (and could be) really useful... as I said though this s only driven by
user requirements.