nutch readdb db -stats & Pages Fetched Not Consistent

[Bill Goffe <go...@Oswego.EDU>]

Now that I've got my reverse proxy up, one less pressing question. With
  ~/nutch/bin/nutch  readdb db -stats
I get 
  Number of pages: 1,399,730
  Number of links: 5,369,361
Yet when I search my log (have a perl script that outputs everything to
it) with "grep fetching log  | wc -l" I get
  313,998
(commas added).

I'm thinking that readdb db -stats isn't counting the number of downloaded
pages -- am I correct?

         - Bill

-- 
         *------------------------------------------------------*
         | Bill Goffe                 goffe@oswego.edu          |
         | Department of Economics    voice: (315) 312-3444     |
         | SUNY Oswego                fax:   (315) 312-5444     |
         | 416 Mahar Hall             <http://cook.rfe.org>     |          
         | Oswego, NY  13126                                    |
*--------*------------------------------------------------------*-----------*
| "It shocks them actually. They never knew that such a world actually      |
| exists, because they have their own problems."                            |
|  -- Mustapha Ahansal, of the U.S. Navy. He was describing the response of |
|     Iraqis to his boarding party of himself (a Moroccan-American), an     |
|     African-American, an Hispanic, and often a woman. "Sinbad vs. the     |
|     Mermaids," Thomas L. Friedman, New York Times, 10/5/05.               |
*---------------------------------------------------------------------------*

Re: nutch readdb db -stats & Pages Fetched Not Consistent

[Andrzej Bialecki <ab...@getopt.org>]

Bill Goffe wrote:

>Now that I've got my reverse proxy up, one less pressing question. With
>  ~/nutch/bin/nutch  readdb db -stats
>I get 
>  Number of pages: 1,399,730
>  Number of links: 5,369,361
>Yet when I search my log (have a perl script that outputs everything to
>it) with "grep fetching log  | wc -l" I get
>  313,998
>(commas added).
>
>I'm thinking that readdb db -stats isn't counting the number of downloaded
>pages -- am I correct?
>  
>

Correct. -stats gives you the total number of unique URLs in the db, 
whether they are fetched or not. So, you can easily calculate how many 
pages you still need to crawl... ;-)

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com