You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by remi tassing <ta...@gmail.com> on 2012/02/18 20:57:43 UTC
URLNormalizer not working properly
Hi,
I'm witnessing a weird problem. I configured regex-normalize.xml to escape
whitespaces, curly braces...and it works while checking with
URLNormalizerChecker:
*echo "URL non escaped" | bin/nutch
org.apache.nutch.net.URLNormalizerChecker*
*output: escaped URL*
But when I run crawl with Nutch, I can still see "bad" URLs being fetched.
Any explanation for this?
Remi
Re: URLNormalizer not working properly
Posted by remi tassing <ta...@gmail.com>.
Ok, it makes sense, thanks Markus!
Remi
On Saturday, February 18, 2012, Markus Jelsma <ma...@openindex.io>
wrote:
>
>> That works just fine!
>>
>> I wonder why crawldb has to be updated first. All these URLs are in
>> segments and similarly the regex-urlfilter works immediately without the
>> need of updating the db.
>
> They work immediately indeed, but the DB is not magically updated. Also,
> segment crawldatum objects are not normalized so they will not match
anymore
> with the contents of the crawldb.
>
>>
>> Any particular reason?
>>
>> Remi
>>
>> On Saturday, February 18, 2012, Markus Jelsma <markus.jelsma@openindex.io
>
>>
>> wrote:
>> > Did you update the entire crawldb with that normalizer?
>> >
>> >> Hi,
>> >>
>> >> I'm witnessing a weird problem. I configured regex-normalize.xml to
>>
>> escape
>>
>> >> whitespaces, curly braces...and it works while checking with
>> >> URLNormalizerChecker:
>> >> *echo "URL non escaped" | bin/nutch
>> >> org.apache.nutch.net.URLNormalizerChecker*
>> >> *output: escaped URL*
>> >>
>> >> But when I run crawl with Nutch, I can still see "bad" URLs being
>>
>> fetched.
>>
>> >> Any explanation for this?
>> >>
>> >> Remi
>
Re: URLNormalizer not working properly
Posted by Markus Jelsma <ma...@openindex.io>.
> That works just fine!
>
> I wonder why crawldb has to be updated first. All these URLs are in
> segments and similarly the regex-urlfilter works immediately without the
> need of updating the db.
They work immediately indeed, but the DB is not magically updated. Also,
segment crawldatum objects are not normalized so they will not match anymore
with the contents of the crawldb.
>
> Any particular reason?
>
> Remi
>
> On Saturday, February 18, 2012, Markus Jelsma <ma...@openindex.io>
>
> wrote:
> > Did you update the entire crawldb with that normalizer?
> >
> >> Hi,
> >>
> >> I'm witnessing a weird problem. I configured regex-normalize.xml to
>
> escape
>
> >> whitespaces, curly braces...and it works while checking with
> >> URLNormalizerChecker:
> >> *echo "URL non escaped" | bin/nutch
> >> org.apache.nutch.net.URLNormalizerChecker*
> >> *output: escaped URL*
> >>
> >> But when I run crawl with Nutch, I can still see "bad" URLs being
>
> fetched.
>
> >> Any explanation for this?
> >>
> >> Remi
Re: URLNormalizer not working properly
Posted by remi tassing <ta...@gmail.com>.
That works just fine!
I wonder why crawldb has to be updated first. All these URLs are in
segments and similarly the regex-urlfilter works immediately without the
need of updating the db.
Any particular reason?
Remi
On Saturday, February 18, 2012, Markus Jelsma <ma...@openindex.io>
wrote:
> Did you update the entire crawldb with that normalizer?
>
>> Hi,
>>
>> I'm witnessing a weird problem. I configured regex-normalize.xml to
escape
>> whitespaces, curly braces...and it works while checking with
>> URLNormalizerChecker:
>> *echo "URL non escaped" | bin/nutch
>> org.apache.nutch.net.URLNormalizerChecker*
>> *output: escaped URL*
>>
>> But when I run crawl with Nutch, I can still see "bad" URLs being
fetched.
>>
>> Any explanation for this?
>>
>> Remi
>
Re: URLNormalizer not working properly
Posted by Markus Jelsma <ma...@openindex.io>.
Are you using 1.4? It keeps a previous version of the DB in crawldb/old/ by
default.
> I had 18000 db_fetched, now only 54. Pretty dangerous command :-(
>
> On Saturday, February 18, 2012, Markus Jelsma <ma...@openindex.io>
>
> wrote:
> > Did you update the entire crawldb with that normalizer?
> >
> >> Hi,
> >>
> >> I'm witnessing a weird problem. I configured regex-normalize.xml to
>
> escape
>
> >> whitespaces, curly braces...and it works while checking with
> >> URLNormalizerChecker:
> >> *echo "URL non escaped" | bin/nutch
> >> org.apache.nutch.net.URLNormalizerChecker*
> >> *output: escaped URL*
> >>
> >> But when I run crawl with Nutch, I can still see "bad" URLs being
>
> fetched.
>
> >> Any explanation for this?
> >>
> >> Remi
Re: URLNormalizer not working properly
Posted by remi tassing <ta...@gmail.com>.
I had 18000 db_fetched, now only 54. Pretty dangerous command :-(
On Saturday, February 18, 2012, Markus Jelsma <ma...@openindex.io>
wrote:
> Did you update the entire crawldb with that normalizer?
>
>> Hi,
>>
>> I'm witnessing a weird problem. I configured regex-normalize.xml to
escape
>> whitespaces, curly braces...and it works while checking with
>> URLNormalizerChecker:
>> *echo "URL non escaped" | bin/nutch
>> org.apache.nutch.net.URLNormalizerChecker*
>> *output: escaped URL*
>>
>> But when I run crawl with Nutch, I can still see "bad" URLs being
fetched.
>>
>> Any explanation for this?
>>
>> Remi
>
Re: URLNormalizer not working properly
Posted by Markus Jelsma <ma...@openindex.io>.
Did you update the entire crawldb with that normalizer?
> Hi,
>
> I'm witnessing a weird problem. I configured regex-normalize.xml to escape
> whitespaces, curly braces...and it works while checking with
> URLNormalizerChecker:
> *echo "URL non escaped" | bin/nutch
> org.apache.nutch.net.URLNormalizerChecker*
> *output: escaped URL*
>
> But when I run crawl with Nutch, I can still see "bad" URLs being fetched.
>
> Any explanation for this?
>
> Remi