Duplicate pages with and without www. prefix being indexed

[Arthur Yarwood <ar...@fubaby.com>]

I have a Nutch 1.10 and Solr 5.2 setup. Still playing around with it all 
and quite new to me, but I've noticed for one site I have crawled, I'm 
getting content indexed twice in Solr, once with the www. domain prefix 
and once without. E.g.

http://www.example.com/somepage.html
http://example.com/somepage.html

How can I avoid this duplication? At least from being indexed into Solr.
And preferable a generic solution that will work with any other site I 
crawl in the future, some of which may default to www.xyz.com and some 
xyz.com. i.e. I know I could add a regex-urlfilter for this one domain, 
but I'd like to avoid this duplication in call instances in arise.

Thanks!


-- 
Arthur Yarwood

Re: Duplicate pages with and without www. prefix being indexed

[Arthur Yarwood <ar...@fubaby.com>]

Cool, thanks, that looks like it may do the job.
Am I correct in saying I need to define mapping for each specific domain 
in host-urlnormalizer.txt? Or can I add a generic rule to cover any host?

i.e.
www.example1.org .example1.org
www.example2.org .example2.org
www.example3.org .example3.org
...

or can I do something along the lines of:-
www.* .($1)

Arthur.

On 07/07/2015 09:01, Markus Jelsma wrote:
> You can use the host normalizer for this.
> https://issues.apache.org/jira/browse/NUTCH-1319
>   
> -----Original message-----
>> From:Arthur Yarwood <ar...@fubaby.com>
>> Sent: Tuesday 7th July 2015 0:02
>> To: user@nutch.apache.org
>> Subject: Duplicate pages with and without www. prefix being indexed
>>
>> I have a Nutch 1.10 and Solr 5.2 setup. Still playing around with it all
>> and quite new to me, but I've noticed for one site I have crawled, I'm
>> getting content indexed twice in Solr, once with the www. domain prefix
>> and once without. E.g.
>>
>> http://www.example.com/somepage.html
>> http://example.com/somepage.html
>>
>> How can I avoid this duplication? At least from being indexed into Solr.
>> And preferable a generic solution that will work with any other site I
>> crawl in the future, some of which may default to www.xyz.com and some
>> xyz.com. i.e. I know I could add a regex-urlfilter for this one domain,
>> but I'd like to avoid this duplication in call instances in arise.
>>
>> Thanks!
>>
>>
>> -- 
>> Arthur Yarwood
>>
>>


-- 
Arthur Yarwood
http://www.fubaby.com

RE: Duplicate pages with and without www. prefix being indexed

[Markus Jelsma <ma...@openindex.io>]

You can use the host normalizer for this.
https://issues.apache.org/jira/browse/NUTCH-1319
 
-----Original message-----
> From:Arthur Yarwood <ar...@fubaby.com>
> Sent: Tuesday 7th July 2015 0:02
> To: user@nutch.apache.org
> Subject: Duplicate pages with and without www. prefix being indexed
> 
> I have a Nutch 1.10 and Solr 5.2 setup. Still playing around with it all 
> and quite new to me, but I've noticed for one site I have crawled, I'm 
> getting content indexed twice in Solr, once with the www. domain prefix 
> and once without. E.g.
> 
> http://www.example.com/somepage.html
> http://example.com/somepage.html
> 
> How can I avoid this duplication? At least from being indexed into Solr.
> And preferable a generic solution that will work with any other site I 
> crawl in the future, some of which may default to www.xyz.com and some 
> xyz.com. i.e. I know I could add a regex-urlfilter for this one domain, 
> but I'd like to avoid this duplication in call instances in arise.
> 
> Thanks!
> 
> 
> -- 
> Arthur Yarwood
> 
>