how to restrict search result in defined domains?

[ianwong <yi...@hotmail.com>]

hi, everyone

I used nutch to crawl about 150 websites, result is quite good. 

If I only want to search result in a list of defined domains, how can I make
it?

Thanks!

Ian
-- 
View this message in context: http://www.nabble.com/how-to-restrict-search-result--in-defined-domains--tp23136172p23136172.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: how to restrict search result in defined domains?

[Dennis Kubes <ku...@apache.org>]

You can use the domain-urlfilter to limit the crawl to specific domains. 
  Limiting the search to specific domains would take one of the following:

1) A plugin that changes the search query to include "and (site:xxx or 
site:xxx).  I don't recommend this way, just including for completeness.

2) Another field in your index, say sitegroup, with value x.  Same value 
for all 150 sites.  Then do query and sitegroup:x.

Dennis

Ian.huang wrote:
> Dmitry,
> 
> Thanks. What I want is to restrict search result in limited domains 
> after whole index has built.
> 
> say, I built index for 150 sites, but want to search index from 5 of 
> them. It is like searching by : site:www.a.com or site:www.b.com or 
> site:www.c.com
> 
> Ian
> 
> --------------------------------------------------
> From: "Dmitry Lihachev" <ld...@openteam.ru>
> Sent: Wednesday, April 22, 2009 7:45 AM
> To: <nu...@lucene.apache.org>
> Subject: Re: how to restrict search result  in defined domains?
> 
>> Hi Ian.
>>> hi, everyone
>>>
>>> I used nutch to crawl about 150 websites, result is quite good.
>>>
>>> If I only want to search result in a list of defined domains, how can I
>>> make it?
>>>
>>> Thanks!
>>>
>>> Ian
>>
>> Put below lines in your nutch-site.xml
>>
>> <property>
>>  <name>db.ignore.external.links</name>
>>  <value>true</value>
>>  <description>If true, outlinks leading from a page to external hosts
>>  will be ignored. This is an effective way to limit the crawl to include
>>  only initially injected hosts, without creating complex URLFilters.
>>  </description>
>> </property>
>>
>> -- 
>> Regards,
>> Dmitry Lihachev
>>

Re: how to restrict search result in defined domains?

["Ian.huang" <yi...@hotmail.com>]

Dmitry,

Thanks. What I want is to restrict search result in limited domains after 
whole index has built.

say, I built index for 150 sites, but want to search index from 5 of them. 
It is like searching by : site:www.a.com or site:www.b.com or site:www.c.com

Ian

--------------------------------------------------
From: "Dmitry Lihachev" <ld...@openteam.ru>
Sent: Wednesday, April 22, 2009 7:45 AM
To: <nu...@lucene.apache.org>
Subject: Re: how to restrict search result  in defined domains?

> Hi Ian.
>> hi, everyone
>>
>> I used nutch to crawl about 150 websites, result is quite good.
>>
>> If I only want to search result in a list of defined domains, how can I
>> make it?
>>
>> Thanks!
>>
>> Ian
>
> Put below lines in your nutch-site.xml
>
> <property>
>  <name>db.ignore.external.links</name>
>  <value>true</value>
>  <description>If true, outlinks leading from a page to external hosts
>  will be ignored. This is an effective way to limit the crawl to include
>  only initially injected hosts, without creating complex URLFilters.
>  </description>
> </property>
>
> -- 
> Regards,
> Dmitry Lihachev
> 

Re: how to restrict search result in defined domains?

[Dmitry Lihachev <ld...@openteam.ru>]

Hi Ian.
> hi, everyone
>
> I used nutch to crawl about 150 websites, result is quite good.
>
> If I only want to search result in a list of defined domains, how can I
> make it?
>
> Thanks!
>
> Ian

Put below lines in your nutch-site.xml

<property>
  <name>db.ignore.external.links</name>
  <value>true</value>
  <description>If true, outlinks leading from a page to external hosts
  will be ignored. This is an effective way to limit the crawl to include
  only initially injected hosts, without creating complex URLFilters.
  </description>
</property>

-- 
Regards,
Dmitry Lihachev