crawl-urlfilter.txt & regex-urlfilter.txt

[Ken Ken <ke...@yahoo.com>]

Hello,

I have a few questions on these two files in nutch-1.0/conf


I am doing the whole-web crawling that has seven steps:

1.  bin/nutch inject
2.  bin/nutch generate
3.  bin/nutch fetch
4.  bin/nutch updatedb
5.  bin/nutch invertlinks
6.  bin/nutch index
7.  bin/nutch dedup

My questions:  

1. At which step will the'+^http://([a-z0-9]*\.)*apache.org/' in crawl-urlfilter.txt & regex-urlfilter.txt come in to filter out the urls?
2. If I want to only allow certain top-level-domains (e.g. .org), do I put the below in both urlfilter.txt files 

+^http://([a-z0-9]*\.)ca       <== (abc.ca)
+^http://([a-z0-9]*\.)*.ca      <== (subdomain.abc.ca)
+^http://([a-z0-9]*\.)*com.ca   <==    (subdomain.com.ca)

Is the above right?

3.  If I just want any domain with .ca extension (abc.ca, abc.com.ca, abc.net.ca, abc.org.ca), can I just use one filter line?
If yes, what do I put?

Thanks in advance.
Ken


      

Re: crawl-urlfilter.txt & regex-urlfilter.txt

[MilleBii <mi...@gmail.com>]

Crawl-urlfilter.txt is only used when using the nutch crawl command
and not when running individual commands.

Also I miss step 8.

+Nutch merge

Which is essential if you ever want to search the index... This the
step that makes it Lucene/Luke usable.


2010/1/6, J.G.Konrad <ko...@gmail.com>:
> Hi Ken,
>   If you are doing a whole-web crawl then only the regex-urlfiter.txt file
> is used. The regex-filter.txt is used during the inject and fetchlist
> generation. You can also use the regex-urlfitler when updating the crawldb
> but you will have to add the '-filter' option when running the updatedb
> command.
>
>   -Jason
> On Wed, Jan 6, 2010 at 7:04 AM, Godmar Back <go...@gmail.com> wrote:
>
>> On Wed, Jan 6, 2010 at 5:36 AM, Ken Ken <ke...@yahoo.com> wrote:
>>
>> >
>> >
>> > Hello,
>> >
>> > I have a few questions on these two files in nutch-1.0/conf
>> >
>> >
>>
>> Ken,
>>
>> I can't answer your question, but I'm struggling with a similar one;
>> perhaps
>> both questions can be answered at the same time.
>> My question is which crawl-urlfilter.txt is being used when, and if it
>> matters.
>>
>> If you have your nutch in say /opt/nutch, and your tomcat in /opt/tomcat,
>> then after deploying the .war you have /opt/nutch/conf/crawl-urlfilter.txt
>> and /opt/tomcat/webapps/nutch/WEB-INF/classes/crawl-urlfilter.txt
>>
>> I'd assume that bin/nutch will look at the former, and Tomcat will look at
>> the latter, but how do you keep the two in sync?
>>
>> Specifically, if you rebuild nutch.war and redeploy, won't that override
>> the
>> crawl-urlfilter.txt in the WEB-INF tree?
>>
>>  - Godmar
>>
>


-- 
-MilleBii-

Re: crawl-urlfilter.txt & regex-urlfilter.txt

["J.G.Konrad" <ko...@gmail.com>]

Hi Ken,
  If you are doing a whole-web crawl then only the regex-urlfiter.txt file
is used. The regex-filter.txt is used during the inject and fetchlist
generation. You can also use the regex-urlfitler when updating the crawldb
but you will have to add the '-filter' option when running the updatedb
command.

  -Jason
On Wed, Jan 6, 2010 at 7:04 AM, Godmar Back <go...@gmail.com> wrote:

> On Wed, Jan 6, 2010 at 5:36 AM, Ken Ken <ke...@yahoo.com> wrote:
>
> >
> >
> > Hello,
> >
> > I have a few questions on these two files in nutch-1.0/conf
> >
> >
>
> Ken,
>
> I can't answer your question, but I'm struggling with a similar one;
> perhaps
> both questions can be answered at the same time.
> My question is which crawl-urlfilter.txt is being used when, and if it
> matters.
>
> If you have your nutch in say /opt/nutch, and your tomcat in /opt/tomcat,
> then after deploying the .war you have /opt/nutch/conf/crawl-urlfilter.txt
> and /opt/tomcat/webapps/nutch/WEB-INF/classes/crawl-urlfilter.txt
>
> I'd assume that bin/nutch will look at the former, and Tomcat will look at
> the latter, but how do you keep the two in sync?
>
> Specifically, if you rebuild nutch.war and redeploy, won't that override
> the
> crawl-urlfilter.txt in the WEB-INF tree?
>
>  - Godmar
>

Re: crawl-urlfilter.txt & regex-urlfilter.txt

[Godmar Back <go...@gmail.com>]

On Wed, Jan 6, 2010 at 5:36 AM, Ken Ken <ke...@yahoo.com> wrote:

>
>
> Hello,
>
> I have a few questions on these two files in nutch-1.0/conf
>
>

Ken,

I can't answer your question, but I'm struggling with a similar one; perhaps
both questions can be answered at the same time.
My question is which crawl-urlfilter.txt is being used when, and if it
matters.

If you have your nutch in say /opt/nutch, and your tomcat in /opt/tomcat,
then after deploying the .war you have /opt/nutch/conf/crawl-urlfilter.txt
and /opt/tomcat/webapps/nutch/WEB-INF/classes/crawl-urlfilter.txt

I'd assume that bin/nutch will look at the former, and Tomcat will look at
the latter, but how do you keep the two in sync?

Specifically, if you rebuild nutch.war and redeploy, won't that override the
crawl-urlfilter.txt in the WEB-INF tree?

 - Godmar