You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Shameema Umer <sh...@gmail.com> on 2012/06/07 12:09:59 UTC
recrawl a certain site
I would like to recrawl a certain site I admin at a much smaller interval
- say, every 10 mins.
I've specified my db.fetch.interval.default to be 1200, but if I attempt
to recrawl using the crawl directory and segments from last time, I can't
seem to get it to work. I think the problem is adddays... is there any
way to reuse my crawl segments with a fetch segment less than 1 day? I
currently delete my old crawl directory. While that works, I would like
to preserve the entries and timestamps and such in there between crawls.
Is this possible?
Re: recrawl a certain site
Posted by David MISTRETTA <go...@gmail.com>.
Bonjour,
Y a t-il des utilisateurs nutch français ?
Y is there any users nutch French
Cordialement, david
Re: recrawl a certain site
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Simple search for script recrawl returned
http://www.mail-archive.com/user@nutch.apache.org/msg00655.html
Although I've not mined the content, I would imagine your answer (or
something similar) will be be in or around those search results.
hth
Lewis
On Thu, Jun 7, 2012 at 5:37 PM, Shameema Umer <sh...@gmail.com> wrote:
> Hi Lewis,
>
> Please specify the link of user at archives.
>
> Thanks.Shameema
>
> On Thu, Jun 7, 2012 at 4:35 PM, Lewis John Mcgibbney
> <le...@gmail.com> wrote:
>>
>> utilise a script of some sort and specify these values in the script.
>> This is a question which comes up quite regularly and there should be
>> plenty of threads on the topic on the user@ archives.
>>
>> hth
>>
>> On Thu, Jun 7, 2012 at 11:09 AM, Shameema Umer <sh...@gmail.com> wrote:
>> > I would like to recrawl a certain site I admin at a much smaller interval
>> > - say, every 10 mins.
>> >
>> > I've specified my db.fetch.interval.default to be 1200, but if I attempt
>> > to recrawl using the crawl directory and segments from last time, I can't
>> > seem to get it to work. I think the problem is adddays... is there any
>> > way to reuse my crawl segments with a fetch segment less than 1 day? I
>> > currently delete my old crawl directory. While that works, I would like
>> > to preserve the entries and timestamps and such in there between crawls.
>> > Is this possible?
>>
>>
>>
>> --
>> Lewis
--
Lewis
Re: recrawl a certain site
Posted by Shameema Umer <sh...@gmail.com>.
Hi Lewis,
Please specify the link of user at archives.
Thanks.Shameema
On Thu, Jun 7, 2012 at 4:35 PM, Lewis John Mcgibbney
<le...@gmail.com> wrote:
>
> utilise a script of some sort and specify these values in the script.
> This is a question which comes up quite regularly and there should be
> plenty of threads on the topic on the user@ archives.
>
> hth
>
> On Thu, Jun 7, 2012 at 11:09 AM, Shameema Umer <sh...@gmail.com> wrote:
> > I would like to recrawl a certain site I admin at a much smaller interval
> > - say, every 10 mins.
> >
> > I've specified my db.fetch.interval.default to be 1200, but if I attempt
> > to recrawl using the crawl directory and segments from last time, I can't
> > seem to get it to work. I think the problem is adddays... is there any
> > way to reuse my crawl segments with a fetch segment less than 1 day? I
> > currently delete my old crawl directory. While that works, I would like
> > to preserve the entries and timestamps and such in there between crawls.
> > Is this possible?
>
>
>
> --
> Lewis
Re: recrawl a certain site
Posted by Lewis John Mcgibbney <le...@gmail.com>.
utilise a script of some sort and specify these values in the script.
This is a question which comes up quite regularly and there should be
plenty of threads on the topic on the user@ archives.
hth
On Thu, Jun 7, 2012 at 11:09 AM, Shameema Umer <sh...@gmail.com> wrote:
> I would like to recrawl a certain site I admin at a much smaller interval
> - say, every 10 mins.
>
> I've specified my db.fetch.interval.default to be 1200, but if I attempt
> to recrawl using the crawl directory and segments from last time, I can't
> seem to get it to work. I think the problem is adddays... is there any
> way to reuse my crawl segments with a fetch segment less than 1 day? I
> currently delete my old crawl directory. While that works, I would like
> to preserve the entries and timestamps and such in there between crawls.
> Is this possible?
--
Lewis