You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Feng Ji <fe...@gmail.com> on 2006/08/30 14:40:43 UTC
when to use cmd "parse" to parse a segment's pages
hi,
I follow the nutch08 tutorial. The step to do crawling is "inject,
generator, fetch, update".
But there is a command in nutch/bin, called "parse", which parse a segment's
page. I wonder if I should use it before "update" in the above steps.
Currently, I didn't use "parse" cmd and "update" still see the parsed
outlinks. Seems some kind of parsing is already done in "fetch" step.
thanks,
Michael,
Re: when to use cmd "parse" to parse a segment's pages
Posted by Feng Ji <fe...@gmail.com>.
Exactly!
It solves my puzzle,
thanks,
Michael,
On 8/30/06, Zaheed Haque <za...@gmail.com> wrote:
>
> Hi
>
> Cos you have parse option true in nutch-site.xml. Try set it to false
> if you want to parse it manually. Or overide config with fetch
> -noParsing option.
>
> Cheers
>
> On 8/30/06, Feng Ji <fe...@gmail.com> wrote:
> > hi,
> >
> > I follow the nutch08 tutorial. The step to do crawling is "inject,
> > generator, fetch, update".
> >
> > But there is a command in nutch/bin, called "parse", which parse a
> segment's
> > page. I wonder if I should use it before "update" in the above steps.
> >
> > Currently, I didn't use "parse" cmd and "update" still see the parsed
> > outlinks. Seems some kind of parsing is already done in "fetch" step.
> >
> > thanks,
> >
> > Michael,
> >
> >
>
Re: when to use cmd "parse" to parse a segment's pages
Posted by Zaheed Haque <za...@gmail.com>.
Hi
Cos you have parse option true in nutch-site.xml. Try set it to false
if you want to parse it manually. Or overide config with fetch
-noParsing option.
Cheers
On 8/30/06, Feng Ji <fe...@gmail.com> wrote:
> hi,
>
> I follow the nutch08 tutorial. The step to do crawling is "inject,
> generator, fetch, update".
>
> But there is a command in nutch/bin, called "parse", which parse a segment's
> page. I wonder if I should use it before "update" in the above steps.
>
> Currently, I didn't use "parse" cmd and "update" still see the parsed
> outlinks. Seems some kind of parsing is already done in "fetch" step.
>
> thanks,
>
> Michael,
>
>