You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by eyal edri <ey...@gmail.com> on 2007/09/03 13:15:34 UTC
Fetching single / choosen URL's
Hi everyone,
Anyone knows if there is an option to fetch a single or a group of wanted
urls using the fetcher, but (!!) not fetching previous links extracted from
urls which have already been fetched (meaning depth 1+x).
for e.g.
if i did a inject-->generate-->fetch-->update loop once
now the crawldb is updated with depth 1 links from the root url list.
i now want to inject a new url list for fetching, but want to exclude the
current links from fetching.
--
Eyal Edri
Re: Fetching single / choosen URL's
Posted by eyal edri <ey...@gmail.com>.
i think i found an answer.
Since you can always inject-generate-fetch into a different db - you can use
the mergedb command.
what exaclty do i mean?
1. Inject to crawldb1-->Generate-->fetch-->update (now the fetch list is
updated with depth 1 links)
2. Inject(single url) to crawldb2-->Generate-->fetch (will only fetch the
one url injected)-->update
3. merge crawl1 and crawl2 into crawl using mergedb command.
i think this will do the trick...
On 9/3/07, Marcin Okraszewski <ok...@o2.pl> wrote:
> If I understand you correctly, you want to fetch only injected URLs and
> you do not want to follow any link found in fetched pages. I'm not aware of
> any configuration that could do this easily. But you could:
>
> 1. Use regex URL filter to allow only the links you want. Eg. add all URLs
> you inject to allowed URLs, but it can be very expensive.
> 2. Write your own custom URL filter that would accept only your URLs.
> 3. Write a HtmlParserFilter which will override the parse output and
> remove all links from the parsed page. But then, you won't have those links
> in your crawl db.
>
> Regards,
> Marcin
>
> > Hi everyone,
> >
> > Anyone knows if there is an option to fetch a single or a group of
> wanted
> > urls using the fetcher, but (!!) not fetching previous links extracted
> from
> > urls which have already been fetched (meaning depth 1+x).
> >
> > for e.g.
> >
> > if i did a inject-->generate-->fetch-->update loop once
> > now the crawldb is updated with depth 1 links from the root url list.
> >
> > i now want to inject a new url list for fetching, but want to exclude
> the
> > current links from fetching.
> >
> >
> >
> > --
> > Eyal Edri
>
>
--
Eyal Edri
Re: Fetching single / choosen URL's
Posted by Marcin Okraszewski <ok...@o2.pl>.
If I understand you correctly, you want to fetch only injected URLs and you do not want to follow any link found in fetched pages. I'm not aware of any configuration that could do this easily. But you could:
1. Use regex URL filter to allow only the links you want. Eg. add all URLs you inject to allowed URLs, but it can be very expensive.
2. Write your own custom URL filter that would accept only your URLs.
3. Write a HtmlParserFilter which will override the parse output and remove all links from the parsed page. But then, you won't have those links in your crawl db.
Regards,
Marcin
> Hi everyone,
>
> Anyone knows if there is an option to fetch a single or a group of wanted
> urls using the fetcher, but (!!) not fetching previous links extracted from
> urls which have already been fetched (meaning depth 1+x).
>
> for e.g.
>
> if i did a inject-->generate-->fetch-->update loop once
> now the crawldb is updated with depth 1 links from the root url list.
>
> i now want to inject a new url list for fetching, but want to exclude the
> current links from fetching.
>
>
>
> --
> Eyal Edri