Re: Storing website urls instead of complete urls in index

[Dennis Kubes <ku...@apache.org>]

Hi Shreemoyee,

As each page is stored by key in the different Nutch files and this key 
is the url, stripping down the URL to just its domain wouldn't work 
unless you only had a single page per domain.  All Nutch programs, 
including generator and fetcher, work off of the URL as key.

You can extract the domain from the key using something like this:

String host = URLUtil.getHost(key.toString());

And if you are looking to store it during the fetch/parse I would 
suggest looking at storing it in the crawl or parse metadata.  To do 
this though you may have to modify the Fetcher job.

Dennis

On 05/30/2010 02:22 AM, Shreemoyee Sarkar wrote:
> Hi,
>
> Is it possible to store the website url instead of complete url without
> affecting the crawl?
>
> e.g. store http://example.com instead of
> http://example.com/foo/bar/page.html
>
> would the generate URLs and fetch for the subsequent depths go smoothly?
>
> Thanks
> Shreemoyee
>
>