file to http mapping

[Jeff Breidenbach <je...@jab.org>]

Hi all,

I'm playing with Nutch to index a single website with a lot of static
HTML files. It would be nice to crawl and index using the local
filesytem to keep from slamming the webserver. However, actual
searches should work with http urls.

I looked for this feature in code and documentation, and don't
anything that helps. Specifically, I want to say something like:

   file:///var/lib/www  ->  http://foo.bar.baz/

Does it already exist? Should I try to write it? Where to start?

Cheers,
Jeff

Re: file to http mapping

[Nutch Newbie <nu...@gmail.com>]

Hi:

I agree. It would be nice if one could do this. Some sort of mapping
based on pre-defined value. There is a plugin that might be of value.
You could start by looking at "Creative-Commons" plugin. Maybe one
could modify the file-protocol plugin to implement such option.

Just some thoughts.

Regards

On 12/26/05, Jeff Breidenbach <je...@jab.org> wrote:
>
> Hi all,
>
> I'm playing with Nutch to index a single website with a lot of static
> HTML files. It would be nice to crawl and index using the local
> filesytem to keep from slamming the webserver. However, actual
> searches should work with http urls.
>
> I looked for this feature in code and documentation, and don't
> anything that helps. Specifically, I want to say something like:
>
>   file:///var/lib/www  ->  http://foo.bar.baz/
>
> Does it already exist? Should I try to write it? Where to start?
>
> Cheers,
> Jeff
>

Re: file to http mapping

[Jeff Breidenbach <je...@jab.org>]

Thank you, this approach worked nicely.

On Tue, 27 Dec 2005 3:03 am, Stefan Groschupf wrote:
> Jeff,
> no such solution does not exists.
> Take a look to the index-filters.
> I suggest following solution:
> + write a index filter that add a field 'webserverUrl' to the lucene  
> document that contains your rewritten  url (should by from type  
> keyword).
> + change the jsp page such that the 'webserverUrl' is used as link  url 
> instead the real url.
> Chainging the url behind the sense make no sense since nutch uses the  
> url as key for all records.
>
> HTH
> Stefan

Re: file to http mapping

[Stefan Groschupf <sg...@media-style.com>]

Jeff,
no such solution does not exists.
Take a look to the index-filters.
I suggest following solution:
+ write a index filter that add a field 'webserverUrl' to the lucene  
document that contains your rewritten  url (should by from type  
keyword).
+ change the jsp page such that the 'webserverUrl' is used as link  
url instead the real url.
Chainging the url behind the sense make no sense since nutch uses the  
url as key for all records.

HTH
Stefan

Am 26.12.2005 um 15:21 schrieb Jeff Breidenbach:

>
> Hi all,
>
> I'm playing with Nutch to index a single website with a lot of static
> HTML files. It would be nice to crawl and index using the local
> filesytem to keep from slamming the webserver. However, actual
> searches should work with http urls.
>
> I looked for this feature in code and documentation, and don't
> anything that helps. Specifically, I want to say something like:
>
>    file:///var/lib/www  ->  http://foo.bar.baz/
>
> Does it already exist? Should I try to write it? Where to start?
>
> Cheers,
> Jeff
>