Index web folders.

[Shane Wood <sh...@cbm8bit.com>]

I have a few sites i mirror via FTP and would like too index them with 
nutch but not hit the site too save bandwidth.
/var/www/www.somesite.com/ lets say one is located in this folder o the 
same server nutch is located on, how would
you index it and have the search results point too the actual site. 
http://somesite.com ?

Thanks
Shane.

Re: Index web folders.

[John Lafitte <jl...@brandextract.com>]

Hi Shane,

The way I save on bandwidth for internal sites is by adding the internal IP
into the hosts file for the domain.  If it's the local machine you can
probably just point it to 127.0.0.1 but I kind of wonder if you would be
saving anything but a DNS lookup...  I'm not a networking guru but it
should route to itself automatically.


On Tue, Apr 8, 2014 at 8:30 PM, Shane Wood <sh...@cbm8bit.com> wrote:

> I have a few sites i mirror via FTP and would like too index them with
> nutch but not hit the site too save bandwidth.
> /var/www/www.somesite.com/ lets say one is located in this folder o the
> same server nutch is located on, how would
> you index it and have the search results point too the actual site.
> http://somesite.com ?
>
> Thanks
> Shane.
>