You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Zhou LiBing <zh...@gmail.com> on 2005/08/19 12:52:18 UTC

about the nutch function

Can Nutch use one or more start URL to crawl the WEB?
Can Nutch fetch the IMAGE file?
  thank you
 

-- 
---Letter From your friend Blue at HUST CGCL---

Re: about the nutch function

Posted by Jack Tang <hi...@gmail.com>.
Hi Zhou

On 8/19/05, Zhou LiBing <zh...@gmail.com> wrote:
> Can Nutch use one or more start URL to crawl the WEB?
Yes. you can set the seeds in seed file one line by one line
> Can Nutch fetch the IMAGE file?
Yes.
>   thank you
> 

Reagrds
/Jack
> --
> ---Letter From your friend Blue at HUST CGCL---
> 
> 


-- 
Keep Discovering ... ...
http://www.jroller.com/page/jmars

Re: Combining multiple index to single index

Posted by "George A. Papayiannis" <pa...@gmail.com>.
Hi,

Thanks for the reply -- I using that script -- I see now what I was doing 
wrong -- but still I'm not completly satisfied

I have my directories of:

./db
./segments (has many segments inside each indexed in its segment)
./index (main index)

I can I would run: nutch merge indexTemp segments/* ./
This would merge all the segment index's and the main index into directory 
indexTemp

Then I would overwrite indexTemp with index

in any case, playing with it more, i'm sure i'll figure something out

George

On 8/21/05, Michael Ji <fj...@yahoo.com> wrote:
> 
> I mean using "bin/nutch merge"
> 
> --- "George A. Papayiannis" <pa...@gmail.com>
> wrote:
> 
> > Hi,
> >
> > I am in a scenario where I have many small indexes
> > and would like to merge
> > them into a single index. I know I could do searches
> > from multiple indexes,
> > but I would rather combine them. Does anyone know of
> > a way to do this, other
> > than using the Lucene API?
> >
> > Thanks,
> > George
> >
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>

Re: Combining multiple index to single index

Posted by Michael Ji <fj...@yahoo.com>.
I mean using "bin/nutch merge"

--- "George A. Papayiannis" <pa...@gmail.com>
wrote:

> Hi,
> 
> I am in a scenario where I have many small indexes
> and would like to merge 
> them into a single index. I know I could do searches
> from multiple indexes, 
> but I would rather combine them. Does anyone know of
> a way to do this, other 
> than using the Lucene API?
> 
> Thanks,
> George
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Combining multiple index to single index

Posted by Michael Ji <fj...@yahoo.com>.
You can use nutch built-in script to do index merging;

run "bin/nutch mergesegs" to see the required
parameters;

Michael Ji


--- "George A. Papayiannis" <pa...@gmail.com>
wrote:

> Hi,
> 
> I am in a scenario where I have many small indexes
> and would like to merge 
> them into a single index. I know I could do searches
> from multiple indexes, 
> but I would rather combine them. Does anyone know of
> a way to do this, other 
> than using the Lucene API?
> 
> Thanks,
> George
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Combining multiple index to single index

Posted by "George A. Papayiannis" <pa...@gmail.com>.
Hi,

I am in a scenario where I have many small indexes and would like to merge 
them into a single index. I know I could do searches from multiple indexes, 
but I would rather combine them. Does anyone know of a way to do this, other 
than using the Lucene API?

Thanks,
George

Re: [Nutch-general] Re: about the nutch function

Posted by Zhou LiBing <zh...@gmail.com>.
thank u, I will try

 2005/8/21, Fuad Efendi <fu...@efendi.ca>: 
> 
> If you need search engine - you can use Nutch.
> For front-end, you can use links to images from source sites, it's
> better for performance of your own site.
> 
> If you need just to collect image files, probably best is
> http://htmlparser.sourceforge.net/
> org.htmlparser.parserapplications.SiteCapturer
> However, I'd prefer to write some additional readers for Nutch! Better
> for troubleshooting, I need it.
> Thanks
> 
> 
> -----Original Message-----
> From: Zhou LiBing [mailto:zhoulibing@gmail.com]
> Sent: Saturday, August 20, 2005 7:59 PM
> To: user@nutch.org
> Subject: Re: [Nutch-general] Re: about the nutch function
> 
> 
> can I specify more than one URL to crawl the whole web?are you sure? how
> to edit the crawl-urlfilter.txt to fetch the images?how could I extract
> these* segment *images' feature?
> thank you
> 
> 
> 2005/8/19, Piotr Kosiorowski <pk...@gmail.com>:
> >
> > Yes.Yes. :).
> > You can specify more than one url while injecting pages to WebDB. You
> > can fetch the image file (you have to edit crawl-urlfilter.txt or
> > regex-urlfilter.txt to allow particular extension as majority of image
> 
> > extensions are blocked by default). But such data would be only stored
> 
> > in segment - I do not think it would be accesible by search.
> > P.
> >
> >
> > On 8/19/05, Zhou LiBing <zh...@gmail.com> wrote:
> > > Can Nutch use one or more start URL to crawl the WEB?
> > > Can Nutch fetch the IMAGE file?
> > > thank you
> > >
> > >
> > > --
> > > ---Letter From your friend Blue at HUST CGCL---
> > >
> > >
> >
> >
> > -------------------------------------------------------
> > SF.Net email is Sponsored by the Better Software Conference & EXPO
> > September 19-22, 2005 * San Francisco, CA * Development Lifecycle
> > Practices Agile & Plan-Driven Development * Managing Projects & Teams
> > * Testing & QA Security * Process Improvement & Measurement *
> > http://www.sqe.com/bsce5sf
> > _______________________________________________
> > Nutch-general mailing list Nutch-general@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nutch-general
> >
> 
> 
> 
> --
> ---Letter From your friend Blue at HUST CGCL---
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle 
> Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> _______________________________________________
> Nutch-general mailing list
> Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
> 



-- 
---Letter From your friend Blue at HUST CGCL---

RE: [Nutch-general] Re: about the nutch function

Posted by Fuad Efendi <fu...@efendi.ca>.
If you need search engine - you can use Nutch.
For front-end, you can use links to images from source sites, it's
better for performance of your own site.

If you need just to collect image files, probably best is
http://htmlparser.sourceforge.net/ 
org.htmlparser.parserapplications.SiteCapturer
However, I'd prefer to write some additional readers for Nutch! Better
for troubleshooting, I need it.
Thanks


-----Original Message-----
From: Zhou LiBing [mailto:zhoulibing@gmail.com] 
Sent: Saturday, August 20, 2005 7:59 PM
To: user@nutch.org
Subject: Re: [Nutch-general] Re: about the nutch function


can I specify more than one URL to crawl the whole web?are you sure? how
to edit the crawl-urlfilter.txt to fetch the images?how could I extract 
these* segment *images' feature?
 thank you
   

 2005/8/19, Piotr Kosiorowski <pk...@gmail.com>: 
> 
> Yes.Yes. :).
> You can specify more than one url while injecting pages to WebDB. You 
> can fetch the image file (you have to edit crawl-urlfilter.txt or 
> regex-urlfilter.txt to allow particular extension as majority of image

> extensions are blocked by default). But such data would be only stored

> in segment - I do not think it would be accesible by search.
> P.
> 
> 
> On 8/19/05, Zhou LiBing <zh...@gmail.com> wrote:
> > Can Nutch use one or more start URL to crawl the WEB?
> > Can Nutch fetch the IMAGE file?
> > thank you
> >
> >
> > --
> > ---Letter From your friend Blue at HUST CGCL---
> >
> >
> 
> 
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO 
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle 
> Practices Agile & Plan-Driven Development * Managing Projects & Teams 
> * Testing & QA Security * Process Improvement & Measurement * 
> http://www.sqe.com/bsce5sf 
> _______________________________________________
> Nutch-general mailing list Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
> 



-- 
---Letter From your friend Blue at HUST CGCL---


RE: [Nutch-general] Re: about the nutch function

Posted by Fuad Efendi <fu...@efendi.ca>.
Such data is called "Content" (Java Class), it has HTTP headers, you can
check HTTP headers ("text/html", "jpg", etc.... MIME extensions)
It is stored in folder "content"...
You can extract it, and save in hard-drive as regular files, you need
specific code (Nutch does not have it). Simply extract 
Check in Fetcher class, "ArrayFile.Writer contentWriter;"
You need also to modify url-filter files...

Content.getContentType() - MIME extension
byte[] Content.getContent() - file content (JPG, HTML, DOC, PDF, ...)




-----Original Message-----
From: Zhou LiBing [mailto:zhoulibing@gmail.com] 
Sent: Saturday, August 20, 2005 7:59 PM
To: user@nutch.org
Subject: Re: [Nutch-general] Re: about the nutch function


can I specify more than one URL to crawl the whole web?are you sure? how
to edit the crawl-urlfilter.txt to fetch the images?how could I extract 
these* segment *images' feature?
 thank you
   

 2005/8/19, Piotr Kosiorowski <pk...@gmail.com>: 
> 
> Yes.Yes. :).
> You can specify more than one url while injecting pages to WebDB. You 
> can fetch the image file (you have to edit crawl-urlfilter.txt or 
> regex-urlfilter.txt to allow particular extension as majority of image

> extensions are blocked by default). But such data would be only stored

> in segment - I do not think it would be accesible by search.
> P.
> 
> 
> On 8/19/05, Zhou LiBing <zh...@gmail.com> wrote:
> > Can Nutch use one or more start URL to crawl the WEB?
> > Can Nutch fetch the IMAGE file?
> > thank you
> >
> >
> > --
> > ---Letter From your friend Blue at HUST CGCL---
> >
> >
> 
> 
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO 
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle 
> Practices Agile & Plan-Driven Development * Managing Projects & Teams 
> * Testing & QA Security * Process Improvement & Measurement * 
> http://www.sqe.com/bsce5sf 
> _______________________________________________
> Nutch-general mailing list Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
> 



-- 
---Letter From your friend Blue at HUST CGCL---


Re: [Nutch-general] Re: about the nutch function

Posted by Zhou LiBing <zh...@gmail.com>.
can I specify more than one URL to crawl the whole web?are you sure?
how to edit the crawl-urlfilter.txt to fetch the images?how could I extract 
these* segment *images' feature?
 thank you
   

 2005/8/19, Piotr Kosiorowski <pk...@gmail.com>: 
> 
> Yes.Yes. :).
> You can specify more than one url while injecting pages to WebDB.
> You can fetch the image file (you have to edit crawl-urlfilter.txt or
> regex-urlfilter.txt to allow particular extension as majority of image
> extensions are blocked by default).
> But such data would be only stored in segment - I do not think it
> would be accesible by search.
> P.
> 
> 
> On 8/19/05, Zhou LiBing <zh...@gmail.com> wrote:
> > Can Nutch use one or more start URL to crawl the WEB?
> > Can Nutch fetch the IMAGE file?
> > thank you
> >
> >
> > --
> > ---Letter From your friend Blue at HUST CGCL---
> >
> >
> 
> 
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle 
> Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> _______________________________________________
> Nutch-general mailing list
> Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
> 



-- 
---Letter From your friend Blue at HUST CGCL---

Re: about the nutch function

Posted by Piotr Kosiorowski <pk...@gmail.com>.
Yes.Yes. :).
You can specify more than one url while injecting pages to WebDB.
You can fetch the image file (you have to edit crawl-urlfilter.txt or
regex-urlfilter.txt to allow particular extension as majority of image
extensions are blocked by default).
But such data would be only stored in segment - I do not think it
would be accesible by search.
P.


On 8/19/05, Zhou LiBing <zh...@gmail.com> wrote:
> Can Nutch use one or more start URL to crawl the WEB?
> Can Nutch fetch the IMAGE file?
>   thank you
> 
> 
> --
> ---Letter From your friend Blue at HUST CGCL---
> 
>