You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Raghavendra Prabhu <rr...@gmail.com> on 2006/01/29 13:35:38 UTC
benchmark and performance
Is there any benchmark on how nutch performs
I mean say like 1 GB of data is given as the input , how much time it will
take to index this data on a 10 Mb/s network
And while doing crawl what is the volume of data which can be loaded (this
is in terms of search . How much can a crawl segment hold )
Is there any performance limit to it . Is the criteria only the space to
store the indexed data .Is there any limit to it ?
Rgds
Prabhu
Re: Updating the search index
Posted by Byron Miller <by...@yahoo.com>.
With all of the discussions of
killing/restarting/pooling nutch bean has anyone
noticed that you push your luck in doing so?
I often get GC failed to collect, out of memory errors
and such when trying to do anything but a clean
shutdown.
I'm moving to 64bit jvm and java 1.5 so i'll let you
know if those memory errors continue.
-byron
--- Raghavendra Prabhu <rr...@gmail.com> wrote:
> With respect to updating , I had also suggested
> another method
>
> Where we control NutchBean instantiation
>
> But i introduced it into the form of object pooling
>
> This pool will take care of reinstatiating nutch
> bean and returning the
> reference to it
> The pool can have a text file as an input which
> changes on reindexing . From
> this it will understand and recreate new search
> index and return the
> reference to the bean
>
>
>
>
>
>
> On 2/1/06, Raghavendra Prabhu <rr...@gmail.com>
> wrote:
> >
> >
> > Maybe We should have a close method also in
> IndexSearcher
> >
> > Which gives greater flexibility while deleting
> files
> >
> > But even after implemention , the data is locked
> by some search mechanism
> >
> > Maybe Some ArrayFileReader is holding it
> >
> > Are'nt we supposed to release these handlers
> >
> > Rgds
> >
> > Prabhu
> >
> >
> > On 2/1/06, Raghavendra Prabhu
> <rr...@gmail.com> wrote:
> > >
> > > Hi guys
> > >
> > > Even i face the same problem
> > >
> > > I am doing something similar to what howie is
> doing
> > >
> > > But i want to delete the existing files in the
> index and replace them
> > >
> > >
> > > I even implement a new method which calls
> close.This closes the
> > >
> > > Indexsearcher and the reader.
> > >
> > > Then also i am not able to delete the file in
> windows
> > >
> > > Do i have to close anything other than the
> indexsearcher and the reader
> > > to delete the files
> > >
> > >
> > > On 2/1/06, Howie Wang <howie_wang@hotmail.com >
> wrote:
> > > >
> > > > I haven't tested this out too well, but I've
> been using it on my
> > > > development box. In case people are wondering
> how to get a
> > > > new NutchBean which will re-read the index,
> what I did was write
> > > > a little JSP page called reset_nutch_bean.jsp.
> All it does it remove
> > > > the "nutchBean" attr from the application
> context. So the next
> > > > search that comes along will create a new
> bean.
> > > >
> > > > Any time I upload a new index, I just run this
> page. Pretty easy.
> > > > It's not automated, but it's easy enough to do
> after uploading
> > > > an index. You could automate it by adding it
> to a script
> > > > after finishing an index or after doing an
> upload or just run it
> > > > periodically with cron.
> > > >
> > > > The code is after my sig. Let me know if what
> I've done is bad in
> > > > some way.
> > > >
> > > > Howie
> > > >
> > > > <%@ page
> > > > contentType="text/html; charset=UTF-8"
> > > > pageEncoding="UTF-8"
> > > >
> > > > import="javax.servlet.*"
> > > > import="javax.servlet.http.*"
> > > >
> > > > %>
> > > > <%
> > > >
> application.removeAttribute("nutchBean");
> > > > %>
> > > >
> > > >
> > > >
> > >
> >
>
Re: Updating the search index
Posted by Raghavendra Prabhu <rr...@gmail.com>.
With respect to updating , I had also suggested another method
Where we control NutchBean instantiation
But i introduced it into the form of object pooling
This pool will take care of reinstatiating nutch bean and returning the
reference to it
The pool can have a text file as an input which changes on reindexing . From
this it will understand and recreate new search index and return the
reference to the bean
On 2/1/06, Raghavendra Prabhu <rr...@gmail.com> wrote:
>
>
> Maybe We should have a close method also in IndexSearcher
>
> Which gives greater flexibility while deleting files
>
> But even after implemention , the data is locked by some search mechanism
>
> Maybe Some ArrayFileReader is holding it
>
> Are'nt we supposed to release these handlers
>
> Rgds
>
> Prabhu
>
>
> On 2/1/06, Raghavendra Prabhu <rr...@gmail.com> wrote:
> >
> > Hi guys
> >
> > Even i face the same problem
> >
> > I am doing something similar to what howie is doing
> >
> > But i want to delete the existing files in the index and replace them
> >
> >
> > I even implement a new method which calls close.This closes the
> >
> > Indexsearcher and the reader.
> >
> > Then also i am not able to delete the file in windows
> >
> > Do i have to close anything other than the indexsearcher and the reader
> > to delete the files
> >
> >
> > On 2/1/06, Howie Wang <howie_wang@hotmail.com > wrote:
> > >
> > > I haven't tested this out too well, but I've been using it on my
> > > development box. In case people are wondering how to get a
> > > new NutchBean which will re-read the index, what I did was write
> > > a little JSP page called reset_nutch_bean.jsp. All it does it remove
> > > the "nutchBean" attr from the application context. So the next
> > > search that comes along will create a new bean.
> > >
> > > Any time I upload a new index, I just run this page. Pretty easy.
> > > It's not automated, but it's easy enough to do after uploading
> > > an index. You could automate it by adding it to a script
> > > after finishing an index or after doing an upload or just run it
> > > periodically with cron.
> > >
> > > The code is after my sig. Let me know if what I've done is bad in
> > > some way.
> > >
> > > Howie
> > >
> > > <%@ page
> > > contentType="text/html; charset=UTF-8"
> > > pageEncoding="UTF-8"
> > >
> > > import="javax.servlet.*"
> > > import="javax.servlet.http.*"
> > >
> > > %>
> > > <%
> > > application.removeAttribute("nutchBean");
> > > %>
> > >
> > >
> > >
> >
>
Re: Updating the search index
Posted by Raghavendra Prabhu <rr...@gmail.com>.
Maybe We should have a close method also in IndexSearcher
Which gives greater flexibility while deleting files
But even after implemention , the data is locked by some search mechanism
Maybe Some ArrayFileReader is holding it
Are'nt we supposed to release these handlers
Rgds
Prabhu
On 2/1/06, Raghavendra Prabhu <rr...@gmail.com> wrote:
>
> Hi guys
>
> Even i face the same problem
>
> I am doing something similar to what howie is doing
>
> But i want to delete the existing files in the index and replace them
>
>
> I even implement a new method which calls close.This closes the
>
> Indexsearcher and the reader.
>
> Then also i am not able to delete the file in windows
>
> Do i have to close anything other than the indexsearcher and the reader to
> delete the files
>
>
> On 2/1/06, Howie Wang <ho...@hotmail.com> wrote:
> >
> > I haven't tested this out too well, but I've been using it on my
> > development box. In case people are wondering how to get a
> > new NutchBean which will re-read the index, what I did was write
> > a little JSP page called reset_nutch_bean.jsp. All it does it remove
> > the "nutchBean" attr from the application context. So the next
> > search that comes along will create a new bean.
> >
> > Any time I upload a new index, I just run this page. Pretty easy.
> > It's not automated, but it's easy enough to do after uploading
> > an index. You could automate it by adding it to a script
> > after finishing an index or after doing an upload or just run it
> > periodically with cron.
> >
> > The code is after my sig. Let me know if what I've done is bad in
> > some way.
> >
> > Howie
> >
> > <%@ page
> > contentType="text/html; charset=UTF-8"
> > pageEncoding="UTF-8"
> >
> > import="javax.servlet.*"
> > import="javax.servlet.http.*"
> >
> > %>
> > <%
> > application.removeAttribute("nutchBean");
> > %>
> >
> >
> >
>
Re: Updating the search index
Posted by Raghavendra Prabhu <rr...@gmail.com>.
Hi guys
Even i face the same problem
I am doing something similar to what howie is doing
But i want to delete the existing files in the index and replace them
I even implement a new method which calls close.This closes the
Indexsearcher and the reader.
Then also i am not able to delete the file in windows
Do i have to close anything other than the indexsearcher and the reader to
delete the files
On 2/1/06, Howie Wang <ho...@hotmail.com> wrote:
>
> I haven't tested this out too well, but I've been using it on my
> development box. In case people are wondering how to get a
> new NutchBean which will re-read the index, what I did was write
> a little JSP page called reset_nutch_bean.jsp. All it does it remove
> the "nutchBean" attr from the application context. So the next
> search that comes along will create a new bean.
>
> Any time I upload a new index, I just run this page. Pretty easy.
> It's not automated, but it's easy enough to do after uploading
> an index. You could automate it by adding it to a script
> after finishing an index or after doing an upload or just run it
> periodically with cron.
>
> The code is after my sig. Let me know if what I've done is bad in
> some way.
>
> Howie
>
> <%@ page
> contentType="text/html; charset=UTF-8"
> pageEncoding="UTF-8"
>
> import="javax.servlet.*"
> import="javax.servlet.http.*"
>
> %>
> <%
> application.removeAttribute("nutchBean");
> %>
>
>
>
Re: Updating the search index
Posted by Howie Wang <ho...@hotmail.com>.
I haven't tested this out too well, but I've been using it on my
development box. In case people are wondering how to get a
new NutchBean which will re-read the index, what I did was write
a little JSP page called reset_nutch_bean.jsp. All it does it remove
the "nutchBean" attr from the application context. So the next
search that comes along will create a new bean.
Any time I upload a new index, I just run this page. Pretty easy.
It's not automated, but it's easy enough to do after uploading
an index. You could automate it by adding it to a script
after finishing an index or after doing an upload or just run it
periodically with cron.
The code is after my sig. Let me know if what I've done is bad in
some way.
Howie
<%@ page
contentType="text/html; charset=UTF-8"
pageEncoding="UTF-8"
import="javax.servlet.*"
import="javax.servlet.http.*"
%>
<%
application.removeAttribute("nutchBean");
%>
Re: benchmark and performance
Posted by Raghavendra Prabhu <rr...@gmail.com>.
Hi
Thanks . I got the explanation .So with mapreduce we will be able to process
crawldb efficiently
Rgds
Prabhu
On 1/31/06, Byron Miller <by...@yahoo.com> wrote:
>
> Prabhu,
>
> For nutch .7x the upper limit of webdb isn't
> necessarily file size but hardware/computation size.
> You basically need 210% of your webdb size to do any
> processing of it so if you have 100 million urls and a
> 1.5 terrabyte webdb you need (on the same server) 3.7
> terrabytes of disk space to process the webdb, do the
> updates, drop the tmp file and update your main webdb.
>
> For .8/mapred branch it breaks down the jobs to
> smaller map jobs and doesn't have one huge file that
> consumes everything. You can better spend what was
> wasted before distributed across more systems for
> redundancy and performance.
>
>
> Hopefully that makes sense.
>
> -byron
> --- Raghavendra Prabhu <rr...@gmail.com> wrote:
>
> > Hi Stefan
> >
> > Thanks for your mail
> >
> > What i would like to know is (since i am using
> > nutch-0.7) ,what is the upper
> > limit on the webdb size if any such limit exists in
> > nutch-0.7
> >
> > Will the generate for a web db formed from one TB of
> > data (just an example)
> > work ?
> >
> > And what is the difference between webdb and
> > nutch-0.8 (crawldb and linkdb)
> > which makes it infinitely possible in nutch-0.8?
> >
> > Rgds
> > Prabhu
> >
> >
> >
> > On 1/30/06, Stefan Groschupf <sg...@media-style.com>
> > wrote:
> > >
> > > You can already use ndfs in 0.7, however in case
> > the webdb is to
> > > lareg it took to much time to generate segments.
> > > So the problem is the webdb size, not the hdd
> > limit.
> > >
> > > Am 30.01.2006 um 07:31 schrieb Raghavendra Prabhu:
> > >
> > > > Hi Stefan
> > > >
> > > > So can i assume that hard disk space is the only
> > constraint in
> > > > nutch-0.7
> > > >
> > > > In nutch-0.8 since you can store it over the
> > ndfs , it is
> > > > theoretically
> > > > unlimited .
> > > >
> > > > Is my above mentioned point true ( In a nutshell
> > , i want to know
> > > > whether
> > > > the only thing is the space for storing the
> > nutch indexed date)
> > > >
> > > > I will try to do some testing and if possible
> > contribute to wiki.
> > > >
> > > > Rgds
> > > >
> > > > Prabhu
> > > >
> > > >
> > > >
> > > > On 1/30/06, Stefan Groschupf
> > <sg...@media-style.com> wrote:
> > > >>
> > > >> Any performance testing contribution to the
> > wiki is welcome, I
> > > >> guess. :)
> > > >> So there are no such values except of some
> > statements regarding
> > > >> search speed in the wiki,
> > > >> With nutch .8 there theoretically no size limit
> > any more.
> > > >> Stefan
> > > >> Am 29.01.2006 um 13:35 schrieb Raghavendra
> > Prabhu:
> > > >>
> > > >>> Is there any benchmark on how nutch performs
> > > >>>
> > > >>> I mean say like 1 GB of data is given as the
> > input , how much time
> > > >>> it will
> > > >>> take to index this data on a 10 Mb/s network
> > > >>>
> > > >>> And while doing crawl what is the volume of
> > data which can be
> > > >>> loaded (this
> > > >>> is in terms of search . How much can a crawl
> > segment hold )
> > > >>>
> > > >>> Is there any performance limit to it . Is the
> > criteria only the
> > > >>> space to
> > > >>> store the indexed data .Is there any limit to
> > it ?
> > > >>>
> > > >>> Rgds
> > > >>>
> > > >>> Prabhu
> > > >>
> > > >>
> >
> ---------------------------------------------------------------
> > > >> company: http://www.media-style.com
> > > >> forum: http://www.text-mining.org
> > > >> blog: http://www.find23.net
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> > >
> >
>
>
Re: benchmark and performance
Posted by Byron Miller <by...@yahoo.com>.
Prabhu,
For nutch .7x the upper limit of webdb isn't
necessarily file size but hardware/computation size.
You basically need 210% of your webdb size to do any
processing of it so if you have 100 million urls and a
1.5 terrabyte webdb you need (on the same server) 3.7
terrabytes of disk space to process the webdb, do the
updates, drop the tmp file and update your main webdb.
For .8/mapred branch it breaks down the jobs to
smaller map jobs and doesn't have one huge file that
consumes everything. You can better spend what was
wasted before distributed across more systems for
redundancy and performance.
Hopefully that makes sense.
-byron
--- Raghavendra Prabhu <rr...@gmail.com> wrote:
> Hi Stefan
>
> Thanks for your mail
>
> What i would like to know is (since i am using
> nutch-0.7) ,what is the upper
> limit on the webdb size if any such limit exists in
> nutch-0.7
>
> Will the generate for a web db formed from one TB of
> data (just an example)
> work ?
>
> And what is the difference between webdb and
> nutch-0.8 (crawldb and linkdb)
> which makes it infinitely possible in nutch-0.8?
>
> Rgds
> Prabhu
>
>
>
> On 1/30/06, Stefan Groschupf <sg...@media-style.com>
> wrote:
> >
> > You can already use ndfs in 0.7, however in case
> the webdb is to
> > lareg it took to much time to generate segments.
> > So the problem is the webdb size, not the hdd
> limit.
> >
> > Am 30.01.2006 um 07:31 schrieb Raghavendra Prabhu:
> >
> > > Hi Stefan
> > >
> > > So can i assume that hard disk space is the only
> constraint in
> > > nutch-0.7
> > >
> > > In nutch-0.8 since you can store it over the
> ndfs , it is
> > > theoretically
> > > unlimited .
> > >
> > > Is my above mentioned point true ( In a nutshell
> , i want to know
> > > whether
> > > the only thing is the space for storing the
> nutch indexed date)
> > >
> > > I will try to do some testing and if possible
> contribute to wiki.
> > >
> > > Rgds
> > >
> > > Prabhu
> > >
> > >
> > >
> > > On 1/30/06, Stefan Groschupf
> <sg...@media-style.com> wrote:
> > >>
> > >> Any performance testing contribution to the
> wiki is welcome, I
> > >> guess. :)
> > >> So there are no such values except of some
> statements regarding
> > >> search speed in the wiki,
> > >> With nutch .8 there theoretically no size limit
> any more.
> > >> Stefan
> > >> Am 29.01.2006 um 13:35 schrieb Raghavendra
> Prabhu:
> > >>
> > >>> Is there any benchmark on how nutch performs
> > >>>
> > >>> I mean say like 1 GB of data is given as the
> input , how much time
> > >>> it will
> > >>> take to index this data on a 10 Mb/s network
> > >>>
> > >>> And while doing crawl what is the volume of
> data which can be
> > >>> loaded (this
> > >>> is in terms of search . How much can a crawl
> segment hold )
> > >>>
> > >>> Is there any performance limit to it . Is the
> criteria only the
> > >>> space to
> > >>> store the indexed data .Is there any limit to
> it ?
> > >>>
> > >>> Rgds
> > >>>
> > >>> Prabhu
> > >>
> > >>
>
---------------------------------------------------------------
> > >> company: http://www.media-style.com
> > >> forum: http://www.text-mining.org
> > >> blog: http://www.find23.net
> > >>
> > >>
> > >>
> > >>
> >
> >
>
Re: benchmark and performance
Posted by Raghavendra Prabhu <rr...@gmail.com>.
Hi Stefan
Thanks for your mail
What i would like to know is (since i am using nutch-0.7) ,what is the upper
limit on the webdb size if any such limit exists in nutch-0.7
Will the generate for a web db formed from one TB of data (just an example)
work ?
And what is the difference between webdb and nutch-0.8 (crawldb and linkdb)
which makes it infinitely possible in nutch-0.8?
Rgds
Prabhu
On 1/30/06, Stefan Groschupf <sg...@media-style.com> wrote:
>
> You can already use ndfs in 0.7, however in case the webdb is to
> lareg it took to much time to generate segments.
> So the problem is the webdb size, not the hdd limit.
>
> Am 30.01.2006 um 07:31 schrieb Raghavendra Prabhu:
>
> > Hi Stefan
> >
> > So can i assume that hard disk space is the only constraint in
> > nutch-0.7
> >
> > In nutch-0.8 since you can store it over the ndfs , it is
> > theoretically
> > unlimited .
> >
> > Is my above mentioned point true ( In a nutshell , i want to know
> > whether
> > the only thing is the space for storing the nutch indexed date)
> >
> > I will try to do some testing and if possible contribute to wiki.
> >
> > Rgds
> >
> > Prabhu
> >
> >
> >
> > On 1/30/06, Stefan Groschupf <sg...@media-style.com> wrote:
> >>
> >> Any performance testing contribution to the wiki is welcome, I
> >> guess. :)
> >> So there are no such values except of some statements regarding
> >> search speed in the wiki,
> >> With nutch .8 there theoretically no size limit any more.
> >> Stefan
> >> Am 29.01.2006 um 13:35 schrieb Raghavendra Prabhu:
> >>
> >>> Is there any benchmark on how nutch performs
> >>>
> >>> I mean say like 1 GB of data is given as the input , how much time
> >>> it will
> >>> take to index this data on a 10 Mb/s network
> >>>
> >>> And while doing crawl what is the volume of data which can be
> >>> loaded (this
> >>> is in terms of search . How much can a crawl segment hold )
> >>>
> >>> Is there any performance limit to it . Is the criteria only the
> >>> space to
> >>> store the indexed data .Is there any limit to it ?
> >>>
> >>> Rgds
> >>>
> >>> Prabhu
> >>
> >> ---------------------------------------------------------------
> >> company: http://www.media-style.com
> >> forum: http://www.text-mining.org
> >> blog: http://www.find23.net
> >>
> >>
> >>
> >>
>
>
Re: benchmark and performance
Posted by Stefan Groschupf <sg...@media-style.com>.
You can already use ndfs in 0.7, however in case the webdb is to
lareg it took to much time to generate segments.
So the problem is the webdb size, not the hdd limit.
Am 30.01.2006 um 07:31 schrieb Raghavendra Prabhu:
> Hi Stefan
>
> So can i assume that hard disk space is the only constraint in
> nutch-0.7
>
> In nutch-0.8 since you can store it over the ndfs , it is
> theoretically
> unlimited .
>
> Is my above mentioned point true ( In a nutshell , i want to know
> whether
> the only thing is the space for storing the nutch indexed date)
>
> I will try to do some testing and if possible contribute to wiki.
>
> Rgds
>
> Prabhu
>
>
>
> On 1/30/06, Stefan Groschupf <sg...@media-style.com> wrote:
>>
>> Any performance testing contribution to the wiki is welcome, I
>> guess. :)
>> So there are no such values except of some statements regarding
>> search speed in the wiki,
>> With nutch .8 there theoretically no size limit any more.
>> Stefan
>> Am 29.01.2006 um 13:35 schrieb Raghavendra Prabhu:
>>
>>> Is there any benchmark on how nutch performs
>>>
>>> I mean say like 1 GB of data is given as the input , how much time
>>> it will
>>> take to index this data on a 10 Mb/s network
>>>
>>> And while doing crawl what is the volume of data which can be
>>> loaded (this
>>> is in terms of search . How much can a crawl segment hold )
>>>
>>> Is there any performance limit to it . Is the criteria only the
>>> space to
>>> store the indexed data .Is there any limit to it ?
>>>
>>> Rgds
>>>
>>> Prabhu
>>
>> ---------------------------------------------------------------
>> company: http://www.media-style.com
>> forum: http://www.text-mining.org
>> blog: http://www.find23.net
>>
>>
>>
>>
Re: benchmark and performance
Posted by Raghavendra Prabhu <rr...@gmail.com>.
Hi Stefan
So can i assume that hard disk space is the only constraint in nutch-0.7
In nutch-0.8 since you can store it over the ndfs , it is theoretically
unlimited .
Is my above mentioned point true ( In a nutshell , i want to know whether
the only thing is the space for storing the nutch indexed date)
I will try to do some testing and if possible contribute to wiki.
Rgds
Prabhu
On 1/30/06, Stefan Groschupf <sg...@media-style.com> wrote:
>
> Any performance testing contribution to the wiki is welcome, I guess. :)
> So there are no such values except of some statements regarding
> search speed in the wiki,
> With nutch .8 there theoretically no size limit any more.
> Stefan
> Am 29.01.2006 um 13:35 schrieb Raghavendra Prabhu:
>
> > Is there any benchmark on how nutch performs
> >
> > I mean say like 1 GB of data is given as the input , how much time
> > it will
> > take to index this data on a 10 Mb/s network
> >
> > And while doing crawl what is the volume of data which can be
> > loaded (this
> > is in terms of search . How much can a crawl segment hold )
> >
> > Is there any performance limit to it . Is the criteria only the
> > space to
> > store the indexed data .Is there any limit to it ?
> >
> > Rgds
> >
> > Prabhu
>
> ---------------------------------------------------------------
> company: http://www.media-style.com
> forum: http://www.text-mining.org
> blog: http://www.find23.net
>
>
>
>
Re: benchmark and performance
Posted by Stefan Groschupf <sg...@media-style.com>.
Any performance testing contribution to the wiki is welcome, I guess. :)
So there are no such values except of some statements regarding
search speed in the wiki,
With nutch .8 there theoretically no size limit any more.
Stefan
Am 29.01.2006 um 13:35 schrieb Raghavendra Prabhu:
> Is there any benchmark on how nutch performs
>
> I mean say like 1 GB of data is given as the input , how much time
> it will
> take to index this data on a 10 Mb/s network
>
> And while doing crawl what is the volume of data which can be
> loaded (this
> is in terms of search . How much can a crawl segment hold )
>
> Is there any performance limit to it . Is the criteria only the
> space to
> store the indexed data .Is there any limit to it ?
>
> Rgds
>
> Prabhu
---------------------------------------------------------------
company: http://www.media-style.com
forum: http://www.text-mining.org
blog: http://www.find23.net