You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Burton-West, Tom" <tb...@umich.edu> on 2010/02/17 23:16:26 UTC
What is largest reasonable setting for ramBufferSizeMB?
Hello all,
At some point we will need to re-build an index that totals about 2 terrabytes in size (split over 10 shards). At our current indexing speed we estimate that this will take about 3 weeks. We would like to reduce that time. It appears that our main bottleneck is disk I/O.
We currently have ramBufferSizeMB set to 32 and our merge factor is 10. If we increase ramBufferSizeMB to 320, we avoid a merge and the 9 disk writes and reads to merge 9+1 32MB segments into a 320MB segment.
Assuming we allocate enough memory to the JVM, would it make sense to increase ramBufferSize to 3200MB? What are people's experiences with very large ramBufferSizeMB sizes?
Tom Burton-West
University of Michigan Library
www.hathitrust.org
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Feb 18, 2010 at 8:52 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> 32MB is very low, 320MB is medium, and I think you could go higher, just pick whichever garbage collector is good for throughput. I know Java 1.6 update 18 also has some Hotspot and maybe also GC fixes, so I'd use that.
I think you misread Tom's email - it sounds like you are talking about
JVM heap sizes, not ramBufferSizeMB?
32MB is certainly not low, and 320MB is not medium.
-Yonik
http://www.lucidimagination.com
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Jay Hill <ja...@gmail.com>.
Looks like multi-threaded support was added to the DIH recently:
http://issues.apache.org/jira/browse/SOLR-1352
-Jay
On Fri, Feb 19, 2010 at 6:27 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> Glen may be referring to LuSql indexing with multiple threads?
> Does/can DIH do that, too?
>
>
> Otis ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
>
> ----- Original Message ----
> > From: Yonik Seeley <yo...@lucidimagination.com>
> > To: solr-user@lucene.apache.org
> > Sent: Fri, February 19, 2010 11:41:07 AM
> > Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
> >
> > On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton wrote:
> > > You may consider using LuSql[1] to create the indexes, if your source
> > > content is in a JDBC accessible db. It is quite a bit faster than
> > > Solr, as it is a tool specifically created and tuned for Lucene
> > > indexing.
> >
> > Any idea why it's faster?
> > AFAIK, the main purpose of DIH is indexing databases too. If DIH is
> > much slower, we should speed it up!
> >
> > -Yonik
> > http://www.lucidimagination.com
>
>
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Glen may be referring to LuSql indexing with multiple threads?
Does/can DIH do that, too?
Otis ----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: Yonik Seeley <yo...@lucidimagination.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, February 19, 2010 11:41:07 AM
> Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
>
> On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton wrote:
> > You may consider using LuSql[1] to create the indexes, if your source
> > content is in a JDBC accessible db. It is quite a bit faster than
> > Solr, as it is a tool specifically created and tuned for Lucene
> > indexing.
>
> Any idea why it's faster?
> AFAIK, the main purpose of DIH is indexing databases too. If DIH is
> much slower, we should speed it up!
>
> -Yonik
> http://www.lucidimagination.com
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton <gl...@gmail.com> wrote:
> You may consider using LuSql[1] to create the indexes, if your source
> content is in a JDBC accessible db. It is quite a bit faster than
> Solr, as it is a tool specifically created and tuned for Lucene
> indexing.
Any idea why it's faster?
AFAIK, the main purpose of DIH is indexing databases too. If DIH is
much slower, we should speed it up!
-Yonik
http://www.lucidimagination.com
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Tom Burton-West <tb...@gmail.com>.
Hi Glen,
I'd love to use LuSql, but our data is not in a db. Its 6-8TB of files
containing OCR (one file per page for about 1.5 billion pages) gzipped on
disk which are ugzipped, concatenated, and converted to Solr documents
on-the-fly. We have multiple instances of our Solr document producer script
running. At this point we can run enough producers, so that the rate at
which Solr can ingest and index documents is our current bottleneck and so
far the bottleneck we see for indexing appears to be disk I/O for
Solr/Lucene during merges.
Is there any obvious relationship between the size of the ramBuffer and how
much heap you need to give the JVM, or is there some reasonable method of
finding this out by experimentation?
We would rather not find out by decreasing the amount of memory allocated to
the JVM until we get an OOM.
Tom
I've run Lucene with heap sizes as large as 28GB of RAM (on a 32GB
machine, 64bit, Linux) and a ramBufferSize of 3GB. While I haven't
noticed the GC issues mark mentioned in this configuration, I have
seen them in the ranges he discusses (on 1.6 <update 18).
You may consider using LuSql[1] to create the indexes, if your source
content is in a JDBC accessible db. It is quite a bit faster than
Solr, as it is a tool specifically created and tuned for Lucene
indexing. But it is command-line, not RESTful like Solr. The released
version of LuSql only runs single machine (though designed for many
threads), the new release will allow distributing indexing across any
number of machines (with each machine building a shard). The new
release also has plugable sources, so it is not restricted to JDBC.
-Glen
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
--
View this message in context: http://old.nabble.com/What-is-largest-reasonable-setting-for-ramBufferSizeMB--tp27631231p27658384.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Mark Miller <ma...@gmail.com>.
>>and a ramBufferSize of 3GB
If you had actually used great than 2GB of it, you would have seen problems
as an int overflowed -
which is why its now hard limited -
if (mb > 2048.0) {
throw new IllegalArgumentException("ramBufferSize " + mb + " is too
large; should be comfortably less than 2048");
}
On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton <gl...@gmail.com> wrote:
> I've run Lucene with heap sizes as large as 28GB of RAM (on a 32GB
> machine, 64bit, Linux) and a ramBufferSize of 3GB. While I haven't
> noticed the GC issues mark mentioned in this configuration, I have
> seen them in the ranges he discusses (on 1.6 <update 18).
>
> You may consider using LuSql[1] to create the indexes, if your source
> content is in a JDBC accessible db. It is quite a bit faster than
> Solr, as it is a tool specifically created and tuned for Lucene
> indexing. But it is command-line, not RESTful like Solr. The released
> version of LuSql only runs single machine (though designed for many
> threads), the new release will allow distributing indexing across any
> number of machines (with each machine building a shard). The new
> release also has plugable sources, so it is not restricted to JDBC.
>
> -Glen
> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>
> On 18 February 2010 21:34, Otis Gospodnetic <ot...@yahoo.com>
> wrote:
> > Hi Tom,
> >
> > It wouldn't. I didn't see the mention of parallel indexing in the
> original email. :)
> >
> > Otis
> > ----
> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > Hadoop ecosystem search :: http://search-hadoop.com/
> >
> >
> >
> > ----- Original Message ----
> >> From: Tom Burton-West <tb...@gmail.com>
> >> To: solr-user@lucene.apache.org
> >> Sent: Thu, February 18, 2010 3:30:05 PM
> >> Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
> >>
> >>
> >> Thanks Otis,
> >>
> >> I don't know enough about Hadoop to understand the advantage of using
> Hadoop
> >> in this use case. How would using Hadoop differ from distributing the
> >> indexing over 10 shards on 10 machines with Solr?
> >>
> >> Tom
> >>
> >>
> >>
> >> Otis Gospodnetic wrote:
> >> >
> >> > Hi Tom,
> >> >
> >> > 32MB is very low, 320MB is medium, and I think you could go higher,
> just
> >> > pick whichever garbage collector is good for throughput. I know Java
> 1.6
> >> > update 18 also has some Hotspot and maybe also GC fixes, so I'd use
> that.
> >> > Finally, this sounds like a good use case for reindexing with Hadoop!
> >> >
> >> > Otis
> >> > ----
> >> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> >> > Hadoop ecosystem search :: http://search-hadoop.com/
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/What-is-largest-reasonable-setting-for-ramBufferSizeMB--tp27631231p27645167.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
>
>
>
> --
>
> -
>
--
- Mark
http://www.lucidimagination.com
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Glen Newton <gl...@gmail.com>.
I've run Lucene with heap sizes as large as 28GB of RAM (on a 32GB
machine, 64bit, Linux) and a ramBufferSize of 3GB. While I haven't
noticed the GC issues mark mentioned in this configuration, I have
seen them in the ranges he discusses (on 1.6 <update 18).
You may consider using LuSql[1] to create the indexes, if your source
content is in a JDBC accessible db. It is quite a bit faster than
Solr, as it is a tool specifically created and tuned for Lucene
indexing. But it is command-line, not RESTful like Solr. The released
version of LuSql only runs single machine (though designed for many
threads), the new release will allow distributing indexing across any
number of machines (with each machine building a shard). The new
release also has plugable sources, so it is not restricted to JDBC.
-Glen
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
On 18 February 2010 21:34, Otis Gospodnetic <ot...@yahoo.com> wrote:
> Hi Tom,
>
> It wouldn't. I didn't see the mention of parallel indexing in the original email. :)
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
>
> ----- Original Message ----
>> From: Tom Burton-West <tb...@gmail.com>
>> To: solr-user@lucene.apache.org
>> Sent: Thu, February 18, 2010 3:30:05 PM
>> Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
>>
>>
>> Thanks Otis,
>>
>> I don't know enough about Hadoop to understand the advantage of using Hadoop
>> in this use case. How would using Hadoop differ from distributing the
>> indexing over 10 shards on 10 machines with Solr?
>>
>> Tom
>>
>>
>>
>> Otis Gospodnetic wrote:
>> >
>> > Hi Tom,
>> >
>> > 32MB is very low, 320MB is medium, and I think you could go higher, just
>> > pick whichever garbage collector is good for throughput. I know Java 1.6
>> > update 18 also has some Hotspot and maybe also GC fixes, so I'd use that.
>> > Finally, this sounds like a good use case for reindexing with Hadoop!
>> >
>> > Otis
>> > ----
>> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> > Hadoop ecosystem search :: http://search-hadoop.com/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/What-is-largest-reasonable-setting-for-ramBufferSizeMB--tp27631231p27645167.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
-
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Tom,
It wouldn't. I didn't see the mention of parallel indexing in the original email. :)
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: Tom Burton-West <tb...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Thu, February 18, 2010 3:30:05 PM
> Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
>
>
> Thanks Otis,
>
> I don't know enough about Hadoop to understand the advantage of using Hadoop
> in this use case. How would using Hadoop differ from distributing the
> indexing over 10 shards on 10 machines with Solr?
>
> Tom
>
>
>
> Otis Gospodnetic wrote:
> >
> > Hi Tom,
> >
> > 32MB is very low, 320MB is medium, and I think you could go higher, just
> > pick whichever garbage collector is good for throughput. I know Java 1.6
> > update 18 also has some Hotspot and maybe also GC fixes, so I'd use that.
> > Finally, this sounds like a good use case for reindexing with Hadoop!
> >
> > Otis
> > ----
> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > Hadoop ecosystem search :: http://search-hadoop.com/
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/What-is-largest-reasonable-setting-for-ramBufferSizeMB--tp27631231p27645167.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Tom Burton-West <tb...@gmail.com>.
Thanks Otis,
I don't know enough about Hadoop to understand the advantage of using Hadoop
in this use case. How would using Hadoop differ from distributing the
indexing over 10 shards on 10 machines with Solr?
Tom
Otis Gospodnetic wrote:
>
> Hi Tom,
>
> 32MB is very low, 320MB is medium, and I think you could go higher, just
> pick whichever garbage collector is good for throughput. I know Java 1.6
> update 18 also has some Hotspot and maybe also GC fixes, so I'd use that.
> Finally, this sounds like a good use case for reindexing with Hadoop!
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
--
View this message in context: http://old.nabble.com/What-is-largest-reasonable-setting-for-ramBufferSizeMB--tp27631231p27645167.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Tom,
32MB is very low, 320MB is medium, and I think you could go higher, just pick whichever garbage collector is good for throughput. I know Java 1.6 update 18 also has some Hotspot and maybe also GC fixes, so I'd use that. Finally, this sounds like a good use case for reindexing with Hadoop!
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: "Burton-West, Tom" <tb...@umich.edu>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Wed, February 17, 2010 5:16:26 PM
> Subject: What is largest reasonable setting for ramBufferSizeMB?
>
> Hello all,
>
> At some point we will need to re-build an index that totals about 2 terrabytes
> in size (split over 10 shards). At our current indexing speed we estimate that
> this will take about 3 weeks. We would like to reduce that time. It appears
> that our main bottleneck is disk I/O.
> We currently have ramBufferSizeMB set to 32 and our merge factor is 10. If we
> increase ramBufferSizeMB to 320, we avoid a merge and the 9 disk writes and
> reads to merge 9+1 32MB segments into a 320MB segment.
>
> Assuming we allocate enough memory to the JVM, would it make sense to increase
> ramBufferSize to 3200MB? What are people's experiences with very large
> ramBufferSizeMB sizes?
>
> Tom Burton-West
> University of Michigan Library
> www.hathitrust.org
Re: What is largest reasonable setting for ramBufferSizeMB?
Posted by Mark Miller <ma...@gmail.com>.
Burton-West, Tom wrote:
> Hello all,
>
> At some point we will need to re-build an index that totals about 2 terrabytes in size (split over 10 shards). At our current indexing speed we estimate that this will take about 3 weeks. We would like to reduce that time. It appears that our main bottleneck is disk I/O.
> We currently have ramBufferSizeMB set to 32 and our merge factor is 10. If we increase ramBufferSizeMB to 320, we avoid a merge and the 9 disk writes and reads to merge 9+1 32MB segments into a 320MB segment.
>
> Assuming we allocate enough memory to the JVM, would it make sense to increase ramBufferSize to 3200MB? What are people's experiences with very large ramBufferSizeMB sizes?
>
> Tom Burton-West
> University of Michigan Library
> www.hathitrust.org
>
>
>
There is a hard limit just under about 2 gigs. Appears to be diminishing
returns as you go over a hundred to a few hundred MB. IE, you prob
picked a good number with 320. If you plan to go big anyway ( > 1 gig ),
you really have to give a lot of RAM to the JVM to avoid some nasty
paging / GC effects. I think someone that tested this had to give over 6
gigabytes to go over 1 gig without these affects? Thats remembering from
memory though. If you look at the gain eked out at that point, its not
really worth it. I'd stick to lower hundreds max.
--
- Mark
http://www.lucidimagination.com