You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Roman Alekseenkov <ra...@gmail.com> on 2011/10/28 20:38:57 UTC
large scale indexing issues / single threaded bottleneck
Hi everyone,
I'm looking for some help with Solr indexing issues on a large scale.
We are indexing few terabytes/month on a sizeable Solr cluster (8
masters / serving writes, 16 slaves / serving reads). After certain
amount of tuning we got to the point where a single Solr instance can
handle index size of 100GB without much issues, but after that we are
starting to observe noticeable delays on index flush and they are
getting larger. See the attached picture for details, it's done for a
single JVM on a single machine.
We are posting data in 8 threads using javabin format and doing commit
every 5K documents, merge factor 20, and ram buffer size about 384MB.
>From the picture it can be seen that a single-threaded index flushing
code kicks in on every commit and blocks all other indexing threads.
The hardware is decent (12 physical / 24 virtual cores per machine)
and it is mostly idle when the index is flushing. Very little CPU
utilization and disk I/O (<5%), with the exception of a single CPU
core which actually does index flush (95% CPU, 5% I/O wait).
My questions are:
1) will Solr changes from real-time branch help to resolve these
issues? I was reading
http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html
and it looks like we have exactly the same problem
2) what would be the best way to port these (and only these) changes
to 3.4.0? I tried to dig into the branching and revisions, but got
lost quickly. Tried something like "svn diff
[…]realtime_search@r953476 […]realtime_search@r1097767", but I'm not
sure if it's even possible to merge these into 3.4.0
3) what would you recommend for production 24/7 use? 3.4.0?
4) is there a workaround that can be used? also, I listed the stack trace below
Thank you!
Roman
P.S. This single "index flushing" thread spends 99% of all the time in
"org.apache.lucene.index.BufferedDeletesStream.applyDeletes", and then
the merge seems to go quickly. I looked it up and it looks like the
intent here is deleting old commit points (we are keeping only 1
non-optimized commit point per config). Not sure why is it taking that
long.
pool-2-thread-1 [RUNNABLE] CPU time: 3:31
java.nio.Bits.copyToByteArray(long, Object, long, long)
java.nio.DirectByteBuffer.get(byte[], int, int)
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(byte[], int, int)
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.TermInfosReader.<init>(Directory, String,
FieldInfos, int, int)
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentReader,
Directory, SegmentInfo, int, int)
org.apache.lucene.index.SegmentReader.get(boolean, Directory,
SegmentInfo, int, boolean, int)
org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo,
boolean, int, int)
org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo, boolean)
org.apache.lucene.index.BufferedDeletesStream.applyDeletes(IndexWriter$ReaderPool,
List)
org.apache.lucene.index.IndexWriter.doFlush(boolean)
org.apache.lucene.index.IndexWriter.flush(boolean, boolean)
org.apache.lucene.index.IndexWriter.closeInternal(boolean)
org.apache.lucene.index.IndexWriter.close(boolean)
org.apache.lucene.index.IndexWriter.close()
org.apache.solr.update.SolrIndexWriter.close()
org.apache.solr.update.DirectUpdateHandler2.closeWriter()
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run()
java.util.concurrent.Executors$RunnableAdapter.call()
java.util.concurrent.FutureTask$Sync.innerRun()
java.util.concurrent.FutureTask.run()
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor$ScheduledFutureTask)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
Re: large scale indexing issues / single threaded bottleneck
Posted by Roman Alekseenkov <ra...@gmail.com>.
I'm wondering if this is relevant:
https://issues.apache.org/jira/browse/LUCENE-2680 - Improve how
IndexWriter flushes deletes against existing segments
Roman
On Fri, Oct 28, 2011 at 11:38 AM, Roman Alekseenkov
<ra...@gmail.com> wrote:
> Hi everyone,
>
> I'm looking for some help with Solr indexing issues on a large scale.
>
> We are indexing few terabytes/month on a sizeable Solr cluster (8
> masters / serving writes, 16 slaves / serving reads). After certain
> amount of tuning we got to the point where a single Solr instance can
> handle index size of 100GB without much issues, but after that we are
> starting to observe noticeable delays on index flush and they are
> getting larger. See the attached picture for details, it's done for a
> single JVM on a single machine.
>
> We are posting data in 8 threads using javabin format and doing commit
> every 5K documents, merge factor 20, and ram buffer size about 384MB.
> From the picture it can be seen that a single-threaded index flushing
> code kicks in on every commit and blocks all other indexing threads.
> The hardware is decent (12 physical / 24 virtual cores per machine)
> and it is mostly idle when the index is flushing. Very little CPU
> utilization and disk I/O (<5%), with the exception of a single CPU
> core which actually does index flush (95% CPU, 5% I/O wait).
>
> My questions are:
>
> 1) will Solr changes from real-time branch help to resolve these
> issues? I was reading
> http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html
> and it looks like we have exactly the same problem
>
> 2) what would be the best way to port these (and only these) changes
> to 3.4.0? I tried to dig into the branching and revisions, but got
> lost quickly. Tried something like "svn diff
> […]realtime_search@r953476 […]realtime_search@r1097767", but I'm not
> sure if it's even possible to merge these into 3.4.0
>
> 3) what would you recommend for production 24/7 use? 3.4.0?
>
> 4) is there a workaround that can be used? also, I listed the stack trace below
>
> Thank you!
> Roman
>
> P.S. This single "index flushing" thread spends 99% of all the time in
> "org.apache.lucene.index.BufferedDeletesStream.applyDeletes", and then
> the merge seems to go quickly. I looked it up and it looks like the
> intent here is deleting old commit points (we are keeping only 1
> non-optimized commit point per config). Not sure why is it taking that
> long.
>
> pool-2-thread-1 [RUNNABLE] CPU time: 3:31
> java.nio.Bits.copyToByteArray(long, Object, long, long)
> java.nio.DirectByteBuffer.get(byte[], int, int)
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(byte[], int, int)
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.TermInfosReader.<init>(Directory, String,
> FieldInfos, int, int)
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentReader,
> Directory, SegmentInfo, int, int)
> org.apache.lucene.index.SegmentReader.get(boolean, Directory,
> SegmentInfo, int, boolean, int)
> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo,
> boolean, int, int)
> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo, boolean)
> org.apache.lucene.index.BufferedDeletesStream.applyDeletes(IndexWriter$ReaderPool,
> List)
> org.apache.lucene.index.IndexWriter.doFlush(boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.closeInternal(boolean)
> org.apache.lucene.index.IndexWriter.close(boolean)
> org.apache.lucene.index.IndexWriter.close()
> org.apache.solr.update.SolrIndexWriter.close()
> org.apache.solr.update.DirectUpdateHandler2.closeWriter()
> org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
> org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run()
> java.util.concurrent.Executors$RunnableAdapter.call()
> java.util.concurrent.FutureTask$Sync.innerRun()
> java.util.concurrent.FutureTask.run()
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor$ScheduledFutureTask)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> java.lang.Thread.run()
>
Re: large scale indexing issues / single threaded bottleneck
Posted by Jason Rutherglen <ja...@gmail.com>.
> abstract away the encoding of the index
Robert, this is what you wrote. "Abstract away the encoding of the
index" means pluggable, otherwise it's not abstract and / or it's a
flawed design. Sounds like it's the latter.
Re: large scale indexing issues / single threaded bottleneck
Posted by Robert Muir <rc...@gmail.com>.
On Fri, Oct 28, 2011 at 8:10 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
>> Otherwise we have "flexible indexing" where "flexible" means "slower
>> if you do anything but the default".
>
> The other encodings should exist as modules since they are pluggable.
> 4.0 can ship with the existing codec. 4.1 with additional codecs and
> the bulk postings at a later time.
you don't know what you are talking about: go look at the source
code. the whole problem is that encodings aren't pluggable.
>
> Otherwise it will be 6 months before 4.0 ships, that's too long.
sucks for you.
>
> Also it is an amusing contradiction that your argument flies in the
> face of Lucid shipping 4.x today without said functionality.
>
No it doesn't. trunk is open source. you can use it, too, if you want.
--
lucidimagination.com
Re: large scale indexing issues / single threaded bottleneck
Posted by Jason Rutherglen <ja...@gmail.com>.
> Otherwise we have "flexible indexing" where "flexible" means "slower
> if you do anything but the default".
The other encodings should exist as modules since they are pluggable.
4.0 can ship with the existing codec. 4.1 with additional codecs and
the bulk postings at a later time.
Otherwise it will be 6 months before 4.0 ships, that's too long.
Also it is an amusing contradiction that your argument flies in the
face of Lucid shipping 4.x today without said functionality.
On Fri, Oct 28, 2011 at 5:09 PM, Robert Muir <rc...@gmail.com> wrote:
> On Fri, Oct 28, 2011 at 5:03 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>
>> +1 I suggested it should be backported a while back. Or that Lucene
>> 4.x should be released. I'm not sure what is holding up Lucene 4.x at
>> this point, bulk postings is only needed useful for PFOR.
>
> This is not true, most modern index compression schemes, not just
> PFOR-delta read more than one integer at a time.
>
> Thats why its important not only to abstract away the encoding of the
> index, but to also ensure that the enumeration apis aren't biased
> towards one-at-a-time vInt.
>
> Otherwise we have "flexible indexing" where "flexible" means "slower
> if you do anything but the default".
>
> --
> lucidimagination.com
>
Re: large scale indexing issues / single threaded bottleneck
Posted by Robert Muir <rc...@gmail.com>.
On Fri, Oct 28, 2011 at 5:03 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> +1 I suggested it should be backported a while back. Or that Lucene
> 4.x should be released. I'm not sure what is holding up Lucene 4.x at
> this point, bulk postings is only needed useful for PFOR.
This is not true, most modern index compression schemes, not just
PFOR-delta read more than one integer at a time.
Thats why its important not only to abstract away the encoding of the
index, but to also ensure that the enumeration apis aren't biased
towards one-at-a-time vInt.
Otherwise we have "flexible indexing" where "flexible" means "slower
if you do anything but the default".
--
lucidimagination.com
Re: large scale indexing issues / single threaded bottleneck
Posted by Jason Rutherglen <ja...@gmail.com>.
> We should maybe try to fix this in 3.x too?
+1 I suggested it should be backported a while back. Or that Lucene
4.x should be released. I'm not sure what is holding up Lucene 4.x at
this point, bulk postings is only needed useful for PFOR.
On Fri, Oct 28, 2011 at 3:27 PM, Simon Willnauer
<si...@googlemail.com> wrote:
> On Fri, Oct 28, 2011 at 9:17 PM, Simon Willnauer
> <si...@googlemail.com> wrote:
>> Hey Roman,
>>
>> On Fri, Oct 28, 2011 at 8:38 PM, Roman Alekseenkov
>> <ra...@gmail.com> wrote:
>>> Hi everyone,
>>>
>>> I'm looking for some help with Solr indexing issues on a large scale.
>>>
>>> We are indexing few terabytes/month on a sizeable Solr cluster (8
>>> masters / serving writes, 16 slaves / serving reads). After certain
>>> amount of tuning we got to the point where a single Solr instance can
>>> handle index size of 100GB without much issues, but after that we are
>>> starting to observe noticeable delays on index flush and they are
>>> getting larger. See the attached picture for details, it's done for a
>>> single JVM on a single machine.
>>>
>>> We are posting data in 8 threads using javabin format and doing commit
>>> every 5K documents, merge factor 20, and ram buffer size about 384MB.
>>> From the picture it can be seen that a single-threaded index flushing
>>> code kicks in on every commit and blocks all other indexing threads.
>>> The hardware is decent (12 physical / 24 virtual cores per machine)
>>> and it is mostly idle when the index is flushing. Very little CPU
>>> utilization and disk I/O (<5%), with the exception of a single CPU
>>> core which actually does index flush (95% CPU, 5% I/O wait).
>>>
>>> My questions are:
>>>
>>> 1) will Solr changes from real-time branch help to resolve these
>>> issues? I was reading
>>> http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html
>>> and it looks like we have exactly the same problem
>>
>> did you also read http://bit.ly/ujLw6v - here I try to explain the
>> major difference between Lucene 3.x and 4.0 and why 3.x has these long
>> idle times. In Lucene 3.x a full flush / commit is a single threaded
>> process, as you observed there is only one thread making progress. In
>> Lucene 4 there is still a single thread executing the commit but other
>> threads are not blocked anymore. Depending on how fast the thread can
>> flush other threads might help flushing segments for that commit
>> concurrently or simply index into new documents writers. So basically
>> 4.0 won't have this problem anymore. The realtime branch you talk
>> about is already merged into 4.0 trunk.
>>
>>>
>>> 2) what would be the best way to port these (and only these) changes
>>> to 3.4.0? I tried to dig into the branching and revisions, but got
>>> lost quickly. Tried something like "svn diff
>>> […]realtime_search@r953476 […]realtime_search@r1097767", but I'm not
>>> sure if it's even possible to merge these into 3.4.0
>>
>> Possible yes! Worth the trouble, I would say no!
>> DocumentsWriterPerThread (DWPT) is a very big change and I don't think
>> we should backport this into our stable branch. However, this feature
>> is very stable in 4.0 though.
>>>
>>> 3) what would you recommend for production 24/7 use? 3.4.0?
>>
>> I think 3.4 is a safe bet! I personally tend to use trunk in
>> production too the only problem is that this is basically a moving
>> target and introduces extra overhead on your side to watch changes and
>> index format modification which could basically prevent you from
>> simple upgrades
>>
>>>
>>> 4) is there a workaround that can be used? also, I listed the stack trace below
>>>
>>> Thank you!
>>> Roman
>>>
>>> P.S. This single "index flushing" thread spends 99% of all the time in
>>> "org.apache.lucene.index.BufferedDeletesStream.applyDeletes", and then
>>> the merge seems to go quickly. I looked it up and it looks like the
>>> intent here is deleting old commit points (we are keeping only 1
>>> non-optimized commit point per config). Not sure why is it taking that
>>> long.
>>
>> in 3.x there is no way to apply deletes without doing a flush (afaik).
>> In 3.x a flush means single threaded again - similar to commit just
>> without syncing files to disk and writing a new segments file. In 4.0
>> you have way more control over this via
>> IndexWriterConfig#setMaxBufferedDeleteTerms which are also applied
>> without blocking other threads. In trunk we hijack indexing threads to
>> do all that work concurrently so you get better cpu utilization and
>> due to concurrent flushing better and usually continuous IO
>> utilization.
>>
>> hope that helps.
>>
>> simon
>>>
>>> pool-2-thread-1 [RUNNABLE] CPU time: 3:31
>>> java.nio.Bits.copyToByteArray(long, Object, long, long)
>>> java.nio.DirectByteBuffer.get(byte[], int, int)
>>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(byte[], int, int)
>>> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>>> org.apache.lucene.index.SegmentTermEnum.next()
>>> org.apache.lucene.index.TermInfosReader.<init>(Directory, String,
>>> FieldInfos, int, int)
>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentReader,
>>> Directory, SegmentInfo, int, int)
>>> org.apache.lucene.index.SegmentReader.get(boolean, Directory,
>>> SegmentInfo, int, boolean, int)
>>> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo,
>>> boolean, int, int)
>>> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo, boolean)
>>> org.apache.lucene.index.BufferedDeletesStream.applyDeletes(IndexWriter$ReaderPool,
>>> List)
>>> org.apache.lucene.index.IndexWriter.doFlush(boolean)
>>> org.apache.lucene.index.IndexWriter.flush(boolean, boolean)
>>> org.apache.lucene.index.IndexWriter.closeInternal(boolean)
>>> org.apache.lucene.index.IndexWriter.close(boolean)
>>> org.apache.lucene.index.IndexWriter.close()
>>> org.apache.solr.update.SolrIndexWriter.close()
>>> org.apache.solr.update.DirectUpdateHandler2.closeWriter()
>>> org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
>>> org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run()
>>> java.util.concurrent.Executors$RunnableAdapter.call()
>>> java.util.concurrent.FutureTask$Sync.innerRun()
>>> java.util.concurrent.FutureTask.run()
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor$ScheduledFutureTask)
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run()
>>> java.lang.Thread.run()
>
> one more thing, after somebody (thanks robert) pointed me at the
> stacktrace it seems kind of obvious what the root cause of your
> problem is. Its solr :) Solr closes the IndexWriter on commit which is
> very wasteful since you basically wait until all merges are done. Solr
> trunk has solved this problem.
>
> We should maybe try to fix this in 3.x too?
>
> simon
>>>
>>
>
Re: large scale indexing issues / single threaded bottleneck
Posted by Kiril Menshikov <km...@gmail.com>.
Yonik,
Adding overwrite=false don't help. XMLLoader don't check this HTTP
parameter. Instead it check attribute in <add> XML tag, with the same name.
-Kiril
--
View this message in context: http://lucene.472066.n3.nabble.com/large-scale-indexing-issues-single-threaded-bottleneck-tp3461815p3468463.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: large scale indexing issues / single threaded bottleneck
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Oct 29, 2011 at 6:35 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> I saw a mention somewhere that you can tell Solr not to use
> IW.addDocument (not IW.updateDocument) when you add a document if you
> are certain it's not replacing a previous document with the same ID
Right - adding overwrite=false to the URL when updating should do this.
-Yonik
http://www.lucidimagination.com
Re: large scale indexing issues / single threaded bottleneck
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Oct 28, 2011 at 3:27 PM, Simon Willnauer
<si...@googlemail.com> wrote:
> one more thing, after somebody (thanks robert) pointed me at the
> stacktrace it seems kind of obvious what the root cause of your
> problem is. Its solr :) Solr closes the IndexWriter on commit which is
> very wasteful since you basically wait until all merges are done. Solr
> trunk has solved this problem.
That is very wasteful but I don't think it's actually the cause of the
slowdown here?
The cause looks like it's in applying deletes, which even once Solr
stops closing the IW will still occur (ie, IW.commit must also resolve
all deletes).
When IW resolves deletes it 1) opens a SegmentReader for each segment
in the index, and 2) looks up each deleted term and mark its
document(s) as deleted.
I saw a mention somewhere that you can tell Solr not to use
IW.addDocument (not IW.updateDocument) when you add a document if you
are certain it's not replacing a previous document with the same ID --
I don't know how to do that but if that's true, and you are truly only
adding documents, that could be the easiest fix here.
Failing that... you could try increasing
IndexWriterConfig.setReaderTermsIndexDivisor (not sure if/how this is
exposed in Solr's config)... this will make init time and RAM usage
for each SegmentReader faster, but lookup time slower; whether this
helps depends on if your slowness is in opening the SegmentReader (how
long does it take to IR.open on your index?) or on resolving the
deletes once SR is open.
Do you have a great many terms in your index? Can you run CheckIndex
and post the output? (If so this might mean you have an analysis
problem, ie, putting too many terms in the index).
> We should maybe try to fix this in 3.x too?
+1; having to wait for running merges to complete when the app calls
commit is crazy (Lucene long ago removed that limitation).
Mike McCandless
http://blog.mikemccandless.com
Re: large scale indexing issues / single threaded bottleneck
Posted by Simon Willnauer <si...@googlemail.com>.
On Fri, Oct 28, 2011 at 9:17 PM, Simon Willnauer
<si...@googlemail.com> wrote:
> Hey Roman,
>
> On Fri, Oct 28, 2011 at 8:38 PM, Roman Alekseenkov
> <ra...@gmail.com> wrote:
>> Hi everyone,
>>
>> I'm looking for some help with Solr indexing issues on a large scale.
>>
>> We are indexing few terabytes/month on a sizeable Solr cluster (8
>> masters / serving writes, 16 slaves / serving reads). After certain
>> amount of tuning we got to the point where a single Solr instance can
>> handle index size of 100GB without much issues, but after that we are
>> starting to observe noticeable delays on index flush and they are
>> getting larger. See the attached picture for details, it's done for a
>> single JVM on a single machine.
>>
>> We are posting data in 8 threads using javabin format and doing commit
>> every 5K documents, merge factor 20, and ram buffer size about 384MB.
>> From the picture it can be seen that a single-threaded index flushing
>> code kicks in on every commit and blocks all other indexing threads.
>> The hardware is decent (12 physical / 24 virtual cores per machine)
>> and it is mostly idle when the index is flushing. Very little CPU
>> utilization and disk I/O (<5%), with the exception of a single CPU
>> core which actually does index flush (95% CPU, 5% I/O wait).
>>
>> My questions are:
>>
>> 1) will Solr changes from real-time branch help to resolve these
>> issues? I was reading
>> http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html
>> and it looks like we have exactly the same problem
>
> did you also read http://bit.ly/ujLw6v - here I try to explain the
> major difference between Lucene 3.x and 4.0 and why 3.x has these long
> idle times. In Lucene 3.x a full flush / commit is a single threaded
> process, as you observed there is only one thread making progress. In
> Lucene 4 there is still a single thread executing the commit but other
> threads are not blocked anymore. Depending on how fast the thread can
> flush other threads might help flushing segments for that commit
> concurrently or simply index into new documents writers. So basically
> 4.0 won't have this problem anymore. The realtime branch you talk
> about is already merged into 4.0 trunk.
>
>>
>> 2) what would be the best way to port these (and only these) changes
>> to 3.4.0? I tried to dig into the branching and revisions, but got
>> lost quickly. Tried something like "svn diff
>> […]realtime_search@r953476 […]realtime_search@r1097767", but I'm not
>> sure if it's even possible to merge these into 3.4.0
>
> Possible yes! Worth the trouble, I would say no!
> DocumentsWriterPerThread (DWPT) is a very big change and I don't think
> we should backport this into our stable branch. However, this feature
> is very stable in 4.0 though.
>>
>> 3) what would you recommend for production 24/7 use? 3.4.0?
>
> I think 3.4 is a safe bet! I personally tend to use trunk in
> production too the only problem is that this is basically a moving
> target and introduces extra overhead on your side to watch changes and
> index format modification which could basically prevent you from
> simple upgrades
>
>>
>> 4) is there a workaround that can be used? also, I listed the stack trace below
>>
>> Thank you!
>> Roman
>>
>> P.S. This single "index flushing" thread spends 99% of all the time in
>> "org.apache.lucene.index.BufferedDeletesStream.applyDeletes", and then
>> the merge seems to go quickly. I looked it up and it looks like the
>> intent here is deleting old commit points (we are keeping only 1
>> non-optimized commit point per config). Not sure why is it taking that
>> long.
>
> in 3.x there is no way to apply deletes without doing a flush (afaik).
> In 3.x a flush means single threaded again - similar to commit just
> without syncing files to disk and writing a new segments file. In 4.0
> you have way more control over this via
> IndexWriterConfig#setMaxBufferedDeleteTerms which are also applied
> without blocking other threads. In trunk we hijack indexing threads to
> do all that work concurrently so you get better cpu utilization and
> due to concurrent flushing better and usually continuous IO
> utilization.
>
> hope that helps.
>
> simon
>>
>> pool-2-thread-1 [RUNNABLE] CPU time: 3:31
>> java.nio.Bits.copyToByteArray(long, Object, long, long)
>> java.nio.DirectByteBuffer.get(byte[], int, int)
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(byte[], int, int)
>> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>> org.apache.lucene.index.SegmentTermEnum.next()
>> org.apache.lucene.index.TermInfosReader.<init>(Directory, String,
>> FieldInfos, int, int)
>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentReader,
>> Directory, SegmentInfo, int, int)
>> org.apache.lucene.index.SegmentReader.get(boolean, Directory,
>> SegmentInfo, int, boolean, int)
>> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo,
>> boolean, int, int)
>> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo, boolean)
>> org.apache.lucene.index.BufferedDeletesStream.applyDeletes(IndexWriter$ReaderPool,
>> List)
>> org.apache.lucene.index.IndexWriter.doFlush(boolean)
>> org.apache.lucene.index.IndexWriter.flush(boolean, boolean)
>> org.apache.lucene.index.IndexWriter.closeInternal(boolean)
>> org.apache.lucene.index.IndexWriter.close(boolean)
>> org.apache.lucene.index.IndexWriter.close()
>> org.apache.solr.update.SolrIndexWriter.close()
>> org.apache.solr.update.DirectUpdateHandler2.closeWriter()
>> org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
>> org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run()
>> java.util.concurrent.Executors$RunnableAdapter.call()
>> java.util.concurrent.FutureTask$Sync.innerRun()
>> java.util.concurrent.FutureTask.run()
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor$ScheduledFutureTask)
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run()
>> java.lang.Thread.run()
one more thing, after somebody (thanks robert) pointed me at the
stacktrace it seems kind of obvious what the root cause of your
problem is. Its solr :) Solr closes the IndexWriter on commit which is
very wasteful since you basically wait until all merges are done. Solr
trunk has solved this problem.
We should maybe try to fix this in 3.x too?
simon
>>
>
Re: large scale indexing issues / single threaded bottleneck
Posted by Simon Willnauer <si...@googlemail.com>.
Hey Roman,
On Fri, Oct 28, 2011 at 8:38 PM, Roman Alekseenkov
<ra...@gmail.com> wrote:
> Hi everyone,
>
> I'm looking for some help with Solr indexing issues on a large scale.
>
> We are indexing few terabytes/month on a sizeable Solr cluster (8
> masters / serving writes, 16 slaves / serving reads). After certain
> amount of tuning we got to the point where a single Solr instance can
> handle index size of 100GB without much issues, but after that we are
> starting to observe noticeable delays on index flush and they are
> getting larger. See the attached picture for details, it's done for a
> single JVM on a single machine.
>
> We are posting data in 8 threads using javabin format and doing commit
> every 5K documents, merge factor 20, and ram buffer size about 384MB.
> From the picture it can be seen that a single-threaded index flushing
> code kicks in on every commit and blocks all other indexing threads.
> The hardware is decent (12 physical / 24 virtual cores per machine)
> and it is mostly idle when the index is flushing. Very little CPU
> utilization and disk I/O (<5%), with the exception of a single CPU
> core which actually does index flush (95% CPU, 5% I/O wait).
>
> My questions are:
>
> 1) will Solr changes from real-time branch help to resolve these
> issues? I was reading
> http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html
> and it looks like we have exactly the same problem
did you also read http://bit.ly/ujLw6v - here I try to explain the
major difference between Lucene 3.x and 4.0 and why 3.x has these long
idle times. In Lucene 3.x a full flush / commit is a single threaded
process, as you observed there is only one thread making progress. In
Lucene 4 there is still a single thread executing the commit but other
threads are not blocked anymore. Depending on how fast the thread can
flush other threads might help flushing segments for that commit
concurrently or simply index into new documents writers. So basically
4.0 won't have this problem anymore. The realtime branch you talk
about is already merged into 4.0 trunk.
>
> 2) what would be the best way to port these (and only these) changes
> to 3.4.0? I tried to dig into the branching and revisions, but got
> lost quickly. Tried something like "svn diff
> […]realtime_search@r953476 […]realtime_search@r1097767", but I'm not
> sure if it's even possible to merge these into 3.4.0
Possible yes! Worth the trouble, I would say no!
DocumentsWriterPerThread (DWPT) is a very big change and I don't think
we should backport this into our stable branch. However, this feature
is very stable in 4.0 though.
>
> 3) what would you recommend for production 24/7 use? 3.4.0?
I think 3.4 is a safe bet! I personally tend to use trunk in
production too the only problem is that this is basically a moving
target and introduces extra overhead on your side to watch changes and
index format modification which could basically prevent you from
simple upgrades
>
> 4) is there a workaround that can be used? also, I listed the stack trace below
>
> Thank you!
> Roman
>
> P.S. This single "index flushing" thread spends 99% of all the time in
> "org.apache.lucene.index.BufferedDeletesStream.applyDeletes", and then
> the merge seems to go quickly. I looked it up and it looks like the
> intent here is deleting old commit points (we are keeping only 1
> non-optimized commit point per config). Not sure why is it taking that
> long.
in 3.x there is no way to apply deletes without doing a flush (afaik).
In 3.x a flush means single threaded again - similar to commit just
without syncing files to disk and writing a new segments file. In 4.0
you have way more control over this via
IndexWriterConfig#setMaxBufferedDeleteTerms which are also applied
without blocking other threads. In trunk we hijack indexing threads to
do all that work concurrently so you get better cpu utilization and
due to concurrent flushing better and usually continuous IO
utilization.
hope that helps.
simon
>
> pool-2-thread-1 [RUNNABLE] CPU time: 3:31
> java.nio.Bits.copyToByteArray(long, Object, long, long)
> java.nio.DirectByteBuffer.get(byte[], int, int)
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(byte[], int, int)
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.TermInfosReader.<init>(Directory, String,
> FieldInfos, int, int)
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentReader,
> Directory, SegmentInfo, int, int)
> org.apache.lucene.index.SegmentReader.get(boolean, Directory,
> SegmentInfo, int, boolean, int)
> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo,
> boolean, int, int)
> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo, boolean)
> org.apache.lucene.index.BufferedDeletesStream.applyDeletes(IndexWriter$ReaderPool,
> List)
> org.apache.lucene.index.IndexWriter.doFlush(boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.closeInternal(boolean)
> org.apache.lucene.index.IndexWriter.close(boolean)
> org.apache.lucene.index.IndexWriter.close()
> org.apache.solr.update.SolrIndexWriter.close()
> org.apache.solr.update.DirectUpdateHandler2.closeWriter()
> org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
> org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run()
> java.util.concurrent.Executors$RunnableAdapter.call()
> java.util.concurrent.FutureTask$Sync.innerRun()
> java.util.concurrent.FutureTask.run()
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor$ScheduledFutureTask)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> java.lang.Thread.run()
>
RE: large scale indexing issues / single threaded bottleneck
Posted by Roman Alekseenkov <ra...@gmail.com>.
We have a rate of 2K small docs/sec which translates into 90 GB/day of index
space
You should be fine
Roman
Awasthi, Shishir wrote:
>
> Roman,
> How frequently do you update your index? I have a need to do real time
> add/delete to SOLR documents at a rate of approximately 20/min.
> The total number of documents are in the range of 4 million. Will there
> be any performance issues?
>
> Thanks,
> Shishir
>
--
View this message in context: http://lucene.472066.n3.nabble.com/large-scale-indexing-issues-single-threaded-bottleneck-tp3461815p3472901.html
Sent from the Solr - User mailing list archive at Nabble.com.
AW: large scale indexing issues / single threaded bottleneck
Posted by se...@immonet.de.
Hi,
we are currently thinking about the performance facts too.
I wonder if there are any sites on the net describing what a large index is?
People always talk about huge indexes and heavy commits etc. but i can't find some stats about it in numbers and no information about the hardware used.
Maybe an article in the wiki would help.
I expect our index to be about 4 to 5 gig with 500.000 docs and 80.000 commits a day. Is that considered to be large, medium or small?
Greets
Sebastian
-----Ursprüngliche Nachricht-----
Von: Jaeger, Jay - DOT [mailto:Jay.Jaeger@dot.wi.gov]
Gesendet: Donnerstag, 3. November 2011 14:00
An: 'solr-user@lucene.apache.org'
Betreff: RE: large scale indexing issues / single threaded bottleneck
Shishir, we have 35 million "documents", and should be doing about 5000-10000 new "documents" a day, but with very small "documents": 40 fields which have at most a few terms, with many being single terms.
You may occasionally see some impact from top level index merges but those should be very infrequent given your stated volumes.
For more concrete advice, you should also provide information on the size of your documents, and your search volume.
JRJ
-----Original Message-----
From: Awasthi, Shishir [mailto:shishir.awasthi@baml.com]
Sent: Tuesday, November 01, 2011 10:58 PM
To: solr-user@lucene.apache.org
Subject: RE: large scale indexing issues / single threaded bottleneck
Roman,
How frequently do you update your index? I have a need to do real time add/delete to SOLR documents at a rate of approximately 20/min.
The total number of documents are in the range of 4 million. Will there be any performance issues?
Thanks,
Shishir
-----Original Message-----
From: Roman Alekseenkov [mailto:ralekseenkov@gmail.com]
Sent: Sunday, October 30, 2011 6:11 PM
To: solr-user@lucene.apache.org
Subject: Re: large scale indexing issues / single threaded bottleneck
Guys, thank you for all the replies.
I think I have figured out a partial solution for the problem on Friday night. Adding a whole bunch of debug statements to the info stream showed that every document is following "update document" path instead of "add document" path. Meaning that all document IDs are getting into the "pending deletes" queue, and Solr has to rescan its index on every commit for potential deletions. This is single threaded and seems to get progressively slower with the index size.
Adding overwrite=false to the URL in /update handler did NOT help, as my debug statements showed that messages still go to updateDocument() function with deleteTerm not being null. So, I hacked Lucene a little bit and set deleteTerm=null as a temporary solution in the beginning of updateDocument(), and it does not call applyDeletes() anymore.
This gave a 6-8x performance boost, and now we can index about 9 million documents/hour (producing 20Gb of index every hour). Right now it's at 1TB index size and going, without noticeable degradation of the indexing speed.
This is decent, but still the 24-core machine is barely utilized :)
Now I think it's hitting a merge bottleneck, where all indexing threads are being paused. And ConcurrentMergeScheduler with 4 threads is not helping much. I guess the changes on the trunk would definitely help, but we will likely stay on 3.4
Will dig more into the issue on Monday. Really curious to see why "overwrite=false" didn't help, but the hack did.
Once again, thank you for the answers and recommendations
Roman
--
View this message in context:
http://lucene.472066.n3.nabble.com/large-scale-indexing-issues-single-th
readed-bottleneck-tp3461815p3466523.html
Sent from the Solr - User mailing list archive at Nabble.com.
----------------------------------------------------------------------
This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited.
Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law.
The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses.
References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link:
http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.
RE: large scale indexing issues / single threaded bottleneck
Posted by "Jaeger, Jay - DOT" <Ja...@dot.wi.gov>.
Shishir, we have 35 million "documents", and should be doing about 5000-10000 new "documents" a day, but with very small "documents": 40 fields which have at most a few terms, with many being single terms.
You may occasionally see some impact from top level index merges but those should be very infrequent given your stated volumes.
For more concrete advice, you should also provide information on the size of your documents, and your search volume.
JRJ
-----Original Message-----
From: Awasthi, Shishir [mailto:shishir.awasthi@baml.com]
Sent: Tuesday, November 01, 2011 10:58 PM
To: solr-user@lucene.apache.org
Subject: RE: large scale indexing issues / single threaded bottleneck
Roman,
How frequently do you update your index? I have a need to do real time
add/delete to SOLR documents at a rate of approximately 20/min.
The total number of documents are in the range of 4 million. Will there
be any performance issues?
Thanks,
Shishir
-----Original Message-----
From: Roman Alekseenkov [mailto:ralekseenkov@gmail.com]
Sent: Sunday, October 30, 2011 6:11 PM
To: solr-user@lucene.apache.org
Subject: Re: large scale indexing issues / single threaded bottleneck
Guys, thank you for all the replies.
I think I have figured out a partial solution for the problem on Friday
night. Adding a whole bunch of debug statements to the info stream
showed that every document is following "update document" path instead
of "add document" path. Meaning that all document IDs are getting into
the "pending deletes" queue, and Solr has to rescan its index on every
commit for potential deletions. This is single threaded and seems to get
progressively slower with the index size.
Adding overwrite=false to the URL in /update handler did NOT help, as my
debug statements showed that messages still go to updateDocument()
function with deleteTerm not being null. So, I hacked Lucene a little
bit and set deleteTerm=null as a temporary solution in the beginning of
updateDocument(), and it does not call applyDeletes() anymore.
This gave a 6-8x performance boost, and now we can index about 9 million
documents/hour (producing 20Gb of index every hour). Right now it's at
1TB index size and going, without noticeable degradation of the indexing
speed.
This is decent, but still the 24-core machine is barely utilized :)
Now I think it's hitting a merge bottleneck, where all indexing threads
are being paused. And ConcurrentMergeScheduler with 4 threads is not
helping much. I guess the changes on the trunk would definitely help,
but we will likely stay on 3.4
Will dig more into the issue on Monday. Really curious to see why
"overwrite=false" didn't help, but the hack did.
Once again, thank you for the answers and recommendations
Roman
--
View this message in context:
http://lucene.472066.n3.nabble.com/large-scale-indexing-issues-single-th
readed-bottleneck-tp3461815p3466523.html
Sent from the Solr - User mailing list archive at Nabble.com.
----------------------------------------------------------------------
This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited.
Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law.
The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses.
References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link:
http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.
RE: large scale indexing issues / single threaded bottleneck
Posted by "Awasthi, Shishir" <sh...@baml.com>.
Roman,
How frequently do you update your index? I have a need to do real time
add/delete to SOLR documents at a rate of approximately 20/min.
The total number of documents are in the range of 4 million. Will there
be any performance issues?
Thanks,
Shishir
-----Original Message-----
From: Roman Alekseenkov [mailto:ralekseenkov@gmail.com]
Sent: Sunday, October 30, 2011 6:11 PM
To: solr-user@lucene.apache.org
Subject: Re: large scale indexing issues / single threaded bottleneck
Guys, thank you for all the replies.
I think I have figured out a partial solution for the problem on Friday
night. Adding a whole bunch of debug statements to the info stream
showed that every document is following "update document" path instead
of "add document" path. Meaning that all document IDs are getting into
the "pending deletes" queue, and Solr has to rescan its index on every
commit for potential deletions. This is single threaded and seems to get
progressively slower with the index size.
Adding overwrite=false to the URL in /update handler did NOT help, as my
debug statements showed that messages still go to updateDocument()
function with deleteTerm not being null. So, I hacked Lucene a little
bit and set deleteTerm=null as a temporary solution in the beginning of
updateDocument(), and it does not call applyDeletes() anymore.
This gave a 6-8x performance boost, and now we can index about 9 million
documents/hour (producing 20Gb of index every hour). Right now it's at
1TB index size and going, without noticeable degradation of the indexing
speed.
This is decent, but still the 24-core machine is barely utilized :)
Now I think it's hitting a merge bottleneck, where all indexing threads
are being paused. And ConcurrentMergeScheduler with 4 threads is not
helping much. I guess the changes on the trunk would definitely help,
but we will likely stay on 3.4
Will dig more into the issue on Monday. Really curious to see why
"overwrite=false" didn't help, but the hack did.
Once again, thank you for the answers and recommendations
Roman
--
View this message in context:
http://lucene.472066.n3.nabble.com/large-scale-indexing-issues-single-th
readed-bottleneck-tp3461815p3466523.html
Sent from the Solr - User mailing list archive at Nabble.com.
----------------------------------------------------------------------
This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited.
Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law.
The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses.
References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link:
http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.
Re: large scale indexing issues / single threaded bottleneck
Posted by Roman Alekseenkov <ra...@gmail.com>.
Guys, thank you for all the replies.
I think I have figured out a partial solution for the problem on Friday
night. Adding a whole bunch of debug statements to the info stream showed
that every document is following "update document" path instead of "add
document" path. Meaning that all document IDs are getting into the "pending
deletes" queue, and Solr has to rescan its index on every commit for
potential deletions. This is single threaded and seems to get progressively
slower with the index size.
Adding overwrite=false to the URL in /update handler did NOT help, as my
debug statements showed that messages still go to updateDocument() function
with deleteTerm not being null. So, I hacked Lucene a little bit and set
deleteTerm=null as a temporary solution in the beginning of
updateDocument(), and it does not call applyDeletes() anymore.
This gave a 6-8x performance boost, and now we can index about 9 million
documents/hour (producing 20Gb of index every hour). Right now it's at 1TB
index size and going, without noticeable degradation of the indexing speed.
This is decent, but still the 24-core machine is barely utilized :)
Now I think it's hitting a merge bottleneck, where all indexing threads are
being paused. And ConcurrentMergeScheduler with 4 threads is not helping
much. I guess the changes on the trunk would definitely help, but we will
likely stay on 3.4
Will dig more into the issue on Monday. Really curious to see why
"overwrite=false" didn't help, but the hack did.
Once again, thank you for the answers and recommendations
Roman
--
View this message in context: http://lucene.472066.n3.nabble.com/large-scale-indexing-issues-single-threaded-bottleneck-tp3461815p3466523.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: large scale indexing issues / single threaded bottleneck
Posted by Nagendra Nagarajayya <nn...@transaxtions.com>.
Roman:
>2) what would be the best way to port these (and only these) changes
to 3.4.0? I tried to dig into the branching and revisions, but got
lost quickly. Tried something like "svn diff
[…]realtime_search@r953476 […]realtime_search@r1097767", but I'm not
sure if it's even possible to merge these into 3.4.0
>3) what would you recommend for production 24/7 use? 3.4.0?
If you want try real time indexing without commits with ver 3.4.0, you
can give Solr with RankingAlgorithm a try. It does not need commits
during to add documents ( you can set your commits to every 15 mins or
as desired ).
You can get more information about NRT with 3.4.0 from here:
http://solr-ra.tgels.org/wiki/en/Near_Real_Time_Search_ver_3.x
You can download Solr 3.4.0 with RankingAlgorithm 1.3 from here:
http://solr-ra.tgels.org
Regards,
- Nagendra Nagarajayya
http://solr-ra.tgels.org
http://rankingalgorithm.tgels.org
On 10/28/2011 11:38 AM, Roman Alekseenkov wrote:
> Hi everyone,
>
> I'm looking for some help with Solr indexing issues on a large scale.
>
> We are indexing few terabytes/month on a sizeable Solr cluster (8
> masters / serving writes, 16 slaves / serving reads). After certain
> amount of tuning we got to the point where a single Solr instance can
> handle index size of 100GB without much issues, but after that we are
> starting to observe noticeable delays on index flush and they are
> getting larger. See the attached picture for details, it's done for a
> single JVM on a single machine.
>
> We are posting data in 8 threads using javabin format and doing commit
> every 5K documents, merge factor 20, and ram buffer size about 384MB.
> > From the picture it can be seen that a single-threaded index flushing
> code kicks in on every commit and blocks all other indexing threads.
> The hardware is decent (12 physical / 24 virtual cores per machine)
> and it is mostly idle when the index is flushing. Very little CPU
> utilization and disk I/O (<5%), with the exception of a single CPU
> core which actually does index flush (95% CPU, 5% I/O wait).
>
> My questions are:
>
> 1) will Solr changes from real-time branch help to resolve these
> issues? I was reading
> http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html
> and it looks like we have exactly the same problem
>
> 2) what would be the best way to port these (and only these) changes
> to 3.4.0? I tried to dig into the branching and revisions, but got
> lost quickly. Tried something like "svn diff
> […]realtime_search@r953476 […]realtime_search@r1097767", but I'm not
> sure if it's even possible to merge these into 3.4.0
>
> 3) what would you recommend for production 24/7 use? 3.4.0?
>
> 4) is there a workaround that can be used? also, I listed the stack trace below
>
> Thank you!
> Roman
>
> P.S. This single "index flushing" thread spends 99% of all the time in
> "org.apache.lucene.index.BufferedDeletesStream.applyDeletes", and then
> the merge seems to go quickly. I looked it up and it looks like the
> intent here is deleting old commit points (we are keeping only 1
> non-optimized commit point per config). Not sure why is it taking that
> long.
>
> pool-2-thread-1 [RUNNABLE] CPU time: 3:31
> java.nio.Bits.copyToByteArray(long, Object, long, long)
> java.nio.DirectByteBuffer.get(byte[], int, int)
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(byte[], int, int)
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.TermInfosReader.<init>(Directory, String,
> FieldInfos, int, int)
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentReader,
> Directory, SegmentInfo, int, int)
> org.apache.lucene.index.SegmentReader.get(boolean, Directory,
> SegmentInfo, int, boolean, int)
> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo,
> boolean, int, int)
> org.apache.lucene.index.IndexWriter$ReaderPool.get(SegmentInfo, boolean)
> org.apache.lucene.index.BufferedDeletesStream.applyDeletes(IndexWriter$ReaderPool,
> List)
> org.apache.lucene.index.IndexWriter.doFlush(boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.closeInternal(boolean)
> org.apache.lucene.index.IndexWriter.close(boolean)
> org.apache.lucene.index.IndexWriter.close()
> org.apache.solr.update.SolrIndexWriter.close()
> org.apache.solr.update.DirectUpdateHandler2.closeWriter()
> org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
> org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run()
> java.util.concurrent.Executors$RunnableAdapter.call()
> java.util.concurrent.FutureTask$Sync.innerRun()
> java.util.concurrent.FutureTask.run()
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor$ScheduledFutureTask)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> java.lang.Thread.run()