You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by ga...@graffiti.net on 2010/08/23 09:42:12 UTC
slow search threads during a disk copy
Hi all,
We're observing search threads slowing down during directory copies performed during updates to the index. The thread dump shows search threads blocked on a FSDirectory$FSIndexInput$Descriptor instance:
"Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for monitor entry [0x988ed000..0x988edf30]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
- waiting to lock <0xc79c9900> (a org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
at org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
at org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
at org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.Searcher.search(Searcher.java:118)
...
The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior to applying changes (addDocument calls) on the current "available" segment. This takes >7 mins to copy a directory of size 1.4G. During this time window, the searches are slow and the above thread stacks are observed.
Could there be any system level limits we're hitting?
Our test environment is:
lucene-2.3.2
4x2.6 GHz, 16G memory
Red Hat 3.4.6-9
Thanks and regards,
- Gagan
Re: slow search threads during a disk copy
Posted by Anshum <an...@gmail.com>.
There is bound to be IO contention. I'm sure iostat will give you a much
better picture on it.
--
Anshum Gupta
http://ai-cafe.blogspot.com
On Mon, Aug 23, 2010 at 3:13 PM, <ga...@graffiti.net> wrote:
> Yes, all version directories are on the same disk. iostat output should be
> useful. Using rsync is complicated because of legacy reasons - there's a
> change impact to manage.
>
>
> Intererstingly, the copy is quite fast (around 30s) when there are no
> searches in progress.
>
>
>
> Thanks,
> - Gagan
>
>
> -----Original Message-----
> From: Ian Lea <ia...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Mon, Aug 23, 2010 2:10 pm
> Subject: Re: slow search threads during a disk copy
>
>
> Well, there still sounds like plenty of scope for IO contention. Are
> source and dest directories on the same disk? What does iostat show?
> Could you use rsync rather than cp -rp, or something else that only
> copies changes?
>
>
> --
> Ian.
>
>
> On Mon, Aug 23, 2010 at 9:20 AM, <ga...@graffiti.net> wrote:
> >
> > We suspected the same at first, and so modified the flow to do the copy
> of the
> new version "before" searching is activated on it. So, there are no
> searches on
> the new version dir at the time of copy - yet we still see slowness during
> the
> disk copy.
> >
> >
> > Thanks,
> > - Gagan
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Anshum <an...@gmail.com>
> > To: java-user@lucene.apache.org
> > Sent: Mon, Aug 23, 2010 1:17 pm
> > Subject: Re: slow search threads during a disk copy
> >
> >
> > Seems like a case of I/O issues. You may be reading content off the index
> > while performing searches while the I/O for copy is also happening.
> >
> > --
> > Anshum Gupta
> > http://ai-cafe.blogspot.com
> >
> >
> > On Mon, Aug 23, 2010 at 1:12 PM, <ga...@graffiti.net> wrote:
> >
> >>
> >> Hi all,
> >>
> >>
> >> We're observing search threads slowing down during directory copies
> >> performed during updates to the index. The thread dump shows search
> threads
> >> blocked on a FSDirectory$FSIndexInput$Descriptor instance:
> >>
> >>
> >>
> >> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting
> for
> >> monitor entry [0x988ed000..0x988edf30]
> >> java.lang.Thread.State: BLOCKED (on object monitor)
> >> at
> >>
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
> >> - waiting to lock <0xc79c9900> (a
> >> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
> >> at
> >>
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
> >> at
> >>
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> >> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
> >> at
> >> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
> >> at
> >>
> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
> >> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
> >> at
> >>
> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
> >> at
> >>
> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
> >> at
> >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> >> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> >> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
> >>
> >> ...
> >>
> >>
> >> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command
> prior
> >> to applying changes (addDocument calls) on the current "available"
> segment.
> >> This takes >7 mins to copy a directory of size 1.4G. During this time
> >> window, the searches are slow and the above thread stacks are observed.
> >>
> >>
> >> Could there be any system level limits we're hitting?
> >>
> >>
> >> Our test environment is:
> >> lucene-2.3.2
> >> 4x2.6 GHz, 16G memory
> >> Red Hat 3.4.6-9
> >>
> >>
> >> Thanks and regards,
> >>
> >>
> >> - Gagan
> >>
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
Re: slow search threads during a disk copy
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2010-08-23 at 11:43 +0200, gaganb@graffiti.net wrote:
> Intererstingly, the copy is quite fast (around 30s) when there are no searches in progress.
I agree with Anshum: This looks very much like IO contention.
However, it might not just be a case of seek-time trouble: We've had
similar problems on our Linux server with when copying large files from
harddisk A to A while performing searches on SSD B.
Our low-level guys tells me that this was due to the write-buffer being
filled and that reads were blocked while it was flushed (AFAIR). We've
experimented by mounting with sync instead of async which gave us much
better response times during copying at the cost of a substantially
slower copy. dirsync should also be worth looking into.
Regards,
Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: slow search threads during a disk copy
Posted by ga...@graffiti.net.
Yes, all version directories are on the same disk. iostat output should be useful. Using rsync is complicated because of legacy reasons - there's a change impact to manage.
Intererstingly, the copy is quite fast (around 30s) when there are no searches in progress.
Thanks,
- Gagan
-----Original Message-----
From: Ian Lea <ia...@gmail.com>
To: java-user@lucene.apache.org
Sent: Mon, Aug 23, 2010 2:10 pm
Subject: Re: slow search threads during a disk copy
Well, there still sounds like plenty of scope for IO contention. Are
source and dest directories on the same disk? What does iostat show?
Could you use rsync rather than cp -rp, or something else that only
copies changes?
--
Ian.
On Mon, Aug 23, 2010 at 9:20 AM, <ga...@graffiti.net> wrote:
>
> We suspected the same at first, and so modified the flow to do the copy of the
new version "before" searching is activated on it. So, there are no searches on
the new version dir at the time of copy - yet we still see slowness during the
disk copy.
>
>
> Thanks,
> - Gagan
>
>
>
>
>
> -----Original Message-----
> From: Anshum <an...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Mon, Aug 23, 2010 1:17 pm
> Subject: Re: slow search threads during a disk copy
>
>
> Seems like a case of I/O issues. You may be reading content off the index
> while performing searches while the I/O for copy is also happening.
>
> --
> Anshum Gupta
> http://ai-cafe.blogspot.com
>
>
> On Mon, Aug 23, 2010 at 1:12 PM, <ga...@graffiti.net> wrote:
>
>>
>> Hi all,
>>
>>
>> We're observing search threads slowing down during directory copies
>> performed during updates to the index. The thread dump shows search threads
>> blocked on a FSDirectory$FSIndexInput$Descriptor instance:
>>
>>
>>
>> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
>> monitor entry [0x988ed000..0x988edf30]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
>> - waiting to lock <0xc79c9900> (a
>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
>> at
>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
>> at
>> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
>> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
>> at
>> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
>> at
>> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
>> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
>> at
>> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
>> at
>> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
>> at
>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
>> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> ...
>>
>>
>> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
>> to applying changes (addDocument calls) on the current "available" segment.
>> This takes >7 mins to copy a directory of size 1.4G. During this time
>> window, the searches are slow and the above thread stacks are observed.
>>
>>
>> Could there be any system level limits we're hitting?
>>
>>
>> Our test environment is:
>> lucene-2.3.2
>> 4x2.6 GHz, 16G memory
>> Red Hat 3.4.6-9
>>
>>
>> Thanks and regards,
>>
>>
>> - Gagan
>>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: slow search threads during a disk copy
Posted by Ian Lea <ia...@gmail.com>.
Well, there still sounds like plenty of scope for IO contention. Are
source and dest directories on the same disk? What does iostat show?
Could you use rsync rather than cp -rp, or something else that only
copies changes?
--
Ian.
On Mon, Aug 23, 2010 at 9:20 AM, <ga...@graffiti.net> wrote:
>
> We suspected the same at first, and so modified the flow to do the copy of the new version "before" searching is activated on it. So, there are no searches on the new version dir at the time of copy - yet we still see slowness during the disk copy.
>
>
> Thanks,
> - Gagan
>
>
>
>
>
> -----Original Message-----
> From: Anshum <an...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Mon, Aug 23, 2010 1:17 pm
> Subject: Re: slow search threads during a disk copy
>
>
> Seems like a case of I/O issues. You may be reading content off the index
> while performing searches while the I/O for copy is also happening.
>
> --
> Anshum Gupta
> http://ai-cafe.blogspot.com
>
>
> On Mon, Aug 23, 2010 at 1:12 PM, <ga...@graffiti.net> wrote:
>
>>
>> Hi all,
>>
>>
>> We're observing search threads slowing down during directory copies
>> performed during updates to the index. The thread dump shows search threads
>> blocked on a FSDirectory$FSIndexInput$Descriptor instance:
>>
>>
>>
>> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
>> monitor entry [0x988ed000..0x988edf30]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
>> - waiting to lock <0xc79c9900> (a
>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
>> at
>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
>> at
>> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
>> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
>> at
>> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
>> at
>> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
>> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
>> at
>> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
>> at
>> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
>> at
>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
>> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> ...
>>
>>
>> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
>> to applying changes (addDocument calls) on the current "available" segment.
>> This takes >7 mins to copy a directory of size 1.4G. During this time
>> window, the searches are slow and the above thread stacks are observed.
>>
>>
>> Could there be any system level limits we're hitting?
>>
>>
>> Our test environment is:
>> lucene-2.3.2
>> 4x2.6 GHz, 16G memory
>> Red Hat 3.4.6-9
>>
>>
>> Thanks and regards,
>>
>>
>> - Gagan
>>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: slow search threads during a disk copy
Posted by ga...@graffiti.net.
We suspected the same at first, and so modified the flow to do the copy of the new version "before" searching is activated on it. So, there are no searches on the new version dir at the time of copy - yet we still see slowness during the disk copy.
Thanks,
- Gagan
-----Original Message-----
From: Anshum <an...@gmail.com>
To: java-user@lucene.apache.org
Sent: Mon, Aug 23, 2010 1:17 pm
Subject: Re: slow search threads during a disk copy
Seems like a case of I/O issues. You may be reading content off the index
while performing searches while the I/O for copy is also happening.
--
Anshum Gupta
http://ai-cafe.blogspot.com
On Mon, Aug 23, 2010 at 1:12 PM, <ga...@graffiti.net> wrote:
>
> Hi all,
>
>
> We're observing search threads slowing down during directory copies
> performed during updates to the index. The thread dump shows search threads
> blocked on a FSDirectory$FSIndexInput$Descriptor instance:
>
>
>
> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
> monitor entry [0x988ed000..0x988edf30]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
> - waiting to lock <0xc79c9900> (a
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
> at
> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
> at
> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
> at
> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
> at
> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
> at
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> ...
>
>
> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
> to applying changes (addDocument calls) on the current "available" segment.
> This takes >7 mins to copy a directory of size 1.4G. During this time
> window, the searches are slow and the above thread stacks are observed.
>
>
> Could there be any system level limits we're hitting?
>
>
> Our test environment is:
> lucene-2.3.2
> 4x2.6 GHz, 16G memory
> Red Hat 3.4.6-9
>
>
> Thanks and regards,
>
>
> - Gagan
>
Re: slow search threads during a disk copy
Posted by Anshum <an...@gmail.com>.
Seems like a case of I/O issues. You may be reading content off the index
while performing searches while the I/O for copy is also happening.
--
Anshum Gupta
http://ai-cafe.blogspot.com
On Mon, Aug 23, 2010 at 1:12 PM, <ga...@graffiti.net> wrote:
>
> Hi all,
>
>
> We're observing search threads slowing down during directory copies
> performed during updates to the index. The thread dump shows search threads
> blocked on a FSDirectory$FSIndexInput$Descriptor instance:
>
>
>
> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
> monitor entry [0x988ed000..0x988edf30]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
> - waiting to lock <0xc79c9900> (a
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
> at
> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
> at
> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
> at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
> at
> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
> at
> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
> at
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> ...
>
>
> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
> to applying changes (addDocument calls) on the current "available" segment.
> This takes >7 mins to copy a directory of size 1.4G. During this time
> window, the searches are slow and the above thread stacks are observed.
>
>
> Could there be any system level limits we're hitting?
>
>
> Our test environment is:
> lucene-2.3.2
> 4x2.6 GHz, 16G memory
> Red Hat 3.4.6-9
>
>
> Thanks and regards,
>
>
> - Gagan
>