You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Arthur Smith (JIRA)" <ji...@apache.org> on 2007/01/12 22:31:27 UTC
[jira] Created: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
------------------------------------------------------------------------------
Key: LUCENE-772
URL: https://issues.apache.org/jira/browse/LUCENE-772
Project: Lucene - Java
Issue Type: Bug
Components: Search
Affects Versions: 2.1
Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
Reporter: Arthur Smith
Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
"http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:215)
- locked <0x3d73cba8> (a java.util.zip.Inflater)
at java.util.zip.Inflater.inflate(Inflater.java:232)
at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
- locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
at org.apache.lucene.search.Hits.doc(Hits.java:104)
[...]
Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Arthur Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arthur Smith resolved LUCENE-772.
---------------------------------
Resolution: Fixed
Fix Version/s: 2.1
Ok, running the 1-12-07 nightly build seems to have fixed the problem - we've had no repeats after a couple of days of heavy use. Sounds like we were running into the same issue that was just addressed, thanks!
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
> Fix For: 2.1
>
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Arthur Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464359 ]
Arthur Smith commented on LUCENE-772:
-------------------------------------
By the way, this runaway behavior did happen after we caught 1 i/o exception and retried the query (we close the IndexSearcher and open a new one whenever we get an exception, or if isCurrent is false). Unfortunately I don't have details on the i/o exception - I'll try to get that if we can make it happen again.
After the i/o exception, the runaway thread was using 100% of the CPU for at least 40 minutes.
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Chuck Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464358 ]
Chuck Williams commented on LUCENE-772:
---------------------------------------
I had many concurrency problems with java.util.zip and ended up switching to jzlib (www.jcraft.com/jzlib/), which has the benefit of being a pure Java implementation that you can easily debug and modify if necessary. There were no performance degradations associated with the shift, and jzlib has been solid for me. This would require that you compress and decompress external to Lucene, but that can be much more efficient anyway (especially with LUCENE-362, although the patch in jira probably won't apply cleanly with current code).
Not sure this helps, but I'd bet your issue is somehow related to concurrency.
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716233#action_12716233 ]
Uwe Schindler commented on LUCENE-772:
--------------------------------------
Corrumptions are not automatically fixed, but in 2.4.1 (end even before) there is a tool called CheckIndex, which you can call with thex path to index. But it cannot really repair issues, but can drop corrupt segments (parts of the index).
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
> Fix For: 2.1
>
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reopened LUCENE-772:
---------------------------------------
Reopening; nothing was fixed here...
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
> Fix For: 2.1
>
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-772) Index corruption can cause infinite
spin loop when Lucene attempts to incorrectly uncompress fields
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated LUCENE-772:
-------------------------------
Fix Version/s: (was: 2.1)
3.1
> Index corruption can cause infinite spin loop when Lucene attempts to incorrectly uncompress fields
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
> Fix For: 3.1
>
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-772) Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
Posted by robert engels <re...@ix.netcom.com>.
I think you are mistaken (in a sense).
ZipInputStream is thread-safe in as much as an InputStream is thread-
safe - which is isn't.
I think what you are most likely seeing is that you are attempting to
uncompress 'corrupted data' and the decompressor is getting confused
- thus the hang.
On Jan 12, 2007, at 4:18 PM, Arthur Smith (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENE-772?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel#action_12464372 ]
>
> Arthur Smith commented on LUCENE-772:
> -------------------------------------
>
> Chuck - thanks - though IndexSearcher is supposed to be thread
> safe, so maybe it shouldn't be calling java.util.zip?
>
> But now that I look again, we shouldn't have *any* compressed
> fields in our indexes! This has got to be an index corruption issue
> - we'll get the latest build out there and see if the recent fixes
> (LUCENE-140 in particular) help.
>
>> Lucene infinite loop? In FieldsReader.uncompress called from
>> IndexSearcher.doc
>> ---------------------------------------------------------------------
>> ---------
>>
>> Key: LUCENE-772
>> URL: https://issues.apache.org/jira/browse/LUCENE-772
>> Project: Lucene - Java
>> Issue Type: Bug
>> Components: Search
>> Affects Versions: 2.1
>> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06
>> and _08 at least, tomcat 5.5. IndexSearcher searching index
>> mounted via NFS; using new lockless commits... We're using the
>> 01-05-07 nightly build of lucene
>> Reporter: Arthur Smith
>>
>> Unfortunately I don't have a reproducible case of this (yet), but
>> it's happened twice now on our production servers in the past few
>> days, after we switched to the new lockless commits (thanks!).
>> What we're seeing is the search thread running away in the middle
>> of a seemingly ordinary search, after several hundred thousand
>> queries that worked just fine. The search index is NFS mounted,
>> and is updated every minute or so during the day by an indexing
>> process running on a separate server. We do get occasional I/O
>> errors, but we catch and retry and it seems ok after a few seconds.
>> But twice now we've had run-away threads; the thread dump in both
>> cases was caught in the middle of java.util.zip.Inflater:
>> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable
>> [0x1f17c000..0x1f17e0b0]
>> at java.util.zip.Inflater.inflateBytes(Native Method)
>> at java.util.zip.Inflater.inflate(Inflater.java:215)
>> - locked <0x3d73cba8> (a java.util.zip.Inflater)
>> at java.util.zip.Inflater.inflate(Inflater.java:232)
>> at org.apache.lucene.index.FieldsReader.uncompress
>> (FieldsReader.java:388)
>> at org.apache.lucene.index.FieldsReader.addField
>> (FieldsReader.java:222)
>> at org.apache.lucene.index.FieldsReader.doc
>> (FieldsReader.java:105)
>> at org.apache.lucene.index.SegmentReader.document
>> (SegmentReader.java:324)
>> - locked <0x3cefbdd8> (a
>> org.apache.lucene.index.SegmentReader) at
>> org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
>> at org.apache.lucene.index.IndexReader.document
>> (IndexReader.java:360) at
>> org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
>> at org.apache.lucene.search.Hits.doc(Hits.java:104)
>> [...]
>> Any ideas what this could be? Thanks!
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators: https://issues.apache.org/jira/secure/
> Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/
> software/jira
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Arthur Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464372 ]
Arthur Smith commented on LUCENE-772:
-------------------------------------
Chuck - thanks - though IndexSearcher is supposed to be thread safe, so maybe it shouldn't be calling java.util.zip?
But now that I look again, we shouldn't have *any* compressed fields in our indexes! This has got to be an index corruption issue - we'll get the latest build out there and see if the recent fixes (LUCENE-140 in particular) help.
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716234#action_12716234 ]
Michael McCandless commented on LUCENE-772:
-------------------------------------------
Olivier, in your case are you certain you did not add compressed fields, yet Lucene is trying to decompress the fields on loading?
In which case this sounds like index corruption that unfortunately causes zip to be invoked on invalid data, which it turn can apparently cause zip to enter an infinite loop.
Unfortunately, once an index is corrupt it can cause any number of crazy things to happen. We need to get to the root cause of the corruption. So it's not clear on upgrading to 2.4.1 with a still-corrupt index, whether 2.4.1 will behave any differently. It's still worth a shot: upgrade to 2.4.1 and run CheckIndex with java's assertions enabled, and see what happens?
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
> Fix For: 2.1
>
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-772) Index corruption can cause infinite
spin loop when Lucene attempts to incorrectly uncompress fields
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-772:
--------------------------------------
Summary: Index corruption can cause infinite spin loop when Lucene attempts to incorrectly uncompress fields (was: Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc)
Reopened & changed title
> Index corruption can cause infinite spin loop when Lucene attempts to incorrectly uncompress fields
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
> Fix For: 2.1
>
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-772) Lucene infinite loop? In
FieldsReader.uncompress called from IndexSearcher.doc
Posted by "Olivier Dony (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716229#action_12716229 ]
Olivier Dony commented on LUCENE-772:
-------------------------------------
I know this sounds silly on a 2y+ old issue, but I have a legacy server running in production on linux with jdk1.5.0_11, tomcat 5.0.28, and lucene-core-2.1.0 on which this very problem has just occured (same symptoms and thread dump).
This issue has been marked as fixed in 2.1, but with apparently not much evidence that something was explicitly done to fix it?
I have no idea how to reproduce it, and it has only happened once.
Can I assume that switching to Lucene 2.4.1 will automatically fix the likely index corruption that triggered this or that something else will prevent trying to deflate a non-zipped field? Or is this a wild guess?
I guess I'll have to switch to 2.4.1 anyway before reporting it, but I was wondering if there's a chance this has been explicitly addressed *after* 2.1.0...
Thanks to anyone that would still be reading this...!
> Lucene infinite loop? In FieldsReader.uncompress called from IndexSearcher.doc
> ------------------------------------------------------------------------------
>
> Key: LUCENE-772
> URL: https://issues.apache.org/jira/browse/LUCENE-772
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Linux (gentoo 2.6.17 at least), jdk 1.5.0_06 and _08 at least, tomcat 5.5. IndexSearcher searching index mounted via NFS; using new lockless commits... We're using the 01-05-07 nightly build of lucene
> Reporter: Arthur Smith
> Fix For: 2.1
>
>
> Unfortunately I don't have a reproducible case of this (yet), but it's happened twice now on our production servers in the past few days, after we switched to the new lockless commits (thanks!). What we're seeing is the search thread running away in the middle of a seemingly ordinary search, after several hundred thousand queries that worked just fine. The search index is NFS mounted, and is updated every minute or so during the day by an indexing process running on a separate server. We do get occasional I/O errors, but we catch and retry and it seems ok after a few seconds.
> But twice now we've had run-away threads; the thread dump in both cases was caught in the middle of java.util.zip.Inflater:
> "http-8080-3" daemon prio=1 tid=0x08294688 nid=0x1f0d runnable [0x1f17c000..0x1f17e0b0]
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:215)
> - locked <0x3d73cba8> (a java.util.zip.Inflater)
> at java.util.zip.Inflater.inflate(Inflater.java:232)
> at org.apache.lucene.index.FieldsReader.uncompress(FieldsReader.java:388)
> at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:222)
> at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:105)
> at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:324)
> - locked <0x3cefbdd8> (a org.apache.lucene.index.SegmentReader) at org.apache.lucene.index.MultiReader.document(MultiReader.java:108)
> at org.apache.lucene.index.IndexReader.document(IndexReader.java:360) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:84)
> at org.apache.lucene.search.Hits.doc(Hits.java:104)
> [...]
> Any ideas what this could be? Thanks!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org