You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Lynn Settle <ly...@hotmail.com> on 2013/05/01 17:30:52 UTC
intermittent NPE with search
Hello,
Using Lucene 4.1.0, my application is designed to automatically run the exact same facet search every few minutes, varying only by range on a date-time field. However, occasionally the app fails with the below exception. Once this occurs, all Lucene searches begin to fail - even if the problem index is completely closed and reopened with a new searcher. Only an app restart makes searches functional again. It can take anywhere from hours to days for this exception to occur. I have been unable to reproduce it in the Eclipse environment.
(Tue Apr 23 13:50:24 PST 2013) EE:ERR [LuciusDataChangeListener v0] {null-14:202} EXCEPTION: java.lang.NullPointerException at org.apache.lucene.search.NumericRangeQuery$NumericRangeTermsEnum.accept(NumericRangeQuery.java:526) at org.apache.lucene.index.FilteredTermsEnum.next(FilteredTermsEnum.java:241) at org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:78) at org.apache.lucene.search.ConstantScoreAutoRewrite.rewrite(ConstantScoreAutoRewrite.java:95) at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:220) at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:286) at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:429) at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:616) at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:663) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)...
Note that line 526 is a call to nextRange(). But the stack trace stops there with no indication of what might be null. Did it even make it into the nextRange() method?
I downloaded and compiled the Lucene 4.1.0 source. I modified NumericRangeQuery NumericRangeTermsEnum accept method to record what value is null when this failure occurs.
protected final AcceptStatus accept(BytesRef term) { try { while (currentUpperBound == null || termComp.compare(term, currentUpperBound) > 0) { if (rangeBounds.isEmpty()) return AcceptStatus.END; // peek next sub-range, only seek if the current term is smaller than next lower bound if (termComp.compare(term, rangeBounds.getFirst()) < 0) return AcceptStatus.NO_AND_SEEK; // step forward to next range without seeking, as next lower range bound is less or equal current term nextRange(); } return AcceptStatus.YES; } catch (Throwable t) { String msg; if (term == null) { msg = "term is null"; } else if (rangeBounds == null) { msg = "rangeBounds is null"; } else if (termComp == null) { msg = "termComp == null"; } else { msg = "we don't know what is null!"; } throw new RuntimeException(msg, t); } } }
When the exception occurs, msg is "we don't know what is null!". So what could possibly be null? I'm completely lost at this point.
Thoughts?-Lynn
RE: intermittent NPE with search
Posted by Lynn Settle <ly...@hotmail.com>.
jrockit-jre1.6.0_03-Xms3000M-Xmx3000M-Djava.awt.fonts=$fontsdir
range query: timefield:[1366740416 TO 1366826816}
We index and search the date-time field as long values. We do plan on someday updating to a recent Sun JRE, but we have to get around to solving some server-agent NIO API conflicts first. In the meantime we are using jrockit.
The particular stack trace I first listed showed BooleanQuery which we also search on. I should have shown this stack trace which is what we usually get with the initial failure:
(Wed Apr 24 17:29:54 PST 2013) EE:ERR [LuciusDataChangeListener v0] {null-29:198} EXCEPTION: java.lang.NullPointerException at org.apache.lucene.search.NumericRangeQuery$NumericRangeTermsEnum.accept(NumericRangeQuery.java:526) at org.apache.lucene.index.FilteredTermsEnum.next(FilteredTermsEnum.java:241) at org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:78) at org.apache.lucene.search.ConstantScoreAutoRewrite.rewrite(ConstantScoreAutoRewrite.java:95) at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:220) at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:286) at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:616) at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:663) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)...
I appreciate any suggestions - even if just pointing me in some direction to dig further. Let me know if I can provide any more information.
thanks!-Lynn
> From: uwe@thetaphi.de
> To: java-user@lucene.apache.org
> Subject: RE: intermittent NPE with search
> Date: Wed, 1 May 2013 17:43:50 +0200
>
> Hi,
>
> This looks crazy! What Java version from which JVM vendor are you using? What are your JVM settings and how does the range query look like?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
RE: intermittent NPE with search
Posted by Lynn Settle <ly...@hotmail.com>.
Uwe
Thank you very much for your prompt responses. Yes, the old JRockit JVM version is the likely culprit. Initial testing with -XNoOpt has not yet produced the failure, although it is too early to be certain this fixes it.
I am very interested in attending your talk on testing Lucene/Solr with various JVMs.
>>Finally, I would recommend to use Oracle Java 7 instead of JRockit!
Agreed!
Thanks again
-Lynn
----------------------------------------
> From: uwe@thetaphi.de
> To: java-user@lucene.apache.org
> Subject: RE: intermittent NPE with search
> Date: Wed, 1 May 2013 18:42:18 +0200
>
> Hi,
>
> JRockit is unfortunately very buggy with Lucene, especially this very old version (1.6.0_03 is more than antique). The current one is jrockit-jdk1.6.0_45-R28.2.7-4.1.0. Unfortunately the later version still have bugs, that may also lead to corrupt indexes. Because of this we are running nightly tests at the moment only with the "-XnoOpt" setting (we are testing Lucene/Solr with Oracle, IBM J9 and JRockit), otherwise the tests fail regularly because of JVM bugs. Those bugs are listed on: http://wiki.apache.org/lucene-java/JavaBugs
>
> You might also be interested on the following talk: http://www.berlinbuzzwords.de/sessions/testing-lucene-and-solr-various-jvms-bugs-bugs-bugs
>
> Finally, I would recommend to use Oracle Java 7 instead of JRockit!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Lynn Settle [mailto:lynntsettle@hotmail.com]
> > Sent: Wednesday, May 01, 2013 6:28 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: intermittent NPE with search
> >
> > complete java version info:
> > java version "1.6.0_03"Java(TM) SE Runtime Environment (build 1.6.0_03-
> > b05)BEA JRockit(R) (build R27.5.0-110_o_CR366951-99572-1.6.0_03-20080529-
> > 1718-linux-x86_64, compiled mode)
> > > From: uwe@thetaphi.de
> > > To: java-user@lucene.apache.org
> > > Subject: RE: intermittent NPE with search
> > > Date: Wed, 1 May 2013 17:43:50 +0200
> > >
> > > Hi,
> > >
> > > This looks crazy! What Java version from which JVM vendor are you using?
> > What are your JVM settings and how does the range query look like?
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: intermittent NPE with search
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
JRockit is unfortunately very buggy with Lucene, especially this very old version (1.6.0_03 is more than antique). The current one is jrockit-jdk1.6.0_45-R28.2.7-4.1.0. Unfortunately the later version still have bugs, that may also lead to corrupt indexes. Because of this we are running nightly tests at the moment only with the "-XnoOpt" setting (we are testing Lucene/Solr with Oracle, IBM J9 and JRockit), otherwise the tests fail regularly because of JVM bugs. Those bugs are listed on: http://wiki.apache.org/lucene-java/JavaBugs
You might also be interested on the following talk: http://www.berlinbuzzwords.de/sessions/testing-lucene-and-solr-various-jvms-bugs-bugs-bugs
Finally, I would recommend to use Oracle Java 7 instead of JRockit!
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Lynn Settle [mailto:lynntsettle@hotmail.com]
> Sent: Wednesday, May 01, 2013 6:28 PM
> To: java-user@lucene.apache.org
> Subject: RE: intermittent NPE with search
>
> complete java version info:
> java version "1.6.0_03"Java(TM) SE Runtime Environment (build 1.6.0_03-
> b05)BEA JRockit(R) (build R27.5.0-110_o_CR366951-99572-1.6.0_03-20080529-
> 1718-linux-x86_64, compiled mode)
> > From: uwe@thetaphi.de
> > To: java-user@lucene.apache.org
> > Subject: RE: intermittent NPE with search
> > Date: Wed, 1 May 2013 17:43:50 +0200
> >
> > Hi,
> >
> > This looks crazy! What Java version from which JVM vendor are you using?
> What are your JVM settings and how does the range query look like?
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: intermittent NPE with search
Posted by Lynn Settle <ly...@hotmail.com>.
complete java version info:
java version "1.6.0_03"Java(TM) SE Runtime Environment (build 1.6.0_03-b05)BEA JRockit(R) (build R27.5.0-110_o_CR366951-99572-1.6.0_03-20080529-1718-linux-x86_64, compiled mode)
> From: uwe@thetaphi.de
> To: java-user@lucene.apache.org
> Subject: RE: intermittent NPE with search
> Date: Wed, 1 May 2013 17:43:50 +0200
>
> Hi,
>
> This looks crazy! What Java version from which JVM vendor are you using? What are your JVM settings and how does the range query look like?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
RE: intermittent NPE with search
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
This looks crazy! What Java version from which JVM vendor are you using? What are your JVM settings and how does the range query look like?
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Lynn Settle [mailto:lynntsettle@hotmail.com]
> Sent: Wednesday, May 01, 2013 5:31 PM
> To: java-user@lucene.apache.org
> Subject: intermittent NPE with search
>
> Hello,
> Using Lucene 4.1.0, my application is designed to automatically run the exact
> same facet search every few minutes, varying only by range on a date-time
> field. However, occasionally the app fails with the below exception. Once this
> occurs, all Lucene searches begin to fail - even if the problem index is
> completely closed and reopened with a new searcher. Only an app restart
> makes searches functional again. It can take anywhere from hours to days for
> this exception to occur. I have been unable to reproduce it in the Eclipse
> environment.
> (Tue Apr 23 13:50:24 PST 2013) EE:ERR [LuciusDataChangeListener v0] {null-
> 14:202} EXCEPTION: java.lang.NullPointerException at
> org.apache.lucene.search.NumericRangeQuery$NumericRangeTermsEnum.a
> ccept(NumericRangeQuery.java:526) at
> org.apache.lucene.index.FilteredTermsEnum.next(FilteredTermsEnum.java:
> 241) at
> org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectin
> gRewrite.java:78) at
> org.apache.lucene.search.ConstantScoreAutoRewrite.rewrite(ConstantScor
> eAutoRewrite.java:95) at
> org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewr
> ite(MultiTermQuery.java:220) at
> org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:28
> 6) at
> org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:429) at
> org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:616) at
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSea
> rcher.java:663) at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)...
> Note that line 526 is a call to nextRange(). But the stack trace stops there with
> no indication of what might be null. Did it even make it into the nextRange()
> method?
> I downloaded and compiled the Lucene 4.1.0 source. I modified
> NumericRangeQuery NumericRangeTermsEnum accept method to record
> what value is null when this failure occurs.
> protected final AcceptStatus accept(BytesRef term) { try { while
> (currentUpperBound == null || termComp.compare(term,
> currentUpperBound) > 0) { if (rangeBounds.isEmpty()) return
> AcceptStatus.END; // peek next sub-range, only seek if the current term
> is smaller than next lower bound if (termComp.compare(term,
> rangeBounds.getFirst()) < 0) return AcceptStatus.NO_AND_SEEK;
> // step forward to next range without seeking, as next lower range bound is
> less or equal current term nextRange(); } return
> AcceptStatus.YES; } catch (Throwable t) { String msg; if (term ==
> null) { msg = "term is null"; } else if (rangeBounds == null) { msg =
> "rangeBounds is null"; } else if (termComp == null) { msg =
> "termComp == null"; } else { msg = "we don't know what is null!";
> } throw new RuntimeException(msg, t); } } }
> When the exception occurs, msg is "we don't know what is null!". So what
> could possibly be null? I'm completely lost at this point.
> Thoughts?-Lynn
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org