You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alméras Yannick <ya...@free.fr> on 2019/05/07 11:28:37 UTC
Error when merging segments ("terms out of order")
Hello !
I don't understand a problem on my ubuntu 18.04 solr server (version 7.6.0)...
When merge of segments is called, there is an error and then, the index is not
writable. The logs of a failed segments merging are at the end of this message
(here, I forced the merge but it's the same without forcing).
How can I debug this ? I don't understand the cause of the following error
java.lang.IllegalArgumentException: terms out of order: priorTerm=[6e 61 64 69
6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37],currentTerm=[6e 61 64 69 6e 65 2e
76 61 6c 61 64 65 5f 33 35 32 37]"
Another thing : I can't reproduce the bug on archlinux with the same
configuration of solr server (version 7.6.0)... ????
Thanks in advance,
Y. Alméras
*************
Error log :
*************
java.io.IOException: background merge hit exception: _1r(7.6.0):C17/1:delGen=1
_1s(7.6.0):C1 into _1t [maxNumSegments=2147483647]
at
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2020)
at
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1954)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:
659)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:
93)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:
68)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:
1950)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:
1926)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:
160)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:
69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:
62)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:
709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
515)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
377)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
323)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:
1634)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:
132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:
257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:
1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:
255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:
1317)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:
203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:
1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:
201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:
1219)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:
219)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:
126)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:
132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:
335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:
132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:
352)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:
281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:
102)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:
333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:
310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:
168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:
126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:
366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:
762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:
680)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: terms out of order:
priorTerm=[6e 61 64 69 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32
37],currentTerm=[6e 61 64 69 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37]
at
org.apache.lucene.util.StringHelper.bytesDifference(StringHelper.java:47)
at
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addTermsDict(Lucene70DocValuesConsumer.java:
461)
at
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.doAddSortedField(Lucene70DocValuesConsumer.java:
437)
at
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedField(Lucene70DocValuesConsumer.java:
394)
at
org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:
527)
at
org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:139)
at
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:
151)
at
org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:182)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:
126)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:
4068)
at
org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:196)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:
625)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:
662)
Re: Error when merging segments ("terms out of order")
Posted by Alméras Yannick <ya...@gmail.com>.
Hello !
(sorry for my last double post... I thought there was a problem with my other
mail address).
Le mardi 7 mai 2019, 15:25:13 CEST Shawn Heisey a écrit :
> On 5/7/2019 5:28 AM, Alméras Yannick wrote:
> > I don't understand a problem on my ubuntu 18.04 solr server (version
> > 7.6.0)...
> >
> > When merge of segments is called, there is an error and then, the index is
> > not writable. The logs of a failed segments merging are at the end of
> > this message (here, I forced the merge but it's the same without
> > forcing).
> >
> > How can I debug this ? I don't understand the cause of the following error
> >
> > java.lang.IllegalArgumentException: terms out of order: priorTerm=[6e 61
> > 64 69 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37],currentTerm=[6e 61 64 69
> > 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37]"
> >
> > Another thing : I can't reproduce the bug on archlinux with the same
> > configuration of solr server (version 7.6.0)... ????
>
> Are the two systems running different Java? What vendor and version of
> Java is on each server? Knowing the vendor is very important. Some of
> this information is on the admin UI dashboard, and you can get
> definitive information by running "java -version" at the commandline.
> Since it is easy to have multiple Java versions on a system, and only
> one of those versions is likely the be accessible via the system PATH,
> you need to make sure you're running the right one.
Here are the java versions which run solr :
* On Ubuntu (with error) :
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Ubuntu-3ubuntu118.04.3)
OpenJDK Server VM (build 11.0.2+9-Ubuntu-3ubuntu118.04.3, mixed mode, sharing)
* On archlinux (without being able to produce the error) :
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+4)
OpenJDK 64-Bit Server VM (build 11.0.3+4, mixed mode)
(I also tried with version 10 on archlinux, without being able to produce the
error ; note that Ubuntu is the prod server which is slowest than my archlinux
local dev server).
> Can you share the entire solr.log file that contains the problem, so we
> can see the entire sequence of errors?
Here it is :
https://www.dropbox.com/s/h1c8qpb3wzaim5k/solr.log?dl=0
If necessary, here are solrconfig.xml and schema.xml :
https://www.dropbox.com/s/pn19va6ss6n2gql/solrconfig.xml?dl=0
https://www.dropbox.com/s/9sjo7q1zi1y1o1k/schema.xml?dl=0
Best Regards,
Y. Alméras
Re: Error when merging segments ("terms out of order")
Posted by Alméras Yannick <ya...@gmail.com>.
Hello !
I'm going deeper into the mystery...
There is something wrong in my configuration or with solr because segments
merging never work as it should.
Example (step by step) :
* I clear my index.
* I had entries one by one. No problem when looking at items ; all fields are
ok -if no segments merging is done-.
* When segments merging is done, the fields of my entries go wrong ! For
example, here is the result of a query :
- before merging (ok ; all item_id equals is_nid) :
{
"response":{"numFound":3,"start":0,"docs":[
{
"item_id":"84",
"is_nid":84},
{
"item_id":"85",
"is_nid":85},
{
"item_id":"87",
"is_nid":87}]
}}
- after merging (wrong ; look at the second item where item_id <> is_nid and
the good number is is_nid) :
{
"response":{"numFound":3,"start":0,"docs":[
{
"item_id":"84",
"is_nid":84},
{
"item_id":"87",
"is_nid":85},
{
"item_id":"87",
"is_nid":87}]
}}
Here are links to my configuration files if it helps...
https://www.dropbox.com/s/pn19va6ss6n2gql/solrconfig.xml?dl=0
https://www.dropbox.com/s/9sjo7q1zi1y1o1k/schema.xml?dl=0
Best Regards,
Y. Alméras
Le mardi 7 mai 2019, 15:25:13 CEST Shawn Heisey a écrit :
> On 5/7/2019 5:28 AM, Alméras Yannick wrote:
> > I don't understand a problem on my ubuntu 18.04 solr server (version
> > 7.6.0)...
> >
> > When merge of segments is called, there is an error and then, the index is
> > not writable. The logs of a failed segments merging are at the end of
> > this message (here, I forced the merge but it's the same without
> > forcing).
> >
> > How can I debug this ? I don't understand the cause of the following error
> >
> > java.lang.IllegalArgumentException: terms out of order: priorTerm=[6e 61
> > 64 69 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37],currentTerm=[6e 61 64 69
> > 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37]"
> >
> > Another thing : I can't reproduce the bug on archlinux with the same
> > configuration of solr server (version 7.6.0)... ????
>
> Are the two systems running different Java? What vendor and version of
> Java is on each server? Knowing the vendor is very important. Some of
> this information is on the admin UI dashboard, and you can get
> definitive information by running "java -version" at the commandline.
> Since it is easy to have multiple Java versions on a system, and only
> one of those versions is likely the be accessible via the system PATH,
> you need to make sure you're running the right one.
>
> Can you share the entire solr.log file that contains the problem, so we
> can see the entire sequence of errors?
>
> You won't be able to attach the log to an email message. The mailing
> list filters out most attachments. You'll need to use a file sharing site.
>
> Normally I would say there's nothing sensitive in the log, but people
> disagree with that all the time. If you do redact anything, please do
> so sparingly, and do it in a way that we can tell redacted things apart
> from each other.
>
> Thanks,
> Shawn
Re: Error when merging segments ("terms out of order")
Posted by Yannick Alméras <ya...@gmail.com>.
Hello,
I installed it with the openjdk-11-jdk package from Ubuntu 18.04 repository with apt-get...
For the moment, no more problem with openjdk-8... I don't know the exact reason of problems with openjdk-11 32bit (?).
I will go on a 64bit system when possible (server is installed as 32bit since long time with many different things on it... That will be a big work).
Best regards,
Y. Alméras
Le 8 mai 2019 21:24:19 GMT+02:00, Shawn Heisey <ap...@elyograg.org> a écrit :
>On 5/8/2019 10:47 AM, Alméras Yannick wrote:
>> The problem of segments merging seems to be solved when I replace
>Java 11
>> 32bit with Java 8 32bit on my prod Ubuntu server... (On my dev
>archlinux
>> computer, no problem with Java 11 64bit...).
>
>It is strongly recommended to run a 64-bit version of Java if you can
>... but it does seem very weird that 32-bit Java would cause that
>particular issue.
>
>It's not that there's anything inherently wrong with 32-bit software
>...
>but 32-bit Java can only access 2GB of heap memory, and more is often
>needed.
>
>I'm curious how you obtained a 32 bit version of Java 11. Looking at
>the websites for Oracle Java and OpenJDK, I do not see any way to
>download it. Oracle stopped putting 32-bit versions on their public
>download page with Java 9.
>
>Thanks,
>Shawn
Re: Error when merging segments ("terms out of order")
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/8/2019 10:47 AM, Alméras Yannick wrote:
> The problem of segments merging seems to be solved when I replace Java 11
> 32bit with Java 8 32bit on my prod Ubuntu server... (On my dev archlinux
> computer, no problem with Java 11 64bit...).
It is strongly recommended to run a 64-bit version of Java if you can
... but it does seem very weird that 32-bit Java would cause that
particular issue.
It's not that there's anything inherently wrong with 32-bit software ...
but 32-bit Java can only access 2GB of heap memory, and more is often
needed.
I'm curious how you obtained a 32 bit version of Java 11. Looking at
the websites for Oracle Java and OpenJDK, I do not see any way to
download it. Oracle stopped putting 32-bit versions on their public
download page with Java 9.
Thanks,
Shawn
Re: Error when merging segments ("terms out of order")
Posted by Alméras Yannick <ya...@gmail.com>.
Hello !
The problem of segments merging seems to be solved when I replace Java 11
32bit with Java 8 32bit on my prod Ubuntu server... (On my dev archlinux
computer, no problem with Java 11 64bit...).
Hope this will be the real solution... "I cross my fingers" (from french "je
croise les doigts" ;-) ).
Best regards,
Y. Alméras
Le mardi 7 mai 2019, 15:25:13 CEST Shawn Heisey a écrit :
> On 5/7/2019 5:28 AM, Alméras Yannick wrote:
> > I don't understand a problem on my ubuntu 18.04 solr server (version
> > 7.6.0)...
> >
> > When merge of segments is called, there is an error and then, the index is
> > not writable. The logs of a failed segments merging are at the end of
> > this message (here, I forced the merge but it's the same without
> > forcing).
> >
> > How can I debug this ? I don't understand the cause of the following error
> >
> > java.lang.IllegalArgumentException: terms out of order: priorTerm=[6e 61
> > 64 69 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37],currentTerm=[6e 61 64 69
> > 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37]"
> >
> > Another thing : I can't reproduce the bug on archlinux with the same
> > configuration of solr server (version 7.6.0)... ????
>
> Are the two systems running different Java? What vendor and version of
> Java is on each server? Knowing the vendor is very important. Some of
> this information is on the admin UI dashboard, and you can get
> definitive information by running "java -version" at the commandline.
> Since it is easy to have multiple Java versions on a system, and only
> one of those versions is likely the be accessible via the system PATH,
> you need to make sure you're running the right one.
>
> Can you share the entire solr.log file that contains the problem, so we
> can see the entire sequence of errors?
>
> You won't be able to attach the log to an email message. The mailing
> list filters out most attachments. You'll need to use a file sharing site.
>
> Normally I would say there's nothing sensitive in the log, but people
> disagree with that all the time. If you do redact anything, please do
> so sparingly, and do it in a way that we can tell redacted things apart
> from each other.
>
> Thanks,
> Shawn
Re: Error when merging segments ("terms out of order")
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/7/2019 5:28 AM, Alméras Yannick wrote:
> I don't understand a problem on my ubuntu 18.04 solr server (version 7.6.0)...
>
> When merge of segments is called, there is an error and then, the index is not
> writable. The logs of a failed segments merging are at the end of this message
> (here, I forced the merge but it's the same without forcing).
>
> How can I debug this ? I don't understand the cause of the following error
>
> java.lang.IllegalArgumentException: terms out of order: priorTerm=[6e 61 64 69
> 6e 65 2e 76 61 6c 61 64 65 5f 33 35 32 37],currentTerm=[6e 61 64 69 6e 65 2e
> 76 61 6c 61 64 65 5f 33 35 32 37]"
>
> Another thing : I can't reproduce the bug on archlinux with the same
> configuration of solr server (version 7.6.0)... ????
Are the two systems running different Java? What vendor and version of
Java is on each server? Knowing the vendor is very important. Some of
this information is on the admin UI dashboard, and you can get
definitive information by running "java -version" at the commandline.
Since it is easy to have multiple Java versions on a system, and only
one of those versions is likely the be accessible via the system PATH,
you need to make sure you're running the right one.
Can you share the entire solr.log file that contains the problem, so we
can see the entire sequence of errors?
You won't be able to attach the log to an email message. The mailing
list filters out most attachments. You'll need to use a file sharing site.
Normally I would say there's nothing sensitive in the log, but people
disagree with that all the time. If you do redact anything, please do
so sparingly, and do it in a way that we can tell redacted things apart
from each other.
Thanks,
Shawn