You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zisis Tachtsidis <zi...@runbox.com> on 2015/02/12 17:30:04 UTC

Index directory containing only segments.gen

I'm using SolrCloud 4.10.3 and the current setup is simple using 3 nodes with
1 shard. After a rolling restart of the Solr cluster I've ended up with 2
failing nodes reporting the following

org.apache.solr.servlet.SolrDispatchFilter
null:org.apache.solr.common.SolrException: SolrCore 'core' is not available
due to init failure: Error opening new searcher
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1574)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1686)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:853)
        ... 8 more
Caused by: java.nio.file.NoSuchFileException: /path/to/index/segments_1
        at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:334)
        at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:196)
        at
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
        at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:341)
        at
org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:454)
        at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:906)
        at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
        at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:792)
        at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
        at
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
        at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
        at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1537)
        ... 10 more

Checking the index directory of each node I found out that only
*segments.gen* was inside. What I could not determine is how I ended up with
this single file. Looking at the logs I could not find anything related. The
3rd node had its index intact.
Has anyone else encountered something similar?



--
View this message in context: http://lucene.472066.n3.nabble.com/Index-directory-containing-only-segments-gen-tp4186045.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Index directory containing only segments.gen

Posted by Erick Erickson <er...@gmail.com>.
Zisis:

It's not so much that the behavior has changed, and it's still
perfectly possible to use old-style solr.xml. Rather,
it's that there has been quite a lot of hardening how SolrCloud
creates things. I realize that the Collections API
is more of a black box and you have to take on faith that it's doing
the right thing. That said, the API was
written by people deeply involved in the guts of how SolrCloud works
and has some safeguards built in.

bq: Do you mean getting rid of Zookeeper snapshot and transcation logs,
basically clearing things and removing zknodes like clusterstate.json,
overseer and the like?

This is just a recipe for how I completely blow away Zookeeper's
knowledge of the system so I'm _sure_
nothing's hanging around. Note that I frequently bounce around from
one thing to another rather than maintain
a single system, so I'm pretty cavalier about this. Mostly I've spent
too much time being hammered by something
I'd forgotten I changed when moving from one problem to another. I
mean I'll spin up Solr 5.x, one or two
Solr 4.x versions and maybe trunk over the course of a day while
working on various problems; it's easy to lose track.
Of course you wouldn't want to resort to this in a "real" environment.
"rm -rf /tmp/zookeeper' is just an incantation
I often use ;)

But yes, that's what's going on. The clusterstate and all
non-ephemeral nodes are just gone. There are more
sophisticated ways to do this that aren't so blunt if you'd prefer. Do
be a bit aware, though, that if the replicas
are still on the Solr nodes, they can re-register themselves after you
blow away the Zookeeper info.

bq: Do you see anything unusual there.
Unusual, but not necessarily bad. The collections API takes much of
the guesswork out of this though. For instance,
are you quite sure you're naming each replica such that there are no
collisions? Note that you can also specify
what nodes the leaders and replicas go on, and you can script this if
using the Collections API. Ditto with adding
replicas (of course this latter came in later than 4.4 IIRC).

Not intimately familiar with HttpShardHandlerFactory, but on a quick
glance it's handling pooling threads for
sub-requests to other shards. May be way off base here.

Erick

On Fri, Feb 13, 2015 at 9:21 AM, Zisis Tachtsidis <zi...@runbox.com> wrote:
> Erick Erickson wrote
>> OK, I think this is the root of your problem:
>>
>> bq:  Everything was setup using the - now deprecated - tags
>> <cores>
>> and
>> <core>
>>   inside solr.xml.
>>
>> There are a bunch of ways this could go wrong. I'm pretty sure you
>> have something that would take quite a while to untangle, so unless
>> you have a _very_ good reason for making this work, I'd blow
>> everything away.
>
> I've started playing with SolrCloud before the new solr.xml made its
> appearance (in the example files of 4.4 distribution If I'm not mistaken)
> and since it was classified only as deprecated I decided to postpone the
> transition to the new solr.xml for the migration to Solr 5.0. Anyway, what
> you are saying is that the use of the new solrcloud-friendly configuration
> file is accompanied by changes in SolrCloud behavior?
>
>
> Erick Erickson wrote
>> If you're using an external Zookeeper shut if off and, 'rm -rf
>> /tmp/zookeeper'. If using embedded, you can remove zoo_data under your
>> SOLR_HOME.
>
> Do you mean getting rid of Zookeeper snapshot and transcation logs,
> basically clearing things and removing zknodes like clusterstate.json,
> overseer and the like?
>
>
> Erick Erickson wrote
>> OK, now use the Collections API to create your collection, see:
>> https://cwiki.apache.org/confluence/display/solr/Collections+API and
>> go from there (don't forget to push your configs to Zookeeper first)
>> and go from there.
>
> I've successfully tried your proposed approach using the new solr.xml but
> I've bypassed the collections API and added core.properties files inside my
> collection directories. Directories contain no other files and configuration
> has been preloaded into Zookeeper. I prefer to have everything ready before
> starting the Solr servers. Do you see anything unusual there?
>
> One last thing, what exactly is HttpShardHandlerFactory responsible for?
> Because there was no such definition in the deprecated solr.xml I was using.
>
> Thanks Erick,
> Zisis T.
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Index-directory-containing-only-segments-gen-tp4186045p4186316.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Index directory containing only segments.gen

Posted by Zisis Tachtsidis <zi...@runbox.com>.
Erick Erickson wrote
> OK, I think this is the root of your problem:
> 
> bq:  Everything was setup using the - now deprecated - tags 
> <cores>
> and 
> <core>
>   inside solr.xml.
> 
> There are a bunch of ways this could go wrong. I'm pretty sure you
> have something that would take quite a while to untangle, so unless
> you have a _very_ good reason for making this work, I'd blow
> everything away.

I've started playing with SolrCloud before the new solr.xml made its
appearance (in the example files of 4.4 distribution If I'm not mistaken)
and since it was classified only as deprecated I decided to postpone the
transition to the new solr.xml for the migration to Solr 5.0. Anyway, what
you are saying is that the use of the new solrcloud-friendly configuration
file is accompanied by changes in SolrCloud behavior?


Erick Erickson wrote
> If you're using an external Zookeeper shut if off and, 'rm -rf
> /tmp/zookeeper'. If using embedded, you can remove zoo_data under your
> SOLR_HOME.

Do you mean getting rid of Zookeeper snapshot and transcation logs,
basically clearing things and removing zknodes like clusterstate.json,
overseer and the like?


Erick Erickson wrote
> OK, now use the Collections API to create your collection, see:
> https://cwiki.apache.org/confluence/display/solr/Collections+API and
> go from there (don't forget to push your configs to Zookeeper first)
> and go from there.

I've successfully tried your proposed approach using the new solr.xml but
I've bypassed the collections API and added core.properties files inside my
collection directories. Directories contain no other files and configuration
has been preloaded into Zookeeper. I prefer to have everything ready before
starting the Solr servers. Do you see anything unusual there?

One last thing, what exactly is HttpShardHandlerFactory responsible for?
Because there was no such definition in the deprecated solr.xml I was using.

Thanks Erick,
Zisis T.





--
View this message in context: http://lucene.472066.n3.nabble.com/Index-directory-containing-only-segments-gen-tp4186045p4186316.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Index directory containing only segments.gen

Posted by Erick Erickson <er...@gmail.com>.
OK, I think this is the root of your problem:

bq:  Everything was setup using the - now deprecated - tags <cores>
and <core>  inside solr.xml.

There are a bunch of ways this could go wrong. I'm pretty sure you
have something that would take quite a while to untangle, so unless
you have a _very_ good reason for making this work, I'd blow
everything away.

First stop Zookeeper and all your Solr instances.

If you're using an external Zookeeper shut if off and, 'rm -rf
/tmp/zookeeper'. If using embedded, you can remove zoo_data under your
SOLR_HOME.

Completely remove all of your cores as in 'rm -rf corename' on all the
nodes. Nuke the entries in solr.xml and use the "cloud friendly"
solr.xml, I'd just copy the one in '...4.10/solr/example/solr'''. You
get the idea. Or you can brute-force this and just remove all of Solr
and re-install 4.10.3.

OK, now use the Collections API to create your collection, see:
https://cwiki.apache.org/confluence/display/solr/Collections+API and
go from there (don't forget to push your configs to Zookeeper first)
and go from there.

Note that when I suggest you uninstall/reinstall Solr it's simply
because that's conceptually easier, of course you can spend some time
untangling things with your existing setup, but I really question
whether it's worth the effort.

Best,
Erick

On Thu, Feb 12, 2015 at 11:53 AM, Zisis Tachtsidis <zi...@runbox.com> wrote:
> From the logs I've got one instance failing as described in my first comment
> and the other two failing during PeerSync recovery when trying to
> communicate with the server that was missing the segments_* files. The
> exception follows
>
>
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at: http://server:host/solr/core
>     at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:566)
>     at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
>     at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
>     at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:157)
>     at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.http.client.ClientProtocolException
>     at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
>     at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>     at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>     at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
>     ... 10 more
> Caused by: org.apache.http.ProtocolException: Invalid header: ,code=500}
>     at
> org.apache.http.impl.io.AbstractMessageParser.parseHeaders(AbstractMessageParser.java:232)
>     at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:267)
>     at
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>     at
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
>     at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
>     at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
>     at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
>     at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
>     at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
>     at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>     ... 13 more
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Index-directory-containing-only-segments-gen-tp4186045p4186113.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Index directory containing only segments.gen

Posted by Zisis Tachtsidis <zi...@runbox.com>.
>From the logs I've got one instance failing as described in my first comment
and the other two failing during PeerSync recovery when trying to
communicate with the server that was missing the segments_* files. The
exception follows


org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at: http://server:host/solr/core
    at
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:566)
    at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
    at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
    at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:157)
    at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.client.ClientProtocolException
    at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
    at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
    at
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
    ... 10 more
Caused by: org.apache.http.ProtocolException: Invalid header: ,code=500}
    at
org.apache.http.impl.io.AbstractMessageParser.parseHeaders(AbstractMessageParser.java:232)
    at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:267)
    at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
    at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
    at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
    at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
    at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
    at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
    at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    ... 13 more




--
View this message in context: http://lucene.472066.n3.nabble.com/Index-directory-containing-only-segments-gen-tp4186045p4186113.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Index directory containing only segments.gen

Posted by Zisis Tachtsidis <zi...@runbox.com>.
Well, I don't know If I'm being helpful but here goes.
My clusterstate.json actually has no leader for the shard in question. I
have 2 nodes as "recovery_failed" and one as "down". No leaders there. I've
not used core admin or collections api to create anything. Everything was
setup using the - now deprecated - tags <cores> and <core>  inside solr.xml. 

Also index directories are different since I ended up copying the index from
the one node that still had it to the other too and restarting again. 




--
View this message in context: http://lucene.472066.n3.nabble.com/Index-directory-containing-only-segments-gen-tp4186045p4186107.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Index directory containing only segments.gen

Posted by Erick Erickson <er...@gmail.com>.
So after adding some docs to the index (and committing) with those two
nodes active,
do segment files magically appear?

My _guess_ is that there's something radially wrong with you set up
the collection. Did
you by any chance use the core admin API to create the cores? That can lead to
"interesting" results of you don't get everything just right. For
instance, if you point
the data dir for all three nodes at the same directory...

What does your clusterstate.json file look like?

Best,
Erick

On Thu, Feb 12, 2015 at 8:30 AM, Zisis Tachtsidis <zi...@runbox.com> wrote:
> I'm using SolrCloud 4.10.3 and the current setup is simple using 3 nodes with
> 1 shard. After a rolling restart of the Solr cluster I've ended up with 2
> failing nodes reporting the following
>
> org.apache.solr.servlet.SolrDispatchFilter
> null:org.apache.solr.common.SolrException: SolrCore 'core' is not available
> due to init failure: Error opening new searcher
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
>         at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1574)
>         at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1686)
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:853)
>         ... 8 more
> Caused by: java.nio.file.NoSuchFileException: /path/to/index/segments_1
>         at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>         at
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>         at java.nio.channels.FileChannel.open(FileChannel.java:287)
>         at java.nio.channels.FileChannel.open(FileChannel.java:334)
>         at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:196)
>         at
> org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
>         at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:341)
>         at
> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:454)
>         at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:906)
>         at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
>         at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
>         at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:792)
>         at
> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
>         at
> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
>         at
> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
>         at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
>         at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1537)
>         ... 10 more
>
> Checking the index directory of each node I found out that only
> *segments.gen* was inside. What I could not determine is how I ended up with
> this single file. Looking at the logs I could not find anything related. The
> 3rd node had its index intact.
> Has anyone else encountered something similar?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Index-directory-containing-only-segments-gen-tp4186045.html
> Sent from the Solr - User mailing list archive at Nabble.com.