index dir of core xxx is already locked.

[Chetas Joshi <ch...@gmail.com>]

Hi,

I have a SolrCloud (on HDFS) of 52 nodes. I have 3 collections each with 50
shards and maxShards per node for every collection is 1.

I am having problem restarting a solr shard for a collection.

When I restart, there is always a particular shard of a particular
collection that remains down. The 2 shards on the same host for the rest of
the collections are up and running.

Before restarting, I delete all the write.lock files from the data dir. But
every time I restart I get the same exception.

index dir yyy of core xxx is already locked. The most likely cause is
another Solr server (or another solr core in this server) also configured
to use this directory; other possible causes may be specific to lockType:
hdfs

Thanks!

Re: index dir of core xxx is already locked.

[Erick Erickson <er...@gmail.com>]

You really need to go through your Solr logs for the
shard(s) in question very carefully. There'll be a lot of
information dumped out, including paths used for
everything.

I suspect you've unknowingly created this situation when
trying to set up Solr, HDFS or whatever but I can't really say
what it would be without inspecting your entire installation,
which I can;t do remotely.

Best,
Erick

On Wed, Nov 16, 2016 at 3:01 PM, Chetas Joshi <ch...@gmail.com> wrote:
> I don't kill the solr instance forcefully using "kill -9".
>
> I checked the core.properties file for that shard. The content is different
> from the core.properties file for all the other shards.
> It has the following two lines which are different
>
> config=solrconfig.xml
>
> schema=schema.xml
>
> In other shards, it is
>
> collection.configName=v4 (name I have given to the config)
>
> name=collectionName_shardNumber_replica1
>
> Should I modify this file before restarting the Cloud?
>
> There is a strange thing I just observed about the data dir of the shard
> that is not coming up. There is an addition index dir that has been created
>
> hdfs://Ingest/solr53/collection/core_node32/data/index/index/
>
> The size and content is same as of
>
> hdfs://Ingest/solr53/collection/core_node32/data/index/
>
>
> What could be the reason of this extra dir? Should I delete it?
>
>
> Thanks!
>
>
> On Wed, Nov 16, 2016 at 1:51 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> bq: Before restarting, I delete all the write.lock files from the data
>> dir. But
>> every time I restart I get the same exception.
>>
>> First, this shouldn't be necessary. Are you by any chance killing the
>> Solr instances with
>> the equivalent of "kill -9"? Allow them to shut down gracefully. That
>> said, until recently
>> the bin/solr script would kill them forcefully after 5 seconds which
>> is too short an interval.
>>
>> But the error really is telling you that somehow two or more Solr
>> cores are pointing at the
>> same data directory. Whichever one gets there first will block any
>> later cores with the
>> message you see. So check your core.properties files and your HDFS magic
>> to see
>> how this is occurring would be my first guess.
>>
>> Best,
>> Erick
>>
>> On Wed, Nov 16, 2016 at 1:38 PM, Chetas Joshi <ch...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I have a SolrCloud (on HDFS) of 52 nodes. I have 3 collections each with
>> 50
>> > shards and maxShards per node for every collection is 1.
>> >
>> > I am having problem restarting a solr shard for a collection.
>> >
>> > When I restart, there is always a particular shard of a particular
>> > collection that remains down. The 2 shards on the same host for the rest
>> of
>> > the collections are up and running.
>> >
>> > Before restarting, I delete all the write.lock files from the data dir.
>> But
>> > every time I restart I get the same exception.
>> >
>> > index dir yyy of core xxx is already locked. The most likely cause is
>> > another Solr server (or another solr core in this server) also configured
>> > to use this directory; other possible causes may be specific to lockType:
>> > hdfs
>> >
>> > Thanks!
>>

Re: index dir of core xxx is already locked.

[Chetas Joshi <ch...@gmail.com>]

I don't kill the solr instance forcefully using "kill -9".

I checked the core.properties file for that shard. The content is different
from the core.properties file for all the other shards.
It has the following two lines which are different

config=solrconfig.xml

schema=schema.xml

In other shards, it is

collection.configName=v4 (name I have given to the config)

name=collectionName_shardNumber_replica1

Should I modify this file before restarting the Cloud?

There is a strange thing I just observed about the data dir of the shard
that is not coming up. There is an addition index dir that has been created

hdfs://Ingest/solr53/collection/core_node32/data/index/index/

The size and content is same as of

hdfs://Ingest/solr53/collection/core_node32/data/index/


What could be the reason of this extra dir? Should I delete it?


Thanks!


On Wed, Nov 16, 2016 at 1:51 PM, Erick Erickson <er...@gmail.com>
wrote:

> bq: Before restarting, I delete all the write.lock files from the data
> dir. But
> every time I restart I get the same exception.
>
> First, this shouldn't be necessary. Are you by any chance killing the
> Solr instances with
> the equivalent of "kill -9"? Allow them to shut down gracefully. That
> said, until recently
> the bin/solr script would kill them forcefully after 5 seconds which
> is too short an interval.
>
> But the error really is telling you that somehow two or more Solr
> cores are pointing at the
> same data directory. Whichever one gets there first will block any
> later cores with the
> message you see. So check your core.properties files and your HDFS magic
> to see
> how this is occurring would be my first guess.
>
> Best,
> Erick
>
> On Wed, Nov 16, 2016 at 1:38 PM, Chetas Joshi <ch...@gmail.com>
> wrote:
> > Hi,
> >
> > I have a SolrCloud (on HDFS) of 52 nodes. I have 3 collections each with
> 50
> > shards and maxShards per node for every collection is 1.
> >
> > I am having problem restarting a solr shard for a collection.
> >
> > When I restart, there is always a particular shard of a particular
> > collection that remains down. The 2 shards on the same host for the rest
> of
> > the collections are up and running.
> >
> > Before restarting, I delete all the write.lock files from the data dir.
> But
> > every time I restart I get the same exception.
> >
> > index dir yyy of core xxx is already locked. The most likely cause is
> > another Solr server (or another solr core in this server) also configured
> > to use this directory; other possible causes may be specific to lockType:
> > hdfs
> >
> > Thanks!
>

Re: index dir of core xxx is already locked.

[Erick Erickson <er...@gmail.com>]

bq: Before restarting, I delete all the write.lock files from the data dir. But
every time I restart I get the same exception.

First, this shouldn't be necessary. Are you by any chance killing the
Solr instances with
the equivalent of "kill -9"? Allow them to shut down gracefully. That
said, until recently
the bin/solr script would kill them forcefully after 5 seconds which
is too short an interval.

But the error really is telling you that somehow two or more Solr
cores are pointing at the
same data directory. Whichever one gets there first will block any
later cores with the
message you see. So check your core.properties files and your HDFS magic to see
how this is occurring would be my first guess.

Best,
Erick

On Wed, Nov 16, 2016 at 1:38 PM, Chetas Joshi <ch...@gmail.com> wrote:
> Hi,
>
> I have a SolrCloud (on HDFS) of 52 nodes. I have 3 collections each with 50
> shards and maxShards per node for every collection is 1.
>
> I am having problem restarting a solr shard for a collection.
>
> When I restart, there is always a particular shard of a particular
> collection that remains down. The 2 shards on the same host for the rest of
> the collections are up and running.
>
> Before restarting, I delete all the write.lock files from the data dir. But
> every time I restart I get the same exception.
>
> index dir yyy of core xxx is already locked. The most likely cause is
> another Solr server (or another solr core in this server) also configured
> to use this directory; other possible causes may be specific to lockType:
> hdfs
>
> Thanks!