You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by William Bell <bi...@gmail.com> on 2011/04/05 20:46:17 UTC

Script to remove all index.* leftovers

There is a bug that leaves old index.* directories in the Solr data directory.

Here is a script that will clean it up. I wanted to make sure this is
okay, without doing a core reload.

Thanks.

#!/bin/bash

DIR="/mnt/servers/solr/data"
LIST=`ls $DIR`
INDEX=`cat $DIR/index.properties | grep index\= | awk 'BEGIN { FS =
"=" } ; { print $2 }'`
echo $INDEX

for file in  $LIST
do
        if [ $INDEX == $file -o $file == "index" -o $file ==
"index.properties" -o $file == "replication.properties" -o $file ==
"spellchecker" ]
        then
                echo "skip: $file"
        else
                echo "rm -rf $DIR/$file"
                rm -rf $DIR/$file
        fi
done

Re: Script to remove all index.* leftovers

Posted by Markus Jelsma <ma...@openindex.io>.
Yes my mistake, you're right about #1.

On Wednesday 06 April 2011 05:25:50 William Bell wrote:
> Thank you for pointing out #2. The commitsToKeep is interesting, but I
> thought each commit would create a segment (before optimized) and be
> self contained in the index.* directory?
> 
> I would only run this on the slave.
> 
> Bill
> 
> 
> On Tue, Apr 5, 2011 at 2:54 PM, Markus Jelsma
> 
> <ma...@openindex.io> wrote:
> > Hi,
> > 
> > This seems alright as it leaves the current index in place, doesn't mess
> > with the spellchecker and leave the properties alone. But, there are two
> > problems:
> > 
> > 1. it doesn't take into account the commitsToKeep value set in the
> > deletion policy, and;
> > 2. it will remove any directory to which a current downloading
> > replication is targetted to.
> > 
> > Issue 1 may not be a big issue as most users leave only one commit on
> > disk but 2 is a real problem in master/slave architectures.
> > 
> > Cheers,
> > 
> >> There is a bug that leaves old index.* directories in the Solr data
> >> directory.
> >> 
> >> Here is a script that will clean it up. I wanted to make sure this is
> >> okay, without doing a core reload.
> >> 
> >> Thanks.
> >> 
> >> #!/bin/bash
> >> 
> >> DIR="/mnt/servers/solr/data"
> >> LIST=`ls $DIR`
> >> INDEX=`cat $DIR/index.properties | grep index\= | awk 'BEGIN { FS =
> >> "=" } ; { print $2 }'`
> >> echo $INDEX
> >> 
> >> for file in  $LIST
> >> do
> >>         if [ $INDEX == $file -o $file == "index" -o $file ==
> >> "index.properties" -o $file == "replication.properties" -o $file ==
> >> "spellchecker" ]
> >>         then
> >>                 echo "skip: $file"
> >>         else
> >>                 echo "rm -rf $DIR/$file"
> >>                 rm -rf $DIR/$file
> >>         fi
> >> done

-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: Script to remove all index.* leftovers

Posted by William Bell <bi...@gmail.com>.
Thank you for pointing out #2. The commitsToKeep is interesting, but I
thought each commit would create a segment (before optimized) and be
self contained in the index.* directory?

I would only run this on the slave.

Bill


On Tue, Apr 5, 2011 at 2:54 PM, Markus Jelsma
<ma...@openindex.io> wrote:
> Hi,
>
> This seems alright as it leaves the current index in place, doesn't mess with
> the spellchecker and leave the properties alone. But, there are two problems:
>
> 1. it doesn't take into account the commitsToKeep value set in the deletion
> policy, and;
> 2. it will remove any directory to which a current downloading replication is
> targetted to.
>
> Issue 1 may not be a big issue as most users leave only one commit on disk but
> 2 is a real problem in master/slave architectures.
>
> Cheers,
>
>> There is a bug that leaves old index.* directories in the Solr data
>> directory.
>>
>> Here is a script that will clean it up. I wanted to make sure this is
>> okay, without doing a core reload.
>>
>> Thanks.
>>
>> #!/bin/bash
>>
>> DIR="/mnt/servers/solr/data"
>> LIST=`ls $DIR`
>> INDEX=`cat $DIR/index.properties | grep index\= | awk 'BEGIN { FS =
>> "=" } ; { print $2 }'`
>> echo $INDEX
>>
>> for file in  $LIST
>> do
>>         if [ $INDEX == $file -o $file == "index" -o $file ==
>> "index.properties" -o $file == "replication.properties" -o $file ==
>> "spellchecker" ]
>>         then
>>                 echo "skip: $file"
>>         else
>>                 echo "rm -rf $DIR/$file"
>>                 rm -rf $DIR/$file
>>         fi
>> done
>

Re: Script to remove all index.* leftovers

Posted by Markus Jelsma <ma...@openindex.io>.
Hi,

This seems alright as it leaves the current index in place, doesn't mess with 
the spellchecker and leave the properties alone. But, there are two problems:

1. it doesn't take into account the commitsToKeep value set in the deletion 
policy, and;
2. it will remove any directory to which a current downloading replication is 
targetted to.

Issue 1 may not be a big issue as most users leave only one commit on disk but 
2 is a real problem in master/slave architectures.

Cheers,

> There is a bug that leaves old index.* directories in the Solr data
> directory.
> 
> Here is a script that will clean it up. I wanted to make sure this is
> okay, without doing a core reload.
> 
> Thanks.
> 
> #!/bin/bash
> 
> DIR="/mnt/servers/solr/data"
> LIST=`ls $DIR`
> INDEX=`cat $DIR/index.properties | grep index\= | awk 'BEGIN { FS =
> "=" } ; { print $2 }'`
> echo $INDEX
> 
> for file in  $LIST
> do
>         if [ $INDEX == $file -o $file == "index" -o $file ==
> "index.properties" -o $file == "replication.properties" -o $file ==
> "spellchecker" ]
>         then
>                 echo "skip: $file"
>         else
>                 echo "rm -rf $DIR/$file"
>                 rm -rf $DIR/$file
>         fi
> done