You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by roger dimitri <ro...@yahoo.com> on 2008/03/20 19:47:58 UTC
backup RAMDirectory to file
Hi,
I am using the Directory class's copy method to periodically sync my RAM based index to a file based index that's supposed to serve as a hot backup. I want to know if this is the right way to maintain a periodic backup of my RAM based index and, if Yes, then is it reliable in a production instance with almost 12 to 15 GB of Lucene index loaded into the RAMDirectory.
Any advice is greatly appreciated.
I have a sample working code snippet here:
public class RAMtoFILEcopy_test1 {
static String[] queries = {
"a OR b AND c",
"(a OR b) AND c",
"a OR (b AND c)",
"a AND b",
"a AND b OR c AND d",
"(a AND b) OR (c AND d)",
"a AND (b OR c) AND d",
"((a AND b) OR c) AND d",
"a AND (b OR (c AND d))",
"a AND b AND c AND d",
"a OR b AND NOT c",
"(a OR b) AND NOT c",
"a OR (b AND NOT c)",
"a AND NOT d",
"a AND NOT b OR c AND NOT d",
"(a AND NOT b) OR (c AND NOT d)",
"a AND NOT (b OR c) AND NOT d",
"((a AND NOT b) OR c) AND NOT d",
"a AND NOT (b OR (c AND NOT d))",
"a AND NOT b AND NOT c AND NOT d",
"a OR NOT b",
"a OR NOT a",
"a b",
"a b c",
"a b (c d e)",
"+a +b",
"a -b",
"a +b -c",
"+a b -c",
"+a -b c",
"a -b -c",
"-a b c",
"a OR b c AND d",
"a OR b c",
"a AND b c",
"a OR b c OR d",
"a OR b c d OR e",
"a AND b c AND d",
"a AND b c d AND e"
};
public static void main(String argv[]) throws Exception {
Directory dir = new RAMDirectory();
Analyzer analyzer = new StandardAnalyzer();
IndexWriter writer = new IndexWriter(dir, analyzer, true);
for ( int i=0; i < queries.length; i++ ) {
Document doc = new Document();
doc.add(new Field("text", queries[i], Field.Store.YES, Field.Index.TOKENIZED));
writer.addDocument(doc);
}
writer.close();
Directory dir2 = FSDirectory.getDirectory("/tmp/TestHotbackup", true);
dir.copy(dir, dir2, true);
}
}
Thanks a lot,
Roger
____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs
Re: backup RAMDirectory to file
Posted by Michael McCandless <lu...@mikemccandless.com>.
I think both the original approach and addIndexes below will work
here, though the original approach should be faster.
But, there are some caveats. You have to make sure you do the backup
with the writer on the ramDir closed. If there is a writer open, it
could be changing files during your backup which will corrupt the
backup image.
Alternatively, you could do a true "hot backup" (not having to close
the writer on the ramDir) by using SnapshotDeletionPolicy.
You may want to do an incremental backup of the index. Since Lucene
is write-once, you only have to look for new filenames to copy (old
filenames will never be changed).
Mike
Grant Ingersoll wrote:
> I think you could try:
>
> IndexWriter writer = new IndexWriter(fileDirectory, ...)
> writer.addIndexes(ramDir)
>
> -Grant
>
> On Mar 20, 2008, at 2:47 PM, roger dimitri wrote:
>
>> Hi,
>> I am using the Directory class's copy method to periodically sync
>> my RAM based index to a file based index that's supposed to serve
>> as a hot backup. I want to know if this is the right way to
>> maintain a periodic backup of my RAM based index and, if Yes, then
>> is it reliable in a production instance with almost 12 to 15 GB of
>> Lucene index loaded into the RAMDirectory.
>> Any advice is greatly appreciated.
>> I have a sample working code snippet here:
>>
>> public class RAMtoFILEcopy_test1 {
>>
>>
>> static String[] queries = {
>> "a OR b AND c",
>> "(a OR b) AND c",
>> "a OR (b AND c)",
>> "a AND b",
>> "a AND b OR c AND d",
>> "(a AND b) OR (c AND d)",
>> "a AND (b OR c) AND d",
>> "((a AND b) OR c) AND d",
>> "a AND (b OR (c AND d))",
>> "a AND b AND c AND d",
>>
>> "a OR b AND NOT c",
>> "(a OR b) AND NOT c",
>> "a OR (b AND NOT c)",
>> "a AND NOT d",
>> "a AND NOT b OR c AND NOT d",
>> "(a AND NOT b) OR (c AND NOT d)",
>> "a AND NOT (b OR c) AND NOT d",
>> "((a AND NOT b) OR c) AND NOT d",
>> "a AND NOT (b OR (c AND NOT d))",
>> "a AND NOT b AND NOT c AND NOT d",
>>
>> "a OR NOT b",
>> "a OR NOT a",
>>
>> "a b",
>> "a b c",
>> "a b (c d e)",
>> "+a +b",
>> "a -b",
>> "a +b -c",
>> "+a b -c",
>> "+a -b c",
>> "a -b -c",
>> "-a b c",
>>
>> "a OR b c AND d",
>> "a OR b c",
>> "a AND b c",
>> "a OR b c OR d",
>> "a OR b c d OR e",
>> "a AND b c AND d",
>> "a AND b c d AND e"
>> };
>>
>> public static void main(String argv[]) throws Exception {
>> Directory dir = new RAMDirectory();
>> Analyzer analyzer = new StandardAnalyzer();
>>
>> IndexWriter writer = new IndexWriter(dir, analyzer, true);
>>
>> for ( int i=0; i < queries.length; i++ ) {
>> Document doc = new Document();
>> doc.add(new Field("text", queries[i], Field.Store.YES,
>> Field.Index.TOKENIZED));
>> writer.addDocument(doc);
>> }
>> writer.close();
>>
>>
>> Directory dir2 = FSDirectory.getDirectory("/tmp/
>> TestHotbackup", true);
>> dir.copy(dir, dir2, true);
>>
>> }
>>
>> }
>>
>>
>> Thanks a lot,
>> Roger
>>
>>
>>
>>
>>
>> _____________________________________________________________________
>> _______________
>> Never miss a thing. Make Yahoo your home page.
>> http://www.yahoo.com/r/hs
>
> --------------------------
> Grant Ingersoll
> http://www.lucenebootcamp.com
> Next Training: April 7, 2008 at ApacheCon Europe in Amsterdam
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: backup RAMDirectory to file
Posted by Grant Ingersoll <gs...@apache.org>.
I think you could try:
IndexWriter writer = new IndexWriter(fileDirectory, ...)
writer.addIndexes(ramDir)
-Grant
On Mar 20, 2008, at 2:47 PM, roger dimitri wrote:
> Hi,
> I am using the Directory class's copy method to periodically sync
> my RAM based index to a file based index that's supposed to serve as
> a hot backup. I want to know if this is the right way to maintain a
> periodic backup of my RAM based index and, if Yes, then is it
> reliable in a production instance with almost 12 to 15 GB of Lucene
> index loaded into the RAMDirectory.
> Any advice is greatly appreciated.
> I have a sample working code snippet here:
>
> public class RAMtoFILEcopy_test1 {
>
>
> static String[] queries = {
> "a OR b AND c",
> "(a OR b) AND c",
> "a OR (b AND c)",
> "a AND b",
> "a AND b OR c AND d",
> "(a AND b) OR (c AND d)",
> "a AND (b OR c) AND d",
> "((a AND b) OR c) AND d",
> "a AND (b OR (c AND d))",
> "a AND b AND c AND d",
>
> "a OR b AND NOT c",
> "(a OR b) AND NOT c",
> "a OR (b AND NOT c)",
> "a AND NOT d",
> "a AND NOT b OR c AND NOT d",
> "(a AND NOT b) OR (c AND NOT d)",
> "a AND NOT (b OR c) AND NOT d",
> "((a AND NOT b) OR c) AND NOT d",
> "a AND NOT (b OR (c AND NOT d))",
> "a AND NOT b AND NOT c AND NOT d",
>
> "a OR NOT b",
> "a OR NOT a",
>
> "a b",
> "a b c",
> "a b (c d e)",
> "+a +b",
> "a -b",
> "a +b -c",
> "+a b -c",
> "+a -b c",
> "a -b -c",
> "-a b c",
>
> "a OR b c AND d",
> "a OR b c",
> "a AND b c",
> "a OR b c OR d",
> "a OR b c d OR e",
> "a AND b c AND d",
> "a AND b c d AND e"
> };
>
> public static void main(String argv[]) throws Exception {
> Directory dir = new RAMDirectory();
> Analyzer analyzer = new StandardAnalyzer();
>
> IndexWriter writer = new IndexWriter(dir, analyzer, true);
>
> for ( int i=0; i < queries.length; i++ ) {
> Document doc = new Document();
> doc.add(new Field("text", queries[i], Field.Store.YES,
> Field.Index.TOKENIZED));
> writer.addDocument(doc);
> }
> writer.close();
>
>
> Directory dir2 = FSDirectory.getDirectory("/tmp/
> TestHotbackup", true);
> dir.copy(dir, dir2, true);
>
> }
>
> }
>
>
> Thanks a lot,
> Roger
>
>
>
>
>
> ____________________________________________________________________________________
> Never miss a thing. Make Yahoo your home page.
> http://www.yahoo.com/r/hs
--------------------------
Grant Ingersoll
http://www.lucenebootcamp.com
Next Training: April 7, 2008 at ApacheCon Europe in Amsterdam
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org