You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Dick de Jong <dd...@gmail.com> on 2006/01/08 05:54:22 UTC
Basic question on opening 2 IndexWriters on same Directory - I do not get IOException ...
Question:
Run attached java class and see the also attached when I ran it.
I assumed the second writer should get an IOException in getWriter (in
first TESTCASE).
However this does not happen! This only happens when I open both
writers with flag 'createIndex'=true (see
also second test case).
Is this a misunderstanding from my side or what is this?
ENVIRONMENT:
Windows Professional XP SP2
Lucene 1.4 (java)
OUTPUT OF MAIN METHOD:
=======================
START TESTCASE
Test open 2 index writers on index c:\temp\indexdir
writer 1 uses createIndex =true
writer 2 uses createIndex =true // why not an
IOException here ??
Opening writer w1 on: c:\temp\indexdir
Writing document to index using writer :w1
Opening writer w2 on: c:\temp\indexdir
Writing document to index using writer :w2
Writing document to index using writer :w1
Closing writer w1
Closing writer w2
END of test case ...
=======================
=======================
START TESTCASE
Test open 2 index writers on index c:\temp\indexdir
writer 1 uses createIndex =true
writer 2 uses createIndex =false
Opening writer w1 on: c:\temp\indexdir
Writing document to index using writer :w1
Opening writer w2 on: c:\temp\indexdir
In open writer caught IOException:
Lock obtain timed out: Lock@C:\DOCUME~1\ddjong\LOCALS~1\Temp\lucene-
ea51d1f7bb672168eda3ee90698e936d-write.lock
Closing writer w1
END of test case ...
=======================
USED JAVA class (src):
==================================
import java.io.IOException;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
public class TestIndexWriter
{
private IndexWriter w1;
private IndexWriter w2;
private String indexDir;
private Analyzer analyzer;
private Document document;
private TestIndexWriter()
{
// just create dummy document for test purposes, document is
reused.
this.document = new Document();
this.document.add(Field.Keyword("fname", "fvalue"));
// indexDirectory Folder to be used
this.indexDir = "c:\\temp\\indexdir";
this.analyzer = new StandardAnalyzer();
}
private IndexWriter getWriter(String name, boolean createIndex) throws
IOException
{
System.out.println("Opening writer " + name + " on: " +
this.indexDir);
return new IndexWriter(this.indexDir, this.analyzer, createIndex);
}
private void closeWriter(String name, IndexWriter w)
{
try {
if (w != null) {
System.out.println("Closing writer " + name);
w.close();
}
}
catch (IOException e) {
System.out.println("Don't bubble error in close writer; error:
\n" + e.getMessage());
}
}
private void addDocument(String name, IndexWriter w) throws IOException
{
System.out.println("Writing document to index using writer :" +
name);
w.addDocument(this.document);
}
public void testOpenIndexWriters(boolean w1_create, boolean w2_create)
{
System.out.println("=======================");
System.out.println("START TESTCASE");
System.out.println("Test open 2 index writers on index " +
this.indexDir);
System.out.println(" \twriter 1 uses createIndex =" + w1_create);
System.out.println(" \twriter 2 uses createIndex =" + w2_create);
this.w1 = this.w2 = null;
try
{
w1 = this.getWriter("w1", w1_create);
addDocument("w1", w1);
w2 = this.getWriter("w2", w2_create);
addDocument("w2", w2);
addDocument("w1", w1);
}
catch (IOException e)
{
System.out.println("In open writer caught IOException:");
System.out.println("\t" + e.getMessage());
}
finally
{
this.closeWriter("w1", w1);
this.closeWriter("w2", w2);
System.out.println("END of test case ...");
System.out.println("=======================");
}
}
public static void main(String[] args)
{
TestIndexWriter tw = new TestIndexWriter();
tw.testOpenIndexWriters(true, true);
tw.testOpenIndexWriters(true, false);
}
}
RE: Basic question on opening 2 IndexWriters on same Directory - I do not get IOException ...
Posted by Koji Sekiguchi <ko...@m4.dion.ne.jp>.
Hi Dick,
I agree with you. Now I cannot understand why your
original code didn't work.
If you find the answer, please let me know!
regrads,
Koji
> -----Original Message-----
> From: Dick de Jong [mailto:ddjong67@gmail.com]
> Sent: Tuesday, January 10, 2006 6:41 PM
> To: java-user@lucene.apache.org
> Subject: Re: Basic question on opening 2 IndexWriters on same
> Directory - I do not get IOException ...
>
>
> Hi Koji,
>
> I might not agree with you ... I did the following: I create new function
> which invokes FSDirectory.getDirectory(dir,true) twice . The result I
> compare (fs1==fs2). In my situation the (fs1==fs2) = true. So even in my
> situation, i have two references to the <b>same</b> Directory object.
>
> If I look in the FSDirectory constructor, this is also what happens. The
> FSDirectory.getDirectory(path,create) method invokese the
> FSDirectory.getDirectory(file, create) method and this checks the
> hashtable
> DIRECTORIES whether an object is already created for that
> directory. If so,
> it returns the existing reference, if not it creates a new
> directory object.
>
> So, I still think the difference is in the closeDir parameter difference
> (true/false).
>
> Do you agree with me sofar?
>
> New function added to java class:
> private void openDirectoryTwiceAndCompareObjects()
> {
> try
> {
> Directory fs1 = FSDirectory.getDirectory(this.indexDir,
> true);
> Directory fs2 = FSDirectory.getDirectory(this.indexDir, true);
> if (fs1 == fs2) {
> System.out.println("Two references to identical object");
> }
> else {
> System.out.println("Two different directory objects");
> }
> }
> catch (IOException e)
> {
> System.out.println("Error in creating directory : " +
> e.getMessage());
> }
> }
>
> And change main to invoke this method:
> public static void main(String[] args)
> {
> TestIndexWriter tw = new TestIndexWriter();
> tw.openDirectoryTwiceAndCompareObjects();
> //tw.testOpenIndexWriters(true, true);
> //tw.testOpenIndexWriters(true, false);
> }
>
>
>
>
>
>
>
> On 1/10/06, Koji Sekiguchi <ko...@m4.dion.ne.jp> wrote:
> >
> > Hi Dick,
> >
> > > I only see one difference in the constructor of the IndexWriter class:
> > > "closeDir" is 'true' in my scenario and 'false' in your scenario. What
> > is
> > > reason for this difference? And if there is a valid reason,
> it might be
> > > useful to add this to the javaDoc of the IndexWriter class.
> >
> > The important difference is:
> >
> > > > private Directory getDirectory() throws IOException {
> > > > if( directory == null ){
> > > > directory = FSDirectory.getDirectory( indexDir, true );
> > > > }
> > > > return directory;
> > > > }
> >
> > this method instantiates FSDirectory object only once.
> > But in your original code, calling IndexWriter( String,Analyzer,boolean)
> > version with create=true:
> >
> > > public IndexWriter(String path, Analyzer a, boolean create)
> > > throws IOException {
> > > this(FSDirectory.getDirectory(path, create), a,
> > create,
> > > true);
> > > }
> >
> > it always makes new FSDirectory object.
> >
> > regards,
> >
> > Koji
> >
> >
> >
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Basic question on opening 2 IndexWriters on same Directory - I do not get IOException ...
Posted by Dick de Jong <dd...@gmail.com>.
Hi Koji,
I might not agree with you ... I did the following: I create new function
which invokes FSDirectory.getDirectory(dir,true) twice . The result I
compare (fs1==fs2). In my situation the (fs1==fs2) = true. So even in my
situation, i have two references to the <b>same</b> Directory object.
If I look in the FSDirectory constructor, this is also what happens. The
FSDirectory.getDirectory(path,create) method invokese the
FSDirectory.getDirectory(file, create) method and this checks the hashtable
DIRECTORIES whether an object is already created for that directory. If so,
it returns the existing reference, if not it creates a new directory object.
So, I still think the difference is in the closeDir parameter difference
(true/false).
Do you agree with me sofar?
New function added to java class:
private void openDirectoryTwiceAndCompareObjects()
{
try
{
Directory fs1 = FSDirectory.getDirectory(this.indexDir,
true);
Directory fs2 = FSDirectory.getDirectory(this.indexDir, true);
if (fs1 == fs2) {
System.out.println("Two references to identical object");
}
else {
System.out.println("Two different directory objects");
}
}
catch (IOException e)
{
System.out.println("Error in creating directory : " +
e.getMessage());
}
}
And change main to invoke this method:
public static void main(String[] args)
{
TestIndexWriter tw = new TestIndexWriter();
tw.openDirectoryTwiceAndCompareObjects();
//tw.testOpenIndexWriters(true, true);
//tw.testOpenIndexWriters(true, false);
}
On 1/10/06, Koji Sekiguchi <ko...@m4.dion.ne.jp> wrote:
>
> Hi Dick,
>
> > I only see one difference in the constructor of the IndexWriter class:
> > "closeDir" is 'true' in my scenario and 'false' in your scenario. What
> is
> > reason for this difference? And if there is a valid reason, it might be
> > useful to add this to the javaDoc of the IndexWriter class.
>
> The important difference is:
>
> > > private Directory getDirectory() throws IOException {
> > > if( directory == null ){
> > > directory = FSDirectory.getDirectory( indexDir, true );
> > > }
> > > return directory;
> > > }
>
> this method instantiates FSDirectory object only once.
> But in your original code, calling IndexWriter( String,Analyzer,boolean)
> version with create=true:
>
> > public IndexWriter(String path, Analyzer a, boolean create)
> > throws IOException {
> > this(FSDirectory.getDirectory(path, create), a,
> create,
> > true);
> > }
>
> it always makes new FSDirectory object.
>
> regards,
>
> Koji
>
>
>
>
>
RE: Basic question on opening 2 IndexWriters on same Directory - I do not get IOException ...
Posted by Koji Sekiguchi <ko...@m4.dion.ne.jp>.
Hi Dick,
> I only see one difference in the constructor of the IndexWriter class:
> "closeDir" is 'true' in my scenario and 'false' in your scenario. What is
> reason for this difference? And if there is a valid reason, it might be
> useful to add this to the javaDoc of the IndexWriter class.
The important difference is:
> > private Directory getDirectory() throws IOException {
> > if( directory == null ){
> > directory = FSDirectory.getDirectory( indexDir, true );
> > }
> > return directory;
> > }
this method instantiates FSDirectory object only once.
But in your original code, calling IndexWriter( String,Analyzer,boolean)
version with create=true:
> public IndexWriter(String path, Analyzer a, boolean create)
> throws IOException {
> this(FSDirectory.getDirectory(path, create), a, create,
> true);
> }
it always makes new FSDirectory object.
regards,
Koji
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Basic question on opening 2 IndexWriters on same Directory - I do not get IOException ...
Posted by Dick de Jong <dd...@gmail.com>.
Hi Koji,
Thanks for your help, would not have found that soon ... You are right, if I
apply your code, the IOException comes.
I only see one difference in the constructor of the IndexWriter class:
"closeDir" is 'true' in my scenario and 'false' in your scenario. What is
reason for this difference? And if there is a valid reason, it might be
useful to add this to the javaDoc of the IndexWriter class.
See below the difference in the constructor:
When passing a Directory object:
public IndexWriter(Directory d, Analyzer a, boolean create)
throws IOException {
this(d, a, create, false);
}
When passing a String path:
public IndexWriter(String path, Analyzer a, boolean create)
throws IOException {
this(FSDirectory.getDirectory(path, create), a, create,
true);
}
Dick
On 1/10/06, Koji Sekiguchi <ko...@m4.dion.ne.jp> wrote:
>
> Hello Dick,
>
> Why you couldn't get an IOException when obtaining the second
> writer because you used IndexWriter(String,Analyzer,boolean) version
> constructor. Try IndexWriter(Directory,Analyzer,boolean) version instead:
>
> To do it, add the following code on your program:
>
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.FSDirectory;
>
> private Directory directory;
>
> private Directory getDirectory() throws IOException {
> if( directory == null ){
> directory = FSDirectory.getDirectory( indexDir, true );
> }
> return directory;
> }
>
> And use the following method:
>
> private IndexWriter getWriter(String name, boolean createIndex)
> throws
> IOException
> {
> System.out.println("Opening writer " + name + " on: " +
> this.indexDir);
> //return new IndexWriter(this.indexDir, this.analyzer,
> createIndex);
> return new IndexWriter(getDirectory(), this.analyzer,
> createIndex);
> }
>
> If IndexWriter(path,Analyzer,true) is used,
> IndexWriter calls FSDirectory.getDirectory(path,true) and
> FSDirectory overwrites the existing lock file.
>
> Hope this helps,
>
> Koji
>
> > -----Original Message-----
> > From: Dick de Jong [mailto:ddjong67@gmail.com]
> > Sent: Sunday, January 08, 2006 1:54 PM
> > To: java-user@lucene.apache.org
> > Subject: Basic question on opening 2 IndexWriters on same
> > Directory - I do not get IOException ...
> >
> >
> > Question:
> > Run attached java class and see the also attached when I ran it.
> > I assumed the second writer should get an IOException in
> > getWriter (in
> > first TESTCASE).
> > However this does not happen! This only happens when I open both
> > writers with flag 'createIndex'=true (see
> > also second test case).
> >
> > Is this a misunderstanding from my side or what is this?
> >
> > ENVIRONMENT:
> > Windows Professional XP SP2
> > Lucene 1.4 (java)
> >
> >
> > OUTPUT OF MAIN METHOD:
> > =======================
> > START TESTCASE
> > Test open 2 index writers on index c:\temp\indexdir
> > writer 1 uses createIndex =true
> > writer 2 uses createIndex =true // why not an
> > IOException here ??
> > Opening writer w1 on: c:\temp\indexdir
> > Writing document to index using writer :w1
> > Opening writer w2 on: c:\temp\indexdir
> > Writing document to index using writer :w2
> > Writing document to index using writer :w1
> > Closing writer w1
> > Closing writer w2
> > END of test case ...
> > =======================
> > =======================
> > START TESTCASE
> > Test open 2 index writers on index c:\temp\indexdir
> > writer 1 uses createIndex =true
> > writer 2 uses createIndex =false
> > Opening writer w1 on: c:\temp\indexdir
> > Writing document to index using writer :w1
> > Opening writer w2 on: c:\temp\indexdir
> > In open writer caught IOException:
> > Lock obtain timed out:
> > Lock@C:\DOCUME~1\ddjong\LOCALS~1\Temp\lucene-
> > ea51d1f7bb672168eda3ee90698e936d-write.lock
> > Closing writer w1
> > END of test case ...
> > =======================
> >
> > USED JAVA class (src):
> > ==================================
> >
> > import java.io.IOException;
> >
> > import org.apache.lucene.analysis.Analyzer;
> > import org.apache.lucene.analysis.standard.StandardAnalyzer;
> > import org.apache.lucene.index.IndexWriter;
> > import org.apache.lucene.document.Document;
> > import org.apache.lucene.document.Field;
> >
> > public class TestIndexWriter
> > {
> >
> > private IndexWriter w1;
> > private IndexWriter w2;
> > private String indexDir;
> >
> > private Analyzer analyzer;
> > private Document document;
> >
> > private TestIndexWriter()
> > {
> > // just create dummy document for test purposes, document is
> > reused.
> > this.document = new Document();
> > this.document.add(Field.Keyword("fname", "fvalue"));
> >
> > // indexDirectory Folder to be used
> > this.indexDir = "c:\\temp\\indexdir";
> >
> > this.analyzer = new StandardAnalyzer();
> > }
> >
> > private IndexWriter getWriter(String name, boolean
> > createIndex) throws
> > IOException
> > {
> > System.out.println("Opening writer " + name + " on: " +
> > this.indexDir);
> > return new IndexWriter(this.indexDir, this.analyzer,
> > createIndex);
> > }
> >
> > private void closeWriter(String name, IndexWriter w)
> > {
> > try {
> > if (w != null) {
> > System.out.println("Closing writer " + name);
> > w.close();
> > }
> > }
> > catch (IOException e) {
> > System.out.println("Don't bubble error in close
> > writer; error:
> > \n" + e.getMessage());
> > }
> > }
> >
> > private void addDocument(String name, IndexWriter w) throws
> > IOException
> > {
> > System.out.println("Writing document to index using writer :" +
> > name);
> > w.addDocument(this.document);
> > }
> >
> >
> > public void testOpenIndexWriters(boolean w1_create, boolean
> > w2_create)
> > {
> > System.out.println("=======================");
> > System.out.println("START TESTCASE");
> > System.out.println("Test open 2 index writers on index " +
> > this.indexDir);
> >
> > System.out.println(" \twriter 1 uses createIndex =" +
> w1_create);
> > System.out.println(" \twriter 2 uses createIndex =" +
> w2_create);
> >
> > this.w1 = this.w2 = null;
> > try
> > {
> > w1 = this.getWriter("w1", w1_create);
> > addDocument("w1", w1);
> > w2 = this.getWriter("w2", w2_create);
> > addDocument("w2", w2);
> > addDocument("w1", w1);
> > }
> > catch (IOException e)
> > {
> > System.out.println("In open writer caught IOException:");
> > System.out.println("\t" + e.getMessage());
> > }
> > finally
> > {
> > this.closeWriter("w1", w1);
> > this.closeWriter("w2", w2);
> > System.out.println("END of test case ...");
> > System.out.println("=======================");
> > }
> > }
> >
> > public static void main(String[] args)
> > {
> > TestIndexWriter tw = new TestIndexWriter();
> > tw.testOpenIndexWriters(true, true);
> > tw.testOpenIndexWriters(true, false);
> > }
> > }
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Basic question on opening 2 IndexWriters on same Directory - I do not get IOException ...
Posted by Koji Sekiguchi <ko...@m4.dion.ne.jp>.
Hello Dick,
Why you couldn't get an IOException when obtaining the second
writer because you used IndexWriter(String,Analyzer,boolean) version
constructor. Try IndexWriter(Directory,Analyzer,boolean) version instead:
To do it, add the following code on your program:
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
private Directory directory;
private Directory getDirectory() throws IOException {
if( directory == null ){
directory = FSDirectory.getDirectory( indexDir, true );
}
return directory;
}
And use the following method:
private IndexWriter getWriter(String name, boolean createIndex) throws
IOException
{
System.out.println("Opening writer " + name + " on: " +
this.indexDir);
//return new IndexWriter(this.indexDir, this.analyzer,
createIndex);
return new IndexWriter(getDirectory(), this.analyzer, createIndex);
}
If IndexWriter(path,Analyzer,true) is used,
IndexWriter calls FSDirectory.getDirectory(path,true) and
FSDirectory overwrites the existing lock file.
Hope this helps,
Koji
> -----Original Message-----
> From: Dick de Jong [mailto:ddjong67@gmail.com]
> Sent: Sunday, January 08, 2006 1:54 PM
> To: java-user@lucene.apache.org
> Subject: Basic question on opening 2 IndexWriters on same
> Directory - I do not get IOException ...
>
>
> Question:
> Run attached java class and see the also attached when I ran it.
> I assumed the second writer should get an IOException in
> getWriter (in
> first TESTCASE).
> However this does not happen! This only happens when I open both
> writers with flag 'createIndex'=true (see
> also second test case).
>
> Is this a misunderstanding from my side or what is this?
>
> ENVIRONMENT:
> Windows Professional XP SP2
> Lucene 1.4 (java)
>
>
> OUTPUT OF MAIN METHOD:
> =======================
> START TESTCASE
> Test open 2 index writers on index c:\temp\indexdir
> writer 1 uses createIndex =true
> writer 2 uses createIndex =true // why not an
> IOException here ??
> Opening writer w1 on: c:\temp\indexdir
> Writing document to index using writer :w1
> Opening writer w2 on: c:\temp\indexdir
> Writing document to index using writer :w2
> Writing document to index using writer :w1
> Closing writer w1
> Closing writer w2
> END of test case ...
> =======================
> =======================
> START TESTCASE
> Test open 2 index writers on index c:\temp\indexdir
> writer 1 uses createIndex =true
> writer 2 uses createIndex =false
> Opening writer w1 on: c:\temp\indexdir
> Writing document to index using writer :w1
> Opening writer w2 on: c:\temp\indexdir
> In open writer caught IOException:
> Lock obtain timed out:
> Lock@C:\DOCUME~1\ddjong\LOCALS~1\Temp\lucene-
> ea51d1f7bb672168eda3ee90698e936d-write.lock
> Closing writer w1
> END of test case ...
> =======================
>
> USED JAVA class (src):
> ==================================
>
> import java.io.IOException;
>
> import org.apache.lucene.analysis.Analyzer;
> import org.apache.lucene.analysis.standard.StandardAnalyzer;
> import org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.document.Document;
> import org.apache.lucene.document.Field;
>
> public class TestIndexWriter
> {
>
> private IndexWriter w1;
> private IndexWriter w2;
> private String indexDir;
>
> private Analyzer analyzer;
> private Document document;
>
> private TestIndexWriter()
> {
> // just create dummy document for test purposes, document is
> reused.
> this.document = new Document();
> this.document.add(Field.Keyword("fname", "fvalue"));
>
> // indexDirectory Folder to be used
> this.indexDir = "c:\\temp\\indexdir";
>
> this.analyzer = new StandardAnalyzer();
> }
>
> private IndexWriter getWriter(String name, boolean
> createIndex) throws
> IOException
> {
> System.out.println("Opening writer " + name + " on: " +
> this.indexDir);
> return new IndexWriter(this.indexDir, this.analyzer,
> createIndex);
> }
>
> private void closeWriter(String name, IndexWriter w)
> {
> try {
> if (w != null) {
> System.out.println("Closing writer " + name);
> w.close();
> }
> }
> catch (IOException e) {
> System.out.println("Don't bubble error in close
> writer; error:
> \n" + e.getMessage());
> }
> }
>
> private void addDocument(String name, IndexWriter w) throws
> IOException
> {
> System.out.println("Writing document to index using writer :" +
> name);
> w.addDocument(this.document);
> }
>
>
> public void testOpenIndexWriters(boolean w1_create, boolean
> w2_create)
> {
> System.out.println("=======================");
> System.out.println("START TESTCASE");
> System.out.println("Test open 2 index writers on index " +
> this.indexDir);
>
> System.out.println(" \twriter 1 uses createIndex =" + w1_create);
> System.out.println(" \twriter 2 uses createIndex =" + w2_create);
>
> this.w1 = this.w2 = null;
> try
> {
> w1 = this.getWriter("w1", w1_create);
> addDocument("w1", w1);
> w2 = this.getWriter("w2", w2_create);
> addDocument("w2", w2);
> addDocument("w1", w1);
> }
> catch (IOException e)
> {
> System.out.println("In open writer caught IOException:");
> System.out.println("\t" + e.getMessage());
> }
> finally
> {
> this.closeWriter("w1", w1);
> this.closeWriter("w2", w2);
> System.out.println("END of test case ...");
> System.out.println("=======================");
> }
> }
>
> public static void main(String[] args)
> {
> TestIndexWriter tw = new TestIndexWriter();
> tw.testOpenIndexWriters(true, true);
> tw.testOpenIndexWriters(true, false);
> }
> }
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org