You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Alex Falca (JIRA)" <ji...@apache.org> on 2008/02/26 12:03:51 UTC
[jira] Created: (LUCENE-1192) FileNotFound exception on adding a
RAMDirectory to an IndexWriter
FileNotFound exception on adding a RAMDirectory to an IndexWriter
-----------------------------------------------------------------
Key: LUCENE-1192
URL: https://issues.apache.org/jira/browse/LUCENE-1192
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 2.3.1
Reporter: Alex Falca
Hi,
I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
{noformat}
java.io.FileNotFoundException: _8.fnm
at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
{noformat}
Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
1. First documents are added to and index which use a RAMDirectory as a storage
2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
Sample code:
{code}
IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
mainIndexWriter.setMergeFactor(10);
mainIndexWriter.setUseCompoundFile(false);
ramIndexWriter.setUseCompoundFile(false);
// Here some code for adding documents to RAM index writer
...................................................................................................................
// method where I'm flushing from memory to disk and getting exception
ramIndexWriter.flush();
mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
ramIndexWriter.close();
ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
ramIndexWriter.setUseCompoundFile(false);
System.gc();
{code}
I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
If you need details on environment(i.e. Java version, settings, hardware details etc.) - please let me know.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1192) FileNotFound exception on adding a
RAMDirectory to an IndexWriter
Posted by "Alex Falca (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Falca updated LUCENE-1192:
-------------------------------
Description:
Hi,
I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
{noformat}
java.io.FileNotFoundException: _8.fnm
at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
{noformat}
Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
1. First documents are added to and index which use a RAMDirectory as a storage
2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
Sample code:
{code}
IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
mainIndexWriter.setMergeFactor(10);
mainIndexWriter.setUseCompoundFile(false);
ramIndexWriter.setUseCompoundFile(false);
// Here some code for adding documents to RAM index writer
...................................................................................................................
// method where I'm flushing from memory to disk and getting exception
ramIndexWriter.flush();
mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
ramIndexWriter.close();
ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
ramIndexWriter.setUseCompoundFile(false);
System.gc();
{code}
I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
was:
Hi,
I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
{noformat}
java.io.FileNotFoundException: _8.fnm
at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
{noformat}
Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
1. First documents are added to and index which use a RAMDirectory as a storage
2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
Sample code:
{code}
IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
mainIndexWriter.setMergeFactor(10);
mainIndexWriter.setUseCompoundFile(false);
ramIndexWriter.setUseCompoundFile(false);
// Here some code for adding documents to RAM index writer
...................................................................................................................
// method where I'm flushing from memory to disk and getting exception
ramIndexWriter.flush();
mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
ramIndexWriter.close();
ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
ramIndexWriter.setUseCompoundFile(false);
System.gc();
{code}
I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
If you need details on environment(i.e. Java version, settings, hardware details etc.) - please let me know.
Environment:
|| Description || Value ||
| java -version | {noformat}java version "1.5.0_14"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
{noformat} |
| JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
-XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
| uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
| CPU | Dual Core AMD Opteron(tm) Processor 280 |
| Total available memory | 8G |
Updated details on environment
> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1192
> URL: https://issues.apache.org/jira/browse/LUCENE-1192
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3.1
> Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
> Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
> at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
> at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1192) FileNotFound exception on adding a
RAMDirectory to an IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1192.
----------------------------------------
Resolution: Invalid
> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1192
> URL: https://issues.apache.org/jira/browse/LUCENE-1192
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3.1
> Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
> Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
> at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
> at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1192) FileNotFound exception on adding a
RAMDirectory to an IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572449#action_12572449 ]
Michael McCandless commented on LUCENE-1192:
--------------------------------------------
This actually isn't quite a valid use of Lucene's APIs, because the
ramIndexWriter is actively changing the files in the RAMDirectory.
This worked in past versions of Lucene because merges were done
synchronously. Now, with background merges by default, a merge can
complete and commit changes to the RAMDirectory while
addIndexesNoOptimize is running. The addIndexesNoOptimize method
requires a "static" Directory, where no writer is actively making
changes.
There are a few ways you could get this to work:
* Sync all concurrent merges before calling addIndexesNoOptimize, by
calling ((ConcurrentMergeScheduler) ramIndexWriter.getMergeScheduler()).sync()
* Switch back to SerialMergeScheduler
* Close the ramIndexWriter before passing its directory to
addIndexesNoOptimize. In this case you need to separately hold
the underlying RAMDirectory since you can't call
IndexWriter.getDirectory after close.
> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1192
> URL: https://issues.apache.org/jira/browse/LUCENE-1192
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3.1
> Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
> Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
> at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
> at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1192) FileNotFound exception on adding a
RAMDirectory to an IndexWriter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572495#action_12572495 ]
Michael McCandless commented on LUCENE-1192:
--------------------------------------------
You're welcome!
> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1192
> URL: https://issues.apache.org/jira/browse/LUCENE-1192
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3.1
> Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
> Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
> at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
> at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Closed: (LUCENE-1192) FileNotFound exception on adding a
RAMDirectory to an IndexWriter
Posted by "Alex Falca (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Falca closed LUCENE-1192.
------------------------------
Tried last option:
{quote}
* Close the ramIndexWriter before passing its directory to
addIndexesNoOptimize. In this case you need to separately hold
the underlying RAMDirectory since you can't call
IndexWriter.getDirectory after close.
{quote}
everything works fine :) Got a 2x increase in speed of indexing.
Thank you again!
Closing ticket.
> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1192
> URL: https://issues.apache.org/jira/browse/LUCENE-1192
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3.1
> Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
> Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
> at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
> at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1192) FileNotFound exception on adding a
RAMDirectory to an IndexWriter
Posted by "Alex Falca (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572457#action_12572457 ]
Alex Falca commented on LUCENE-1192:
------------------------------------
Thanks for the explanations and proposed ways to solve issue:) I'll try one of them today.
> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
> Key: LUCENE-1192
> URL: https://issues.apache.org/jira/browse/LUCENE-1192
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3.1
> Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime
> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
> Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
> at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
> at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory is being added to another index writer, which use a FSDirectory as a storage.
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in LUCENE-1175, but need a confirmation on that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org