You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Jianbao (Jim) Tao" <ji...@gmail.com> on 2016/05/22 17:20:07 UTC
Leftover write.lock file
Hi,
I am new to lucene. I was following the example, IndexFiles, from the
lucene demo package. However, one thing I find surpring is that the
write.lock file is left over, even when the IndexWriter#close() method is
called. Below are details.
Environment:
OS X Yosemite
java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)
lucene dependencies:
<lucene.version>6.0.0</lucene.version>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>${lucene.version}</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-analyzers-common</artifactId>
<version>${lucene.version}</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-queryparser</artifactId>
<version>${lucene.version}</version>
</dependency>
My Indexer and IndexerTest classes:
```
@Slf4j
public class Indexer implements AutoCloseable {
private IndexWriter writer;
public Indexer(String indexDir) throws IOException {
Directory dir = FSDirectory.open(Paths.get(indexDir));
Analyzer analyzer = new StandardAnalyzer();
IndexWriterConfig iwc = new IndexWriterConfig(analyzer);
iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
writer = new IndexWriter(dir, iwc);
}
@Override
public void close() throws IOException {
writer.close();
log.info("Indexer is successfully closed.");
}
public int index(String docDir) throws IOException {
File[] files = new File(docDir).listFiles();
for (File f : files) {
if (f.isDirectory() || f.isHidden()) continue;
indexFile(f.toPath());
}
return writer.numDocs();
}
private void indexFile(Path f) throws IOException {
log.info("Indexing {}", f);
try (InputStream stream = Files.newInputStream(f)){
Document doc = new Document();
doc.add(new StringField("path", f.toString(), Field.Store.YES));
doc.add(new LongPoint("modified",
Files.getLastModifiedTime(f).toMillis()));
doc.add(new TextField("contents", new BufferedReader(new
InputStreamReader(stream, StandardCharsets.UTF_8))));
writer.updateDocument(new Term("path", f.toString()), doc);
}
}
}
@Slf4j
public class IndexerTest {
@Test
public void testIndex() throws Exception {
String indexDir = Shared.NOTES_INDEX_DIR; // Shared contains string
constants.
try (Indexer indexer = new Indexer(indexDir)) {
int indexedDocs = indexer.index(Shared.NOTES_DIR);
log.info("Indexed {} documents in {}", indexedDocs,
Shared.NOTES_DIR);
}
}
}
```
After running the testIndex() method, I can see this log:
2016-05-22 09:50:17,532 INFO [main] Indexer - Indexer is successfully
closed.
This indicates that the IndexWriter is successfully closed. However, I am
surprised to see that the write.lock still remains in the index directory.
I did a little digging. It appears that the `NativeFSLockFactory` is used
in my case. So, is it by design that the user is supposed to remove the
write.lock file manually when NativeFSLockFactory is used? I'd expect this
to be done automatically, but I guess there are probably important reasons
behind this behavior. In any case, I am just curious to know why.
AW: Leftover write.lock file
Posted by Oliver Kaleske <Ol...@ptvgroup.com>.
Hi,
> Never mind, folks. Just found an answer:
> http://www.gossamer-threads.com/lists/lucene/java-user/110708
> > ...
> > I am new to lucene. I was following the example, IndexFiles, from the
> > lucene demo package. However, one thing I find surpring is that the
> > write.lock file is left over, even when the IndexWriter#close() method
> > is called. Below are details.
> > ...
The documentation (codecs package summary), however, claims the following:
"When this file is present, a writer is currently modifying the index (adding or removing documents)."
(see https://lucene.apache.org/core/6_0_1/core/org/apache/lucene/codecs/lucene60/package-summary.html )
which, judging from the code and the above post, is wrong, and perhaps could be corrected...? (Thanks in advance.)
Best regards,
Oliver
Dipl.-Phys. Oliver Kaleske
PTV GROUP
Logistics Geographic Components
Haid-und-Neu-Straße 15
76131 Karlsruhe, Germany
Phone +49 (0) 721 9651-8211
oliver.kaleske@ptvgroup.com
www.ptvgroup.com
PTV Planung Transport Verkehr AG
Headquarters: Karlsruhe
Executive Board: Vincent Kobesen (CEO), Michel Zweers (CFO)
Chairman of the Supervisory Board: Dr. h.c. Frank-Jürgen Weise
Commercial Register (HRB-Nr): 109262
Local Court: Mannheim
Re: Leftover write.lock file
Posted by "Jianbao (Jim) Tao" <ji...@gmail.com>.
Never mind, folks. Just found an answer:
http://www.gossamer-threads.com/lists/lucene/java-user/110708
On Sun, May 22, 2016 at 10:20 AM, Jianbao (Jim) Tao <ji...@gmail.com>
wrote:
> Hi,
>
> I am new to lucene. I was following the example, IndexFiles, from the
> lucene demo package. However, one thing I find surpring is that the
> write.lock file is left over, even when the IndexWriter#close() method is
> called. Below are details.
>
> Environment:
> OS X Yosemite
> java version "1.8.0_72"
> Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
> Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)
>
> lucene dependencies:
> <lucene.version>6.0.0</lucene.version>
> <dependency>
> <groupId>org.apache.lucene</groupId>
> <artifactId>lucene-core</artifactId>
> <version>${lucene.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.lucene</groupId>
> <artifactId>lucene-analyzers-common</artifactId>
> <version>${lucene.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.lucene</groupId>
> <artifactId>lucene-queryparser</artifactId>
> <version>${lucene.version}</version>
> </dependency>
>
> My Indexer and IndexerTest classes:
> ```
> @Slf4j
> public class Indexer implements AutoCloseable {
> private IndexWriter writer;
>
> public Indexer(String indexDir) throws IOException {
> Directory dir = FSDirectory.open(Paths.get(indexDir));
> Analyzer analyzer = new StandardAnalyzer();
> IndexWriterConfig iwc = new IndexWriterConfig(analyzer);
> iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
> writer = new IndexWriter(dir, iwc);
> }
>
> @Override
> public void close() throws IOException {
> writer.close();
> log.info("Indexer is successfully closed.");
> }
>
> public int index(String docDir) throws IOException {
> File[] files = new File(docDir).listFiles();
> for (File f : files) {
> if (f.isDirectory() || f.isHidden()) continue;
> indexFile(f.toPath());
> }
> return writer.numDocs();
> }
>
> private void indexFile(Path f) throws IOException {
> log.info("Indexing {}", f);
> try (InputStream stream = Files.newInputStream(f)){
> Document doc = new Document();
>
> doc.add(new StringField("path", f.toString(),
> Field.Store.YES));
> doc.add(new LongPoint("modified",
> Files.getLastModifiedTime(f).toMillis()));
> doc.add(new TextField("contents", new BufferedReader(new
> InputStreamReader(stream, StandardCharsets.UTF_8))));
> writer.updateDocument(new Term("path", f.toString()), doc);
> }
> }
> }
>
>
> @Slf4j
> public class IndexerTest {
> @Test
> public void testIndex() throws Exception {
> String indexDir = Shared.NOTES_INDEX_DIR; // Shared contains
> string constants.
> try (Indexer indexer = new Indexer(indexDir)) {
> int indexedDocs = indexer.index(Shared.NOTES_DIR);
> log.info("Indexed {} documents in {}", indexedDocs,
> Shared.NOTES_DIR);
> }
> }
> }
> ```
> After running the testIndex() method, I can see this log:
> 2016-05-22 09:50:17,532 INFO [main] Indexer - Indexer is successfully
> closed.
> This indicates that the IndexWriter is successfully closed. However, I am
> surprised to see that the write.lock still remains in the index directory.
> I did a little digging. It appears that the `NativeFSLockFactory` is used
> in my case. So, is it by design that the user is supposed to remove the
> write.lock file manually when NativeFSLockFactory is used? I'd expect this
> to be done automatically, but I guess there are probably important reasons
> behind this behavior. In any case, I am just curious to know why.
>