You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2021/02/08 17:32:00 UTC
[jira] [Updated] (NIFI-8210) If Provenance Repository ages off a
Lucene Index while it's being searched, can lead to index corruption
[ https://issues.apache.org/jira/browse/NIFI-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Payne updated NIFI-8210:
-----------------------------
Description:
When all events have been aged off from a Provenance Lucene Index, the lucene index can be deleted. However, before deleting it, the repository needs to ensure that there are no active readers or writers. Currently, the repository only ensures that there are no active writers. As a result, we can have an active Index Reader/Index Searcher and still delete the index directory. When this happens, we end up with an error similar to:
{code:java}
org.apache.lucene.index.IndexNotFoundException: no segments* file found in MMapDirectory@/Users/mpayne/devel/nifi/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/target/6e442c6d-14f3-48bb-9500-534c136942aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@161b062a: files: []
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
at org.apache.nifi.provenance.lucene.StandardIndexManager.borrowIndexSearcher(StandardIndexManager.java:128)
...{code}
We need to ensure that we account for any Readers/Searchers that are active also
was:
When all events have been aged off from a Provenance Lucene Index, the lucene index can be deleted. However, before deleting it, the repository needs to ensure that there are no active readers or writers. Currently, the repository only ensures that there are no active writers. As a result, we can have an active Index Reader/Index Searcher and still delete the index directory. When this happens, we end up with an error similar to:
{code:java}
org.apache.lucene.index.IndexNotFoundException: no segments* file found in MMapDirectory@/Users/mpayne/devel/nifi/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/target/6e442c6d-14f3-48bb-9500-534c136942aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@161b062a: files: []org.apache.lucene.index.IndexNotFoundException: no segments* file found in MMapDirectory@/Users/mpayne/devel/nifi/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/target/6e442c6d-14f3-48bb-9500-534c136942aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@161b062a: files: []
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) at org.apache.nifi.provenance.lucene.StandardIndexManager.borrowIndexSearcher(StandardIndexManager.java:128) {code}
We need to ensure that we account for any Readers/Searchers that are active also
> If Provenance Repository ages off a Lucene Index while it's being searched, can lead to index corruption
> --------------------------------------------------------------------------------------------------------
>
> Key: NIFI-8210
> URL: https://issues.apache.org/jira/browse/NIFI-8210
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> When all events have been aged off from a Provenance Lucene Index, the lucene index can be deleted. However, before deleting it, the repository needs to ensure that there are no active readers or writers. Currently, the repository only ensures that there are no active writers. As a result, we can have an active Index Reader/Index Searcher and still delete the index directory. When this happens, we end up with an error similar to:
> {code:java}
> org.apache.lucene.index.IndexNotFoundException: no segments* file found in MMapDirectory@/Users/mpayne/devel/nifi/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/target/6e442c6d-14f3-48bb-9500-534c136942aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@161b062a: files: []
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675)
> at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
> at org.apache.nifi.provenance.lucene.StandardIndexManager.borrowIndexSearcher(StandardIndexManager.java:128)
> ...{code}
> We need to ensure that we account for any Readers/Searchers that are active also
--
This message was sent by Atlassian Jira
(v8.3.4#803005)