You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Jannick (Jira)" <ji...@apache.org> on 2022/06/07 11:34:00 UTC
[jira] [Comment Edited] (NIFI-7611) NiFi fails to index provenance events
[ https://issues.apache.org/jira/browse/NIFI-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550947#comment-17550947 ]
Jannick edited comment on NIFI-7611 at 6/7/22 11:33 AM:
--------------------------------------------------------
Dear all, we ran into the same problem. The problem is not inherent in Nifi itself but in the antiviral software running on the same host as Nifi. Apparently, the provenance indexing processes is marked as suspicious and the corresponding directory is locked for writes of the Nifi process. I cannot tell you the actual antiviral software that we are using nor its version. However, I can tell you that an update of the antiviral software to the newest version solved the problem for us. The newest version employs better filters to flag suspicious activity.
was (Author: JIRAUSER290606):
Dear all, we ran into the same problem. The problem is not inherent in Nifi itself but in the antiviral software. Apparently, the provenance indexing processes is marked as suspcious and the corresponding directory is locked for writes of the Nifi process. I cannot tell you the actual antiviral software that we are using nor its version. However, I can tell you that an update of the antiviral software to the newest version solved the problem for us. The newest version employs better filters to flag suspicious activity.
> NiFi fails to index provenance events
> -------------------------------------
>
> Key: NIFI-7611
> URL: https://issues.apache.org/jira/browse/NIFI-7611
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.11.4
> Environment: Microsoft Windows Server 2016 Standard - Intel Xeon Gold 6140 CPU @ 2,30 GHz 8 processors, 32 GB RAM, total disk space 877 GB
> Reporter: Michal W
> Priority: Critical
> Labels: index, java.nio.file.FileSystemException, provenance, repository
>
> Getting error "failed to index provenance events". Nifi.app log displays following information:
> 2020-07-08 09:00:00,406 ERROR [Index Provenance Events-4] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
> at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1281)
> at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1257)
> at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.nio.file.FileSystemException: E:\nifi-storage\provenance_repository\lucene-8-index-1593163985970\_11r.cfe: The process cannot access the file because it is being used by another process.
>
> at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
> at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
> at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
> at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:120)
> at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
> at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
> at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:105)
> at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69)
> at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:100)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4396)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4054)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
>
> Logs eventually grow over time and fill up the partition.
>
> Configuration related to provenance repository:
>
> # Provenance Repository Properties
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
> nifi.provenance.repository.debug.frequency=1_000_000
> nifi.provenance.repository.encryption.key.provider.implementation=
> nifi.provenance.repository.encryption.key.provider.location=
> nifi.provenance.repository.encryption.key.id=
> nifi.provenance.repository.encryption.key=
>
> # Persistent Provenance Repository Properties
> nifi.provenance.repository.directory.default=E:\\nifi-storage\\provenance_repository
> nifi.provenance.repository.directory.content1=F:\\nifi-storage\\provenance_repository
> nifi.provenance.repository.max.storage.time=24 hours
> # nifi.provenance.repository.max.storage.size=1 GB
> nifi.provenance.repository.max.storage.size=8 GB
> nifi.provenance.repository.rollover.time=30 secs
> # nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.rollover.size=1 GB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=4
> #default: nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.compress.on.rollover=false
> nifi.provenance.repository.always.sync=false
> # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
> # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
> nifi.provenance.repository.indexed.attributes=
> # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
> # but should provide better performance
> # nifi.provenance.repository.index.shard.size=500 MB
> nifi.provenance.repository.index.shard.size=4 GB
>
> # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
> # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=2
>
> # Volatile Provenance Respository Properties
> nifi.provenance.repository.buffer.size=100000
--
This message was sent by Atlassian Jira
(v8.20.7#820007)