You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Chirthani, Deepak Reddy" <C-...@charter.com> on 2022/07/17 22:56:29 UTC
RE: [EXTERNAL] Re: Issue with SiteToSiteProvenanceReportingTask 1.15.3

Joe,

Here are some more details.

Logs:

Please see the log file attached. Let me know if you need more.

OS:

We have our instance running on a 3 node cluster. Each machine has 8 CPUs, 32 GB RAM and is a RedHat Linux 7 sitting at our enterprise datacenters.

Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
Architecture: amd 64
Kernel: Linux 3.10.0-1160.53.1.el7.x86_64
NiFi Version: 1.15.3
Java Version:1.8.0_322

Regards

[image005]
Deepak Reddy | Data Engineer
IT Centers of Excellence
13736 Riverport Dr., Maryland Heights, MO 63043

From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Friday, July 15, 2022 3:20 PM
To: users@nifi.apache.org
Subject: [EXTERNAL] Re: Issue with SiteToSiteProvenanceReportingTask 1.15.3

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Hello

Is that the totality of what the logs contain?  Ideally we're going to need to see the full logs.

The other case you link is rooted in having exhausted the open file limits.  Your case appears different.

Would be good to file a JIRA with the logs, details about the JVM version at play, details about the os in use.

Thanks
Joe

On Fri, Jul 15, 2022 at 1:11 PM Chirthani, Deepak Reddy <C-...@charter.com>> wrote:
Hey Guys,

we have started to use the reporting task SiteToSiteProvenanceReportingTask 1.15.3 for auditing purposes on provenance events for a few of our dataflows. We started to see error quite frequently now in the Bulletin Board.
ERROR: Failed to index Provenance Events. See logs for more information.


We also found this on the community https://issues.apache.org/jira/browse/NIFI-9572 that many people are having this issue and the root cause is still not resolved. Can you guys help us with any interim solutions which you might have found to solve this error?

Also, below are snaps of nifi.provenance.properties file and from logs regarding this error. Please let me know if you need anything more.

[cid:image002.png@01D89862.9AE44DE0]
2022-07-15 02:17:47,227 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
              at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
              at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
              at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
              at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
              at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
              at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
              at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-07-15T07:17:47.202562Z, (lock=NativeFSLock(path=/apps/nifi/repos/provenance_repository/lucene-8-index-1657869467201/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-07-15T07:17:47.201562Z))
              at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191)
              at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
              at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
              at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:118)
              at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)
              at org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)
              at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:48)
              at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:55)
              at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:449)
              at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:485)
              at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:208)
              at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:415)
              at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
              ... 9 common frames omitted

Thanks
Deepak


[image005]
Deepak Reddy | Data Engineer
IT Centers of Excellence
13736 Riverport Dr., Maryland Heights, MO 63043



[image005]
Deepak Reddy | Data Engineer
IT Centers of Excellence
13736 Riverport Dr., Maryland Heights, MO 63043

The contents of this e-mail message and
any attachments are intended solely for the
addressee(s) and may contain confidential
and/or legally privileged information. If you
are not the intended recipient of this message
or if this message has been addressed to you
in error, please immediately alert the sender
by reply e-mail and then delete this message
and any attachments. If you are not the
intended recipient, you are notified that
any use, dissemination, distribution, copying,
or storage of this message or any attachment
is strictly prohibited.
E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.