Re: NiFi repositories on GlusterFS

[Michal Klempa <mi...@gmail.com>]

Hi Fabian,
I was wondering of doing the same setup you decribe and found your email.
The issue https://bugzilla.redhat.com/show_bug.cgi?id=1390050 was
closed as of Oct 2018, have you resolved your issues?
What are your experiences with NiFi over GlusterFS? Should I even try
or is it worthless.
The setup would be very similar NiFi in Docker Swarm with Volumes in GlusterFS.

Thank you.
Michal

On Wed, Jan 3, 2018 at 4:42 PM Necci, Fabian
<Fa...@thinkbiganalytics.com> wrote:
>
> Hi all,
>
> I have one NiFi instance running inside a Docker container and managed by Kubernetes.
>
>
>
> To maintain the application state, the NiFi repositories are under the path /usr/nifi/data which is a GlusterFS mount.
>
>
>
> After sometimes, the lucene index of the provenance repository got corrupted:
>
>
>
> 2018-01-03 12:05:07,879 INFO [Provenance Repository Rollover Thread-2] SimpleIndexManager.java:309 Index Writer for ./data/provenance_repository/index-1512127508000 has been returned to Index Manager and is no longer in use. Closing Index Writer
>
> 2018-01-03 12:05:15,669 INFO [Provenance Repository Rollover Thread-2] PersistentProvenanceRepository.java:1903 Successfully merged 16 journal files (5 records) into single Provenance Log File ./data/provenance_repository/492.prov in 18387 milliseconds
>
> 2018-01-03 12:05:15,754 INFO [Provenance Repository Rollover Thread-2] PersistentProvenanceRepository.java:1324 Successfully Rolled over Provenance Event file containing 29 records. In the past 5 minutes, 33 events have been written to the Provenance Repository, totaling 178.78 KB
>
> 2018-01-03 12:05:30,124 INFO [Provenance Maintenance Thread-3] PersistentProvenanceRepository.java:349 Created new Provenance Event Writers for events starting with ID 499
>
> 2018-01-03 12:05:30,128 INFO [Provenance Maintenance Thread-3] KyloPersistentProvenanceEventRepository.java:78 Created 16 KyloRecordWriterDelegate objects starting with: 499
>
> 2018-01-03 12:05:33,494 ERROR [Lucene Merge Thread #0] NiFi.java:83 An Unknown Error Occurred in Thread Thread[Lucene Merge Thread #0,6,main]: org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs") [slice=_ii.fdx]
>
> 2018-01-03 12:05:33,523 ERROR [Lucene Merge Thread #0] NiFi.java:84
>
> org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs") [slice=_ii.fdx]
>
>     at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:549)
>
>     at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:522)
>
> Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs") [slice=_ii.fdx]
>
>     at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
>
>     at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41)
>
>     at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
>
>     at org.apache.lucene.store.DataInput.readLong(DataInput.java:154)
>
>     at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:209)
>
>     at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:124)
>
>     at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)
>
>     at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:133)
>
>     at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:108)
>
>     at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
>
>     at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:664)
>
>     at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4152)
>
>     at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)
>
>     at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)
>
>     at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)
>
>
>
> Has anyboy any idea how could I debug this problem?
>
>
>
> I found these issues that could be related:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1379568
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1390050
>
>
>
> Thanks,
>
> Fabian