You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Ryan H <rh...@gmail.com> on 2015/10/23 18:47:26 UTC
Content Repo Large.. Archive in there?
Hi,
I'm noticing my Content Repo growing large. There's a number of files...
content_repo/837/archive/144...-837
Is this new in 3.0? My conf file says any archiving should be going
into ./conf/archive, but i don't see anything in there.
Thanks,
Ryan
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
We don't.. We've talked about putting it on there. We will in the future.
On Fri, Oct 23, 2015 at 3:46 PM, Mark Payne <ma...@hotmail.com> wrote:
> So digging in a bit more, the issue that I was concerned about is not
> really an issue, as the data is
> still cleaned up elsewhere in the code if the FlowFile repo is filled up.
>
> There certainly must have been something that would have caused the
> content repo to fill up, though.
> Any chance that you have system metrics/reporting applications running on
> this box, such as
> Ganglia? That would help indicate when the disk started filling up to
> narrow down the timeframe to
> look at in the logs.
>
>
>
>
> > On Oct 23, 2015, at 3:27 PM, Ryan H <rh...@gmail.com> wrote:
> >
> > Yea.. they're all on the same partition. (We know.. not good..)
> >
> > I grepped and haven't found any OutOfMem errors...
> >
> > Ryan
> >
> > On Fri, Oct 23, 2015 at 3:15 PM, Mark Payne <ma...@hotmail.com>
> wrote:
> >
> >> This is indicating that it's unable to write to the
> FileSystemRepository -
> >> presumably because it is out of disk space.
> >>
> >> Any other error messages that you find?
> >>
> >> I'm wondering specifically if perhaps you received an OutOfMemoryError
> or
> >> anything of that nature? If it's not cleaning up
> >> after itself, that would tend to indicate that the cleanup thread is no
> >> longer running. So looking for anything that could
> >> explain that.
> >>
> >> Also of note, you're seeing that the Content Repo is out of space, and
> you
> >> also mentioned an issue writing to the Provenance
> >> Repository.
> >>
> >> Was your FlowFile Repository also out of disk space? Looking through the
> >> code now I'm seeing a case where if the FlowFile
> >> Repository is also out of disk space you may start running into issues
> >> with the content repo filling up as well.
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>> On Oct 23, 2015, at 3:00 PM, Ryan H <rh...@gmail.com>
> wrote:
> >>>
> >>> Random mentions of FileSystemRepo
> >>>
> >>> nifi-app_2015-10-23_10.0.log:
> >>> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> >> import data from java.io.ByteArrayInputStream@639fe470 for
> >>
> StandardFlowFileRecord[uuid=881506e8-c62f-442f-a564-88675e4f0372,claim=,offset=0,name=914750081936080,size=0]
> >> due to org.apache.nifi.processor.exception.FlowFileAccessException:
> Unable
> >> to create ContentClaim due to java.io.IOException: Failed to write to
> >> FileSystemRepository Stream [StandardContentClaim
> >> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
> >> container=default, section=347], offset=0, length=-1]]
> >>> nifi-app_2015-10-23_10.0.log:Caused by:
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
> >> create ContentClaim due to java.io.IOException: Failed to write to
> >> FileSystemRepository Stream [StandardContentClaim
> >> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
> >> container=default, section=347], offset=0, length=-1]]
> >>> nifi-app_2015-10-23_10.0.log:Caused by: java.io.IOException: Failed to
> >> write to FileSystemRepository Stream [StandardContentClaim
> >> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
> >> container=default, section=347], offset=0, length=-1]]
> >>> nifi-app_2015-10-23_10.0.log: at
> >>
> org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:913)
> >> ~[nifi-framework-core-0.3.0.jar:0.3.0]
> >>> nifi-app_2015-10-23_10.0.log: at
> >>
> org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:689)
> >> ~[nifi-framework-core-0.3.0.jar:0.3.0]
> >>> nifi-app_2015-10-23_10.0.log: at
> >>
> org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:910)
> >> ~[nifi-framework-core-0.3.0.jar:0.3.0]
> >>>
> >>
> nifi-app_2015-10-23_10.0.log:org.apache.nifi.processor.exception.FlowFileAccessException:
> >> Failed to import data from /opt/nifi/logs/nifi-app_2015-10-23_05.0.log
> for
> >>
> StandardFlowFileRecord[uuid=1b365988-1287-4d12-a31c-75c99dfc12ad,claim=,offset=0,name=914754786738219,size=0]
> >> due to java.io.IOException: Failed to write to FileSystemRepository
> Stream
> >> [StandardContentClaim
> >> [resourceClaim=StandardResourceClaim[id=1445594400001-74079,
> >> container=default, section=351], offset=0, length=-1]]
> >>>
> >>>
> >>>
> >>> On Fri, Oct 23, 2015 at 2:59 PM, Ryan H <rhendrickson.work@gmail.com
> >> <ma...@gmail.com>> wrote:
> >>> Here's our provenance settings:
> >>>
> >>> # Persistent Provenance Repository Properties
> >>> nifi.provenance.repository.directory.default=./provenance_repository
> >>> nifi.provenance.repository.max.storage.time=24 hours
> >>> nifi.provenance.repository.max.storage.size=1 GB
> >>> nifi.provenance.repository.rollover.time=30 secs
> >>> nifi.provenance.repository.rollover.size=100 MB
> >>> nifi.provenance.repository.query.threads=2
> >>> nifi.provenance.repository.index.threads=1
> >>> nifi.provenance.repository.compress.on.rollover=true
> >>> nifi.provenance.repository.always.sync=false
> >>> nifi.provenance.repository.journal.count=16
> >>> # Comma-separated list of fields. Fields that are not indexed will not
> >> be searchable. Valid fields are:
> >>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> >> AlternateIdentifierURI, ContentType, Relationship, Details
> >>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
> >> Filename, ProcessorID, Relationship
> >>> # FlowFile Attributes that should be indexed and made searchable
> >>> nifi.provenance.repository.indexed.attributes=
> >>> # Large values for the shard size will result in more Java heap usage
> >> when searching the Provenance Repository
> >>> # but should provide better performance
> >>> nifi.provenance.repository.index.shard.size=500 MB
> >>> # Indicates the maximum length that a FlowFile attribute can be when
> >> retrieving a Provenance Event from
> >>> # the repository. If the length of any attribute exceeds this value, it
> >> will be truncated when the event is retrieved.
> >>> nifi.provenance.repository.max.attribute.length=65536
> >>>
> >>> # Volatile Provenance Respository Properties
> >>> nifi.provenance.repository.buffer.size=100000
> >>>
> >>> On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz
> >> <eliezer_schwarz@yahoo.com.invalid <mailto:
> >> eliezer_schwarz@yahoo.com.invalid>> wrote:
> >>> We had max storage size of 1GB, but that's for provenance repo and our
> >> problem was with content_repo. Our disk was 60GB, all on one partition,
> and
> >> 55GB were taken up by content_repo. Now, it only contains 233MB.
> >>>
> >>>
> >>>
> >>> On Friday, October 23, 2015 2:50 PM, Mark Payne <
> >> markap14@hotmail.com <ma...@hotmail.com>> wrote:
> >>>
> >>>
> >>>
> >>> OK, so this is interesting. Do you have your content repository and
> >> provenance repository
> >>> both pointing to the same partition? What do you have the
> >> "nifi.provenance.repository.max.storage.size"
> >>> property set to? How large is the actual disk?
> >>>
> >>> Thanks
> >>> -Mark
> >>>
> >>>
> >>>> On Oct 23, 2015, at 2:45 PM, Ryan H <rhendrickson.work@gmail.com
> >> <ma...@gmail.com>> wrote:
> >>>>
> >>>> I've got this one... let me look for that
> >>>>
> >>>> 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
> >>>> o.a.n.p.PersistentProvenanceRepository
> >>>> java.io.IOException: No space left on device
> >>>> at java.io.FileOutputStream.writeBytes(Native Method)
> >> ~[na:1.8.0_51]
> >>>> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> >>>> ~[na:1.8.0_51]
> >>>> at
> >>>>
> >>
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>> java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> >>>> ~[na:1.8.0_51]
> >>>> at
> >>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> >>>> ~[na:1.8.0_51]
> >>>> at
> >>>>
> >>
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>> org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>> org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
> >>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> >>>> at
> >>>>
> >>
> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
> >>>> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> >>>> at
> >>>>
> >>
> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
> >>>> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> >>>> at
> >>>>
> >>
> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
> >>>> [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> >>>>
> >>>> On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <markap14@hotmail.com
> >> <ma...@hotmail.com>> wrote:
> >>>>
> >>>>> Ryan, Elli,
> >>>>>
> >>>>> Do you by chance have any error messages in your logs from the
> >>>>> FileSystemRepository?
> >>>>>
> >>>>> I.e., if you perform:
> >>>>>
> >>>>> grep FileSystemRepository logs/*
> >>>>>
> >>>>> Do you get anything interesting in there?
> >>>>>
> >>>>> Thanks
> >>>>> -Mark
> >>>>>
> >>>>>
> >>>>>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
> >>>>> <el...@yahoo.com.INVALID> wrote:
> >>>>>>
> >>>>>> I've been working with Ryan. There appear to be a few issues here:
> >>>>>>
> >>>>>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
> >>>>> content_repository archive is now true by default. In 0.2.0 it was
> >> false,
> >>>>> and the documentation still states it is false by default.
> >>>>>> - When we ran out of disk space overnight, the problem was solved by
> >>>>> me simply restarting nifi, and that cleared out the archive by
> itself.
> >>>>>>
> >>>>>> - In order to clear up the archive, I had to set archive to true,
> >> and
> >>>>> set max usage to 1%, and restart nifi. That cleared it up, and then I
> >> set
> >>>>> archive to false and restarted again so we don't run out of space.
> >>>>>> - Based on the above, it appears that something happened yesterday
> >>>>> that prevented Nifi from clearing out the archive even though disk
> >> usage
> >>>>> reached 100%. However, restarting nifi apparently enabled it to
> >> perform the
> >>>>> clearing of the archive. So apparently the max usage setting doesn't
> >> work
> >>>>> under some conditions, but we don't know what conditions occurred
> >> overnight
> >>>>> to cause this problem.
> >>>>>>
> >>>>>> Thanks!-Elli
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Friday, October 23, 2015 2:29 PM, Ryan H <
> >>>>> rhendrickson.work@gmail.com <ma...@gmail.com>>
> >> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Agree, they concern the archive... although it sounds like there are
> >> 2
> >>>>>> archives?
> >>>>>>
> >>>>>> Within the content_repository folder, there are subfolders with the
> >> name
> >>>>>> 'archive' and files inside them.
> >>>>>>
> >>>>>> Example:
> >>>>>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>>>>
> >>>>>> Settings:
> >>>>>> nifi.content.repository.archive.max.retention.period=12 hours
> >>>>>> nifi.content.repository.archive.max.usage.percentage=50%
> >>>>>> nifi.content.repository.archive.enabled=true
> >>>>>>
> >>>>>> Last night, our server ran out of disk space because the
> >>>>> content_repository
> >>>>>> grew too large. Nifi didn't crash, but the log file contained
> errors
> >>>>>> saying the disk was full.
> >>>>>>
> >>>>>> We're not sure how, but the content_repository did not respect the
> >> above
> >>>>>> settings.
> >>>>>>
> >>>>>> We restarted Nifi, and it only then started to remove files, such
> as:
> >>>>>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>>>>
> >>>>>> We've turned off archiving for now.
> >>>>>>
> >>>>>> Ryan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinpiri@gmail.com
> >> <ma...@gmail.com>>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Ryan,
> >>>>>>>
> >>>>>>> Those items only concern the archive. Did you have data enqueued
> in
> >>>>>>> connections in your flow? If so, these items are not eligible and
> >> could
> >>>>>>> explain why your disk was filled. Otherwise, can you please
> provide
> >>>>> some
> >>>>>>> additional information so we can dig into why this may have arisen.
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <
> >> rhendrickson.work@gmail.com <ma...@gmail.com>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I've got the following set:
> >>>>>>>>
> >>>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
> >>>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
> >>>>>>>> nifi.content.repository.archive.enabled=true
> >>>>>>>>
> >>>>>>>> Yet, the content repo filled my disk last night...
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <
> aldrinpiri@gmail.com
> >> <ma...@gmail.com>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Ryan,
> >>>>>>>>>
> >>>>>>>>> Those archive folders map to the
> >>>>>>> nifi.content.repository.archive.enabled
> >>>>>>>>> property.
> >>>>>>>>>
> >>>>>>>>> What this property provides is a retention of files no longer in
> >> the
> >>>>>>>> system
> >>>>>>>>> for historical context of your flow's processing and the ability
> >> for
> >>>>>>>>> viewing this in conjunction with provenance events as well as
> >> allowing
> >>>>>>>>> replay. The amount of the archive when enabled is bounded by the
> >>>>>>>>> properties nifi.content.repository.archive.max.retention.period
> >> and
> >>>>>>>>> nifi.content.repository.archive.max.usage.percentage.
> >>>>>>>>>
> >>>>>>>>> Additional detail is available in the system properties of our
> >>>>>>>>> Administration Guide [1]
> >>>>>>>>>
> >>>>>>>>> Let us know if you have additional questions.
> >>>>>>>>>
> >>>>>>>>> --aldrin
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> >> <
> >>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> >>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <
> >> rhendrickson.work@gmail.com <ma...@gmail.com>
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Interesting.. So what would
> >>>>>>>>>>
> >>>>>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>>>>>>>>
> >>>>>>>>>> typically be?
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> >>>>>>>> agrande@hortonworks.com <ma...@hortonworks.com>
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Attachments don't go through, view at imagebin:
> >>>>>>>>>>> http://ibin.co/2K3SwR0z8yWX <http://ibin.co/2K3SwR0z8yWX>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <
> agrande@hortonworks.com
> >> <ma...@hortonworks.com>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Ryan,
> >>>>>>>>>>>>
> >>>>>>>>>>>> ./conf/archive is to create a snapshot of your entire flow,
> not
> >>>>>>> the
> >>>>>>>>>>> content repository data. See the attached screenshot (Settings
> >> menu
> >>>>>>>> on
> >>>>>>>>>> the
> >>>>>>>>>>> right).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Andrew
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com
> >> <ma...@gmail.com> on
> >>>>>>> behalf
> >>>>>>>>> of
> >>>>>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com <mailto:
> >> ryan.andrew.hendrickson@gmail.com> on behalf of
> >>>>>>>>>>> rhendrickson.work@gmail.com <mailto:
> rhendrickson.work@gmail.com>>
> >> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
> >>>>>>> of
> >>>>>>>>>>> files...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> content_repo/837/archive/144...-837
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is this new in 3.0? My conf file says any archiving should
> >> be
> >>>>>>>>> going
> >>>>>>>>>>>>> into ./conf/archive, but i don't see anything in there.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Ryan
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
>
>
Re: Content Repo Large.. Archive in there?
Posted by Mark Payne <ma...@hotmail.com>.
So digging in a bit more, the issue that I was concerned about is not really an issue, as the data is
still cleaned up elsewhere in the code if the FlowFile repo is filled up.
There certainly must have been something that would have caused the content repo to fill up, though.
Any chance that you have system metrics/reporting applications running on this box, such as
Ganglia? That would help indicate when the disk started filling up to narrow down the timeframe to
look at in the logs.
> On Oct 23, 2015, at 3:27 PM, Ryan H <rh...@gmail.com> wrote:
>
> Yea.. they're all on the same partition. (We know.. not good..)
>
> I grepped and haven't found any OutOfMem errors...
>
> Ryan
>
> On Fri, Oct 23, 2015 at 3:15 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> This is indicating that it's unable to write to the FileSystemRepository -
>> presumably because it is out of disk space.
>>
>> Any other error messages that you find?
>>
>> I'm wondering specifically if perhaps you received an OutOfMemoryError or
>> anything of that nature? If it's not cleaning up
>> after itself, that would tend to indicate that the cleanup thread is no
>> longer running. So looking for anything that could
>> explain that.
>>
>> Also of note, you're seeing that the Content Repo is out of space, and you
>> also mentioned an issue writing to the Provenance
>> Repository.
>>
>> Was your FlowFile Repository also out of disk space? Looking through the
>> code now I'm seeing a case where if the FlowFile
>> Repository is also out of disk space you may start running into issues
>> with the content repo filling up as well.
>>
>> Thanks
>> -Mark
>>
>>
>>> On Oct 23, 2015, at 3:00 PM, Ryan H <rh...@gmail.com> wrote:
>>>
>>> Random mentions of FileSystemRepo
>>>
>>> nifi-app_2015-10-23_10.0.log:
>>> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
>> import data from java.io.ByteArrayInputStream@639fe470 for
>> StandardFlowFileRecord[uuid=881506e8-c62f-442f-a564-88675e4f0372,claim=,offset=0,name=914750081936080,size=0]
>> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
>> to create ContentClaim due to java.io.IOException: Failed to write to
>> FileSystemRepository Stream [StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
>> container=default, section=347], offset=0, length=-1]]
>>> nifi-app_2015-10-23_10.0.log:Caused by:
>> org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
>> create ContentClaim due to java.io.IOException: Failed to write to
>> FileSystemRepository Stream [StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
>> container=default, section=347], offset=0, length=-1]]
>>> nifi-app_2015-10-23_10.0.log:Caused by: java.io.IOException: Failed to
>> write to FileSystemRepository Stream [StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
>> container=default, section=347], offset=0, length=-1]]
>>> nifi-app_2015-10-23_10.0.log: at
>> org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:913)
>> ~[nifi-framework-core-0.3.0.jar:0.3.0]
>>> nifi-app_2015-10-23_10.0.log: at
>> org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:689)
>> ~[nifi-framework-core-0.3.0.jar:0.3.0]
>>> nifi-app_2015-10-23_10.0.log: at
>> org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:910)
>> ~[nifi-framework-core-0.3.0.jar:0.3.0]
>>>
>> nifi-app_2015-10-23_10.0.log:org.apache.nifi.processor.exception.FlowFileAccessException:
>> Failed to import data from /opt/nifi/logs/nifi-app_2015-10-23_05.0.log for
>> StandardFlowFileRecord[uuid=1b365988-1287-4d12-a31c-75c99dfc12ad,claim=,offset=0,name=914754786738219,size=0]
>> due to java.io.IOException: Failed to write to FileSystemRepository Stream
>> [StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1445594400001-74079,
>> container=default, section=351], offset=0, length=-1]]
>>>
>>>
>>>
>>> On Fri, Oct 23, 2015 at 2:59 PM, Ryan H <rhendrickson.work@gmail.com
>> <ma...@gmail.com>> wrote:
>>> Here's our provenance settings:
>>>
>>> # Persistent Provenance Repository Properties
>>> nifi.provenance.repository.directory.default=./provenance_repository
>>> nifi.provenance.repository.max.storage.time=24 hours
>>> nifi.provenance.repository.max.storage.size=1 GB
>>> nifi.provenance.repository.rollover.time=30 secs
>>> nifi.provenance.repository.rollover.size=100 MB
>>> nifi.provenance.repository.query.threads=2
>>> nifi.provenance.repository.index.threads=1
>>> nifi.provenance.repository.compress.on.rollover=true
>>> nifi.provenance.repository.always.sync=false
>>> nifi.provenance.repository.journal.count=16
>>> # Comma-separated list of fields. Fields that are not indexed will not
>> be searchable. Valid fields are:
>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>> AlternateIdentifierURI, ContentType, Relationship, Details
>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
>> Filename, ProcessorID, Relationship
>>> # FlowFile Attributes that should be indexed and made searchable
>>> nifi.provenance.repository.indexed.attributes=
>>> # Large values for the shard size will result in more Java heap usage
>> when searching the Provenance Repository
>>> # but should provide better performance
>>> nifi.provenance.repository.index.shard.size=500 MB
>>> # Indicates the maximum length that a FlowFile attribute can be when
>> retrieving a Provenance Event from
>>> # the repository. If the length of any attribute exceeds this value, it
>> will be truncated when the event is retrieved.
>>> nifi.provenance.repository.max.attribute.length=65536
>>>
>>> # Volatile Provenance Respository Properties
>>> nifi.provenance.repository.buffer.size=100000
>>>
>>> On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz
>> <eliezer_schwarz@yahoo.com.invalid <mailto:
>> eliezer_schwarz@yahoo.com.invalid>> wrote:
>>> We had max storage size of 1GB, but that's for provenance repo and our
>> problem was with content_repo. Our disk was 60GB, all on one partition, and
>> 55GB were taken up by content_repo. Now, it only contains 233MB.
>>>
>>>
>>>
>>> On Friday, October 23, 2015 2:50 PM, Mark Payne <
>> markap14@hotmail.com <ma...@hotmail.com>> wrote:
>>>
>>>
>>>
>>> OK, so this is interesting. Do you have your content repository and
>> provenance repository
>>> both pointing to the same partition? What do you have the
>> "nifi.provenance.repository.max.storage.size"
>>> property set to? How large is the actual disk?
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>>> On Oct 23, 2015, at 2:45 PM, Ryan H <rhendrickson.work@gmail.com
>> <ma...@gmail.com>> wrote:
>>>>
>>>> I've got this one... let me look for that
>>>>
>>>> 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
>>>> o.a.n.p.PersistentProvenanceRepository
>>>> java.io.IOException: No space left on device
>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>> ~[na:1.8.0_51]
>>>> at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>>> ~[na:1.8.0_51]
>>>> at
>>>>
>> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>> java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
>>>> ~[na:1.8.0_51]
>>>> at
>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>>>> ~[na:1.8.0_51]
>>>> at
>>>>
>> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>> org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>> org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
>>>> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>>>> at
>>>>
>> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
>>>> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>>>> at
>>>>
>> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
>>>> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>>>> at
>>>>
>> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
>>>> [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>>>>
>>>> On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <markap14@hotmail.com
>> <ma...@hotmail.com>> wrote:
>>>>
>>>>> Ryan, Elli,
>>>>>
>>>>> Do you by chance have any error messages in your logs from the
>>>>> FileSystemRepository?
>>>>>
>>>>> I.e., if you perform:
>>>>>
>>>>> grep FileSystemRepository logs/*
>>>>>
>>>>> Do you get anything interesting in there?
>>>>>
>>>>> Thanks
>>>>> -Mark
>>>>>
>>>>>
>>>>>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
>>>>> <el...@yahoo.com.INVALID> wrote:
>>>>>>
>>>>>> I've been working with Ryan. There appear to be a few issues here:
>>>>>>
>>>>>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
>>>>> content_repository archive is now true by default. In 0.2.0 it was
>> false,
>>>>> and the documentation still states it is false by default.
>>>>>> - When we ran out of disk space overnight, the problem was solved by
>>>>> me simply restarting nifi, and that cleared out the archive by itself.
>>>>>>
>>>>>> - In order to clear up the archive, I had to set archive to true,
>> and
>>>>> set max usage to 1%, and restart nifi. That cleared it up, and then I
>> set
>>>>> archive to false and restarted again so we don't run out of space.
>>>>>> - Based on the above, it appears that something happened yesterday
>>>>> that prevented Nifi from clearing out the archive even though disk
>> usage
>>>>> reached 100%. However, restarting nifi apparently enabled it to
>> perform the
>>>>> clearing of the archive. So apparently the max usage setting doesn't
>> work
>>>>> under some conditions, but we don't know what conditions occurred
>> overnight
>>>>> to cause this problem.
>>>>>>
>>>>>> Thanks!-Elli
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Friday, October 23, 2015 2:29 PM, Ryan H <
>>>>> rhendrickson.work@gmail.com <ma...@gmail.com>>
>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Agree, they concern the archive... although it sounds like there are
>> 2
>>>>>> archives?
>>>>>>
>>>>>> Within the content_repository folder, there are subfolders with the
>> name
>>>>>> 'archive' and files inside them.
>>>>>>
>>>>>> Example:
>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>>>>
>>>>>> Settings:
>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>>> nifi.content.repository.archive.enabled=true
>>>>>>
>>>>>> Last night, our server ran out of disk space because the
>>>>> content_repository
>>>>>> grew too large. Nifi didn't crash, but the log file contained errors
>>>>>> saying the disk was full.
>>>>>>
>>>>>> We're not sure how, but the content_repository did not respect the
>> above
>>>>>> settings.
>>>>>>
>>>>>> We restarted Nifi, and it only then started to remove files, such as:
>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>>>>
>>>>>> We've turned off archiving for now.
>>>>>>
>>>>>> Ryan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinpiri@gmail.com
>> <ma...@gmail.com>>
>>>>> wrote:
>>>>>>
>>>>>>> Ryan,
>>>>>>>
>>>>>>> Those items only concern the archive. Did you have data enqueued in
>>>>>>> connections in your flow? If so, these items are not eligible and
>> could
>>>>>>> explain why your disk was filled. Otherwise, can you please provide
>>>>> some
>>>>>>> additional information so we can dig into why this may have arisen.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <
>> rhendrickson.work@gmail.com <ma...@gmail.com>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I've got the following set:
>>>>>>>>
>>>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>>>>> nifi.content.repository.archive.enabled=true
>>>>>>>>
>>>>>>>> Yet, the content repo filled my disk last night...
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <aldrinpiri@gmail.com
>> <ma...@gmail.com>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ryan,
>>>>>>>>>
>>>>>>>>> Those archive folders map to the
>>>>>>> nifi.content.repository.archive.enabled
>>>>>>>>> property.
>>>>>>>>>
>>>>>>>>> What this property provides is a retention of files no longer in
>> the
>>>>>>>> system
>>>>>>>>> for historical context of your flow's processing and the ability
>> for
>>>>>>>>> viewing this in conjunction with provenance events as well as
>> allowing
>>>>>>>>> replay. The amount of the archive when enabled is bounded by the
>>>>>>>>> properties nifi.content.repository.archive.max.retention.period
>> and
>>>>>>>>> nifi.content.repository.archive.max.usage.percentage.
>>>>>>>>>
>>>>>>>>> Additional detail is available in the system properties of our
>>>>>>>>> Administration Guide [1]
>>>>>>>>>
>>>>>>>>> Let us know if you have additional questions.
>>>>>>>>>
>>>>>>>>> --aldrin
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>> <
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>>>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <
>> rhendrickson.work@gmail.com <ma...@gmail.com>
>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Interesting.. So what would
>>>>>>>>>>
>>>>>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>>>>>>>>
>>>>>>>>>> typically be?
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
>>>>>>>> agrande@hortonworks.com <ma...@hortonworks.com>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Attachments don't go through, view at imagebin:
>>>>>>>>>>> http://ibin.co/2K3SwR0z8yWX <http://ibin.co/2K3SwR0z8yWX>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <agrande@hortonworks.com
>> <ma...@hortonworks.com>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>
>>>>>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
>>>>>>> the
>>>>>>>>>>> content repository data. See the attached screenshot (Settings
>> menu
>>>>>>>> on
>>>>>>>>>> the
>>>>>>>>>>> right).
>>>>>>>>>>>>
>>>>>>>>>>>> Andrew
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com
>> <ma...@gmail.com> on
>>>>>>> behalf
>>>>>>>>> of
>>>>>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com <mailto:
>> ryan.andrew.hendrickson@gmail.com> on behalf of
>>>>>>>>>>> rhendrickson.work@gmail.com <ma...@gmail.com>>
>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
>>>>>>> of
>>>>>>>>>>> files...
>>>>>>>>>>>>>
>>>>>>>>>>>>> content_repo/837/archive/144...-837
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this new in 3.0? My conf file says any archiving should
>> be
>>>>>>>>> going
>>>>>>>>>>>>> into ./conf/archive, but i don't see anything in there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Ryan
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
Yea.. they're all on the same partition. (We know.. not good..)
I grepped and haven't found any OutOfMem errors...
Ryan
On Fri, Oct 23, 2015 at 3:15 PM, Mark Payne <ma...@hotmail.com> wrote:
> This is indicating that it's unable to write to the FileSystemRepository -
> presumably because it is out of disk space.
>
> Any other error messages that you find?
>
> I'm wondering specifically if perhaps you received an OutOfMemoryError or
> anything of that nature? If it's not cleaning up
> after itself, that would tend to indicate that the cleanup thread is no
> longer running. So looking for anything that could
> explain that.
>
> Also of note, you're seeing that the Content Repo is out of space, and you
> also mentioned an issue writing to the Provenance
> Repository.
>
> Was your FlowFile Repository also out of disk space? Looking through the
> code now I'm seeing a case where if the FlowFile
> Repository is also out of disk space you may start running into issues
> with the content repo filling up as well.
>
> Thanks
> -Mark
>
>
> > On Oct 23, 2015, at 3:00 PM, Ryan H <rh...@gmail.com> wrote:
> >
> > Random mentions of FileSystemRepo
> >
> > nifi-app_2015-10-23_10.0.log:
> > org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> import data from java.io.ByteArrayInputStream@639fe470 for
> StandardFlowFileRecord[uuid=881506e8-c62f-442f-a564-88675e4f0372,claim=,offset=0,name=914750081936080,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to java.io.IOException: Failed to write to
> FileSystemRepository Stream [StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
> container=default, section=347], offset=0, length=-1]]
> > nifi-app_2015-10-23_10.0.log:Caused by:
> org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
> create ContentClaim due to java.io.IOException: Failed to write to
> FileSystemRepository Stream [StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
> container=default, section=347], offset=0, length=-1]]
> > nifi-app_2015-10-23_10.0.log:Caused by: java.io.IOException: Failed to
> write to FileSystemRepository Stream [StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1445594395635-74075,
> container=default, section=347], offset=0, length=-1]]
> > nifi-app_2015-10-23_10.0.log: at
> org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:913)
> ~[nifi-framework-core-0.3.0.jar:0.3.0]
> > nifi-app_2015-10-23_10.0.log: at
> org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:689)
> ~[nifi-framework-core-0.3.0.jar:0.3.0]
> > nifi-app_2015-10-23_10.0.log: at
> org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:910)
> ~[nifi-framework-core-0.3.0.jar:0.3.0]
> >
> nifi-app_2015-10-23_10.0.log:org.apache.nifi.processor.exception.FlowFileAccessException:
> Failed to import data from /opt/nifi/logs/nifi-app_2015-10-23_05.0.log for
> StandardFlowFileRecord[uuid=1b365988-1287-4d12-a31c-75c99dfc12ad,claim=,offset=0,name=914754786738219,size=0]
> due to java.io.IOException: Failed to write to FileSystemRepository Stream
> [StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1445594400001-74079,
> container=default, section=351], offset=0, length=-1]]
> >
> >
> >
> > On Fri, Oct 23, 2015 at 2:59 PM, Ryan H <rhendrickson.work@gmail.com
> <ma...@gmail.com>> wrote:
> > Here's our provenance settings:
> >
> > # Persistent Provenance Repository Properties
> > nifi.provenance.repository.directory.default=./provenance_repository
> > nifi.provenance.repository.max.storage.time=24 hours
> > nifi.provenance.repository.max.storage.size=1 GB
> > nifi.provenance.repository.rollover.time=30 secs
> > nifi.provenance.repository.rollover.size=100 MB
> > nifi.provenance.repository.query.threads=2
> > nifi.provenance.repository.index.threads=1
> > nifi.provenance.repository.compress.on.rollover=true
> > nifi.provenance.repository.always.sync=false
> > nifi.provenance.repository.journal.count=16
> > # Comma-separated list of fields. Fields that are not indexed will not
> be searchable. Valid fields are:
> > # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, ContentType, Relationship, Details
> > nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
> Filename, ProcessorID, Relationship
> > # FlowFile Attributes that should be indexed and made searchable
> > nifi.provenance.repository.indexed.attributes=
> > # Large values for the shard size will result in more Java heap usage
> when searching the Provenance Repository
> > # but should provide better performance
> > nifi.provenance.repository.index.shard.size=500 MB
> > # Indicates the maximum length that a FlowFile attribute can be when
> retrieving a Provenance Event from
> > # the repository. If the length of any attribute exceeds this value, it
> will be truncated when the event is retrieved.
> > nifi.provenance.repository.max.attribute.length=65536
> >
> > # Volatile Provenance Respository Properties
> > nifi.provenance.repository.buffer.size=100000
> >
> > On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz
> <eliezer_schwarz@yahoo.com.invalid <mailto:
> eliezer_schwarz@yahoo.com.invalid>> wrote:
> > We had max storage size of 1GB, but that's for provenance repo and our
> problem was with content_repo. Our disk was 60GB, all on one partition, and
> 55GB were taken up by content_repo. Now, it only contains 233MB.
> >
> >
> >
> > On Friday, October 23, 2015 2:50 PM, Mark Payne <
> markap14@hotmail.com <ma...@hotmail.com>> wrote:
> >
> >
> >
> > OK, so this is interesting. Do you have your content repository and
> provenance repository
> > both pointing to the same partition? What do you have the
> "nifi.provenance.repository.max.storage.size"
> > property set to? How large is the actual disk?
> >
> > Thanks
> > -Mark
> >
> >
> > > On Oct 23, 2015, at 2:45 PM, Ryan H <rhendrickson.work@gmail.com
> <ma...@gmail.com>> wrote:
> > >
> > > I've got this one... let me look for that
> > >
> > > 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
> > > o.a.n.p.PersistentProvenanceRepository
> > > java.io.IOException: No space left on device
> > > at java.io.FileOutputStream.writeBytes(Native Method)
> ~[na:1.8.0_51]
> > > at java.io.FileOutputStream.write(FileOutputStream.java:326)
> > > ~[na:1.8.0_51]
> > > at
> > >
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > > java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> > > ~[na:1.8.0_51]
> > > at
> > > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> > > ~[na:1.8.0_51]
> > > at
> > >
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > > org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > > org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
> > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > > at
> > >
> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
> > > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> > > at
> > >
> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
> > > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> > > at
> > >
> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
> > > [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> > >
> > > On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <markap14@hotmail.com
> <ma...@hotmail.com>> wrote:
> > >
> > >> Ryan, Elli,
> > >>
> > >> Do you by chance have any error messages in your logs from the
> > >> FileSystemRepository?
> > >>
> > >> I.e., if you perform:
> > >>
> > >> grep FileSystemRepository logs/*
> > >>
> > >> Do you get anything interesting in there?
> > >>
> > >> Thanks
> > >> -Mark
> > >>
> > >>
> > >>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
> > >> <el...@yahoo.com.INVALID> wrote:
> > >>>
> > >>> I've been working with Ryan. There appear to be a few issues here:
> > >>>
> > >>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
> > >> content_repository archive is now true by default. In 0.2.0 it was
> false,
> > >> and the documentation still states it is false by default.
> > >>> - When we ran out of disk space overnight, the problem was solved by
> > >> me simply restarting nifi, and that cleared out the archive by itself.
> > >>>
> > >>> - In order to clear up the archive, I had to set archive to true,
> and
> > >> set max usage to 1%, and restart nifi. That cleared it up, and then I
> set
> > >> archive to false and restarted again so we don't run out of space.
> > >>> - Based on the above, it appears that something happened yesterday
> > >> that prevented Nifi from clearing out the archive even though disk
> usage
> > >> reached 100%. However, restarting nifi apparently enabled it to
> perform the
> > >> clearing of the archive. So apparently the max usage setting doesn't
> work
> > >> under some conditions, but we don't know what conditions occurred
> overnight
> > >> to cause this problem.
> > >>>
> > >>> Thanks!-Elli
> > >>>
> > >>>
> > >>>
> > >>> On Friday, October 23, 2015 2:29 PM, Ryan H <
> > >> rhendrickson.work@gmail.com <ma...@gmail.com>>
> wrote:
> > >>>
> > >>>
> > >>>
> > >>> Agree, they concern the archive... although it sounds like there are
> 2
> > >>> archives?
> > >>>
> > >>> Within the content_repository folder, there are subfolders with the
> name
> > >>> 'archive' and files inside them.
> > >>>
> > >>> Example:
> > >>> ./nfii/content_repository/837/archive/1445611320767-837
> > >>>
> > >>> Settings:
> > >>> nifi.content.repository.archive.max.retention.period=12 hours
> > >>> nifi.content.repository.archive.max.usage.percentage=50%
> > >>> nifi.content.repository.archive.enabled=true
> > >>>
> > >>> Last night, our server ran out of disk space because the
> > >> content_repository
> > >>> grew too large. Nifi didn't crash, but the log file contained errors
> > >>> saying the disk was full.
> > >>>
> > >>> We're not sure how, but the content_repository did not respect the
> above
> > >>> settings.
> > >>>
> > >>> We restarted Nifi, and it only then started to remove files, such as:
> > >>> ./nfii/content_repository/837/archive/1445611320767-837
> > >>>
> > >>> We've turned off archiving for now.
> > >>>
> > >>> Ryan
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinpiri@gmail.com
> <ma...@gmail.com>>
> > >> wrote:
> > >>>
> > >>>> Ryan,
> > >>>>
> > >>>> Those items only concern the archive. Did you have data enqueued in
> > >>>> connections in your flow? If so, these items are not eligible and
> could
> > >>>> explain why your disk was filled. Otherwise, can you please provide
> > >> some
> > >>>> additional information so we can dig into why this may have arisen.
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <
> rhendrickson.work@gmail.com <ma...@gmail.com>>
> > >>>> wrote:
> > >>>>
> > >>>>> I've got the following set:
> > >>>>>
> > >>>>> nifi.content.repository.archive.max.retention.period=12 hours
> > >>>>> nifi.content.repository.archive.max.usage.percentage=50%
> > >>>>> nifi.content.repository.archive.enabled=true
> > >>>>>
> > >>>>> Yet, the content repo filled my disk last night...
> > >>>>>
> > >>>>>
> > >>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <aldrinpiri@gmail.com
> <ma...@gmail.com>>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Ryan,
> > >>>>>>
> > >>>>>> Those archive folders map to the
> > >>>> nifi.content.repository.archive.enabled
> > >>>>>> property.
> > >>>>>>
> > >>>>>> What this property provides is a retention of files no longer in
> the
> > >>>>> system
> > >>>>>> for historical context of your flow's processing and the ability
> for
> > >>>>>> viewing this in conjunction with provenance events as well as
> allowing
> > >>>>>> replay. The amount of the archive when enabled is bounded by the
> > >>>>>> properties nifi.content.repository.archive.max.retention.period
> and
> > >>>>>> nifi.content.repository.archive.max.usage.percentage.
> > >>>>>>
> > >>>>>> Additional detail is available in the system properties of our
> > >>>>>> Administration Guide [1]
> > >>>>>>
> > >>>>>> Let us know if you have additional questions.
> > >>>>>>
> > >>>>>> --aldrin
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> <
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> >
> > >>>>>>
> > >>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <
> rhendrickson.work@gmail.com <ma...@gmail.com>
> > >>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Interesting.. So what would
> > >>>>>>>
> > >>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
> > >>>>>>>
> > >>>>>>> typically be?
> > >>>>>>>
> > >>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> > >>>>> agrande@hortonworks.com <ma...@hortonworks.com>
> > >>>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Attachments don't go through, view at imagebin:
> > >>>>>>>> http://ibin.co/2K3SwR0z8yWX <http://ibin.co/2K3SwR0z8yWX>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <agrande@hortonworks.com
> <ma...@hortonworks.com>>
> > >>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Ryan,
> > >>>>>>>>>
> > >>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
> > >>>> the
> > >>>>>>>> content repository data. See the attached screenshot (Settings
> menu
> > >>>>> on
> > >>>>>>> the
> > >>>>>>>> right).
> > >>>>>>>>>
> > >>>>>>>>> Andrew
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com
> <ma...@gmail.com> on
> > >>>> behalf
> > >>>>>> of
> > >>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com <mailto:
> ryan.andrew.hendrickson@gmail.com> on behalf of
> > >>>>>>>> rhendrickson.work@gmail.com <ma...@gmail.com>>
> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi,
> > >>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
> > >>>> of
> > >>>>>>>> files...
> > >>>>>>>>>>
> > >>>>>>>>>> content_repo/837/archive/144...-837
> > >>>>>>>>>>
> > >>>>>>>>>> Is this new in 3.0? My conf file says any archiving should
> be
> > >>>>>> going
> > >>>>>>>>>> into ./conf/archive, but i don't see anything in there.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> Ryan
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> >
> >
> >
> >
> >
> >
>
>
Re: Content Repo Large.. Archive in there?
Posted by Mark Payne <ma...@hotmail.com>.
This is indicating that it's unable to write to the FileSystemRepository - presumably because it is out of disk space.
Any other error messages that you find?
I'm wondering specifically if perhaps you received an OutOfMemoryError or anything of that nature? If it's not cleaning up
after itself, that would tend to indicate that the cleanup thread is no longer running. So looking for anything that could
explain that.
Also of note, you're seeing that the Content Repo is out of space, and you also mentioned an issue writing to the Provenance
Repository.
Was your FlowFile Repository also out of disk space? Looking through the code now I'm seeing a case where if the FlowFile
Repository is also out of disk space you may start running into issues with the content repo filling up as well.
Thanks
-Mark
> On Oct 23, 2015, at 3:00 PM, Ryan H <rh...@gmail.com> wrote:
>
> Random mentions of FileSystemRepo
>
> nifi-app_2015-10-23_10.0.log:
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from java.io.ByteArrayInputStream@639fe470 for StandardFlowFileRecord[uuid=881506e8-c62f-442f-a564-88675e4f0372,claim=,offset=0,name=914750081936080,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: Failed to write to FileSystemRepository Stream [StandardContentClaim [resourceClaim=StandardResourceClaim[id=1445594395635-74075, container=default, section=347], offset=0, length=-1]]
> nifi-app_2015-10-23_10.0.log:Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: Failed to write to FileSystemRepository Stream [StandardContentClaim [resourceClaim=StandardResourceClaim[id=1445594395635-74075, container=default, section=347], offset=0, length=-1]]
> nifi-app_2015-10-23_10.0.log:Caused by: java.io.IOException: Failed to write to FileSystemRepository Stream [StandardContentClaim [resourceClaim=StandardResourceClaim[id=1445594395635-74075, container=default, section=347], offset=0, length=-1]]
> nifi-app_2015-10-23_10.0.log: at org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:913) ~[nifi-framework-core-0.3.0.jar:0.3.0]
> nifi-app_2015-10-23_10.0.log: at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:689) ~[nifi-framework-core-0.3.0.jar:0.3.0]
> nifi-app_2015-10-23_10.0.log: at org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:910) ~[nifi-framework-core-0.3.0.jar:0.3.0]
> nifi-app_2015-10-23_10.0.log:org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from /opt/nifi/logs/nifi-app_2015-10-23_05.0.log for StandardFlowFileRecord[uuid=1b365988-1287-4d12-a31c-75c99dfc12ad,claim=,offset=0,name=914754786738219,size=0] due to java.io.IOException: Failed to write to FileSystemRepository Stream [StandardContentClaim [resourceClaim=StandardResourceClaim[id=1445594400001-74079, container=default, section=351], offset=0, length=-1]]
>
>
>
> On Fri, Oct 23, 2015 at 2:59 PM, Ryan H <rhendrickson.work@gmail.com <ma...@gmail.com>> wrote:
> Here's our provenance settings:
>
> # Persistent Provenance Repository Properties
> nifi.provenance.repository.directory.default=./provenance_repository
> nifi.provenance.repository.max.storage.time=24 hours
> nifi.provenance.repository.max.storage.size=1 GB
> nifi.provenance.repository.rollover.time=30 secs
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.journal.count=16
> # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, ContentType, Relationship, Details
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
> # FlowFile Attributes that should be indexed and made searchable
> nifi.provenance.repository.indexed.attributes=
> # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
> # but should provide better performance
> nifi.provenance.repository.index.shard.size=500 MB
> # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
> # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
> nifi.provenance.repository.max.attribute.length=65536
>
> # Volatile Provenance Respository Properties
> nifi.provenance.repository.buffer.size=100000
>
> On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz <eliezer_schwarz@yahoo.com.invalid <ma...@yahoo.com.invalid>> wrote:
> We had max storage size of 1GB, but that's for provenance repo and our problem was with content_repo. Our disk was 60GB, all on one partition, and 55GB were taken up by content_repo. Now, it only contains 233MB.
>
>
>
> On Friday, October 23, 2015 2:50 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>
>
>
> OK, so this is interesting. Do you have your content repository and provenance repository
> both pointing to the same partition? What do you have the "nifi.provenance.repository.max.storage.size"
> property set to? How large is the actual disk?
>
> Thanks
> -Mark
>
>
> > On Oct 23, 2015, at 2:45 PM, Ryan H <rhendrickson.work@gmail.com <ma...@gmail.com>> wrote:
> >
> > I've got this one... let me look for that
> >
> > 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
> > o.a.n.p.PersistentProvenanceRepository
> > java.io.IOException: No space left on device
> > at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_51]
> > at java.io.FileOutputStream.write(FileOutputStream.java:326)
> > ~[na:1.8.0_51]
> > at
> > org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> > ~[na:1.8.0_51]
> > at
> > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> > ~[na:1.8.0_51]
> > at
> > org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
> > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> > at
> > org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
> > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> > at
> > org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
> > [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> >
> > On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
> >
> >> Ryan, Elli,
> >>
> >> Do you by chance have any error messages in your logs from the
> >> FileSystemRepository?
> >>
> >> I.e., if you perform:
> >>
> >> grep FileSystemRepository logs/*
> >>
> >> Do you get anything interesting in there?
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
> >> <el...@yahoo.com.INVALID> wrote:
> >>>
> >>> I've been working with Ryan. There appear to be a few issues here:
> >>>
> >>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
> >> content_repository archive is now true by default. In 0.2.0 it was false,
> >> and the documentation still states it is false by default.
> >>> - When we ran out of disk space overnight, the problem was solved by
> >> me simply restarting nifi, and that cleared out the archive by itself.
> >>>
> >>> - In order to clear up the archive, I had to set archive to true, and
> >> set max usage to 1%, and restart nifi. That cleared it up, and then I set
> >> archive to false and restarted again so we don't run out of space.
> >>> - Based on the above, it appears that something happened yesterday
> >> that prevented Nifi from clearing out the archive even though disk usage
> >> reached 100%. However, restarting nifi apparently enabled it to perform the
> >> clearing of the archive. So apparently the max usage setting doesn't work
> >> under some conditions, but we don't know what conditions occurred overnight
> >> to cause this problem.
> >>>
> >>> Thanks!-Elli
> >>>
> >>>
> >>>
> >>> On Friday, October 23, 2015 2:29 PM, Ryan H <
> >> rhendrickson.work@gmail.com <ma...@gmail.com>> wrote:
> >>>
> >>>
> >>>
> >>> Agree, they concern the archive... although it sounds like there are 2
> >>> archives?
> >>>
> >>> Within the content_repository folder, there are subfolders with the name
> >>> 'archive' and files inside them.
> >>>
> >>> Example:
> >>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>
> >>> Settings:
> >>> nifi.content.repository.archive.max.retention.period=12 hours
> >>> nifi.content.repository.archive.max.usage.percentage=50%
> >>> nifi.content.repository.archive.enabled=true
> >>>
> >>> Last night, our server ran out of disk space because the
> >> content_repository
> >>> grew too large. Nifi didn't crash, but the log file contained errors
> >>> saying the disk was full.
> >>>
> >>> We're not sure how, but the content_repository did not respect the above
> >>> settings.
> >>>
> >>> We restarted Nifi, and it only then started to remove files, such as:
> >>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>
> >>> We've turned off archiving for now.
> >>>
> >>> Ryan
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinpiri@gmail.com <ma...@gmail.com>>
> >> wrote:
> >>>
> >>>> Ryan,
> >>>>
> >>>> Those items only concern the archive. Did you have data enqueued in
> >>>> connections in your flow? If so, these items are not eligible and could
> >>>> explain why your disk was filled. Otherwise, can you please provide
> >> some
> >>>> additional information so we can dig into why this may have arisen.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rhendrickson.work@gmail.com <ma...@gmail.com>>
> >>>> wrote:
> >>>>
> >>>>> I've got the following set:
> >>>>>
> >>>>> nifi.content.repository.archive.max.retention.period=12 hours
> >>>>> nifi.content.repository.archive.max.usage.percentage=50%
> >>>>> nifi.content.repository.archive.enabled=true
> >>>>>
> >>>>> Yet, the content repo filled my disk last night...
> >>>>>
> >>>>>
> >>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <aldrinpiri@gmail.com <ma...@gmail.com>>
> >>>> wrote:
> >>>>>
> >>>>>> Ryan,
> >>>>>>
> >>>>>> Those archive folders map to the
> >>>> nifi.content.repository.archive.enabled
> >>>>>> property.
> >>>>>>
> >>>>>> What this property provides is a retention of files no longer in the
> >>>>> system
> >>>>>> for historical context of your flow's processing and the ability for
> >>>>>> viewing this in conjunction with provenance events as well as allowing
> >>>>>> replay. The amount of the archive when enabled is bounded by the
> >>>>>> properties nifi.content.repository.archive.max.retention.period and
> >>>>>> nifi.content.repository.archive.max.usage.percentage.
> >>>>>>
> >>>>>> Additional detail is available in the system properties of our
> >>>>>> Administration Guide [1]
> >>>>>>
> >>>>>> Let us know if you have additional questions.
> >>>>>>
> >>>>>> --aldrin
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties>
> >>>>>>
> >>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.work@gmail.com <ma...@gmail.com>
> >>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Interesting.. So what would
> >>>>>>>
> >>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>>>>>
> >>>>>>> typically be?
> >>>>>>>
> >>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> >>>>> agrande@hortonworks.com <ma...@hortonworks.com>
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Attachments don't go through, view at imagebin:
> >>>>>>>> http://ibin.co/2K3SwR0z8yWX <http://ibin.co/2K3SwR0z8yWX>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <agrande@hortonworks.com <ma...@hortonworks.com>>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Ryan,
> >>>>>>>>>
> >>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
> >>>> the
> >>>>>>>> content repository data. See the attached screenshot (Settings menu
> >>>>> on
> >>>>>>> the
> >>>>>>>> right).
> >>>>>>>>>
> >>>>>>>>> Andrew
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com <ma...@gmail.com> on
> >>>> behalf
> >>>>>> of
> >>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com <ma...@gmail.com> on behalf of
> >>>>>>>> rhendrickson.work@gmail.com <ma...@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
> >>>> of
> >>>>>>>> files...
> >>>>>>>>>>
> >>>>>>>>>> content_repo/837/archive/144...-837
> >>>>>>>>>>
> >>>>>>>>>> Is this new in 3.0? My conf file says any archiving should be
> >>>>>> going
> >>>>>>>>>> into ./conf/archive, but i don't see anything in there.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Ryan
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
>
>
>
>
>
>
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
Random mentions of FileSystemRepo
nifi-app_2015-10-23_10.0.log:
org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
import data from java.io.ByteArrayInputStream@639fe470 for
StandardFlowFileRecord[uuid=881506e8-c62f-442f-a564-88675e4f0372,claim=,offset=0,name=914750081936080,size=0]
due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
to create ContentClaim due to java.io.IOException: Failed to write to
FileSystemRepository Stream [StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1445594395635-74075,
container=default, section=347], offset=0, length=-1]]
nifi-app_2015-10-23_10.0.log:Caused by:
org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
create ContentClaim due to java.io.IOException: Failed to write to
FileSystemRepository Stream [StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1445594395635-74075,
container=default, section=347], offset=0, length=-1]]
nifi-app_2015-10-23_10.0.log:Caused by: java.io.IOException: Failed to
write to FileSystemRepository Stream [StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1445594395635-74075,
container=default, section=347], offset=0, length=-1]]
nifi-app_2015-10-23_10.0.log: at
org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:913)
~[nifi-framework-core-0.3.0.jar:0.3.0]
nifi-app_2015-10-23_10.0.log: at
org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:689)
~[nifi-framework-core-0.3.0.jar:0.3.0]
nifi-app_2015-10-23_10.0.log: at
org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:910)
~[nifi-framework-core-0.3.0.jar:0.3.0]
nifi-app_2015-10-23_10.0.log:org.apache.nifi.processor.exception.FlowFileAccessException:
Failed to import data from /opt/nifi/logs/nifi-app_2015-10-23_05.0.log for
StandardFlowFileRecord[uuid=1b365988-1287-4d12-a31c-75c99dfc12ad,claim=,offset=0,name=914754786738219,size=0]
due to java.io.IOException: Failed to write to FileSystemRepository Stream
[StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1445594400001-74079,
container=default, section=351], offset=0, length=-1]]
On Fri, Oct 23, 2015 at 2:59 PM, Ryan H <rh...@gmail.com> wrote:
> Here's our provenance settings:
>
> # Persistent Provenance Repository Properties
> nifi.provenance.repository.directory.default=./provenance_repository
> nifi.provenance.repository.max.storage.time=24 hours
> nifi.provenance.repository.max.storage.size=1 GB
> nifi.provenance.repository.rollover.time=30 secs
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.journal.count=16
> # Comma-separated list of fields. Fields that are not indexed will not be
> searchable. Valid fields are:
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, ContentType, Relationship, Details
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
> Filename, ProcessorID, Relationship
> # FlowFile Attributes that should be indexed and made searchable
> nifi.provenance.repository.indexed.attributes=
> # Large values for the shard size will result in more Java heap usage when
> searching the Provenance Repository
> # but should provide better performance
> nifi.provenance.repository.index.shard.size=500 MB
> # Indicates the maximum length that a FlowFile attribute can be when
> retrieving a Provenance Event from
> # the repository. If the length of any attribute exceeds this value, it
> will be truncated when the event is retrieved.
> nifi.provenance.repository.max.attribute.length=65536
>
> # Volatile Provenance Respository Properties
> nifi.provenance.repository.buffer.size=100000
>
> On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz <
> eliezer_schwarz@yahoo.com.invalid> wrote:
>
>> We had max storage size of 1GB, but that's for provenance repo and our
>> problem was with content_repo. Our disk was 60GB, all on one partition, and
>> 55GB were taken up by content_repo. Now, it only contains 233MB.
>>
>>
>>
>> On Friday, October 23, 2015 2:50 PM, Mark Payne <
>> markap14@hotmail.com> wrote:
>>
>>
>>
>> OK, so this is interesting. Do you have your content repository and
>> provenance repository
>> both pointing to the same partition? What do you have the
>> "nifi.provenance.repository.max.storage.size"
>> property set to? How large is the actual disk?
>>
>> Thanks
>> -Mark
>>
>>
>> > On Oct 23, 2015, at 2:45 PM, Ryan H <rh...@gmail.com>
>> wrote:
>> >
>> > I've got this one... let me look for that
>> >
>> > 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
>> > o.a.n.p.PersistentProvenanceRepository
>> > java.io.IOException: No space left on device
>> > at java.io.FileOutputStream.writeBytes(Native Method)
>> ~[na:1.8.0_51]
>> > at java.io.FileOutputStream.write(FileOutputStream.java:326)
>> > ~[na:1.8.0_51]
>> > at
>> >
>> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> > java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
>> > ~[na:1.8.0_51]
>> > at
>> > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>> > ~[na:1.8.0_51]
>> > at
>> >
>> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> > org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> > org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
>> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>> > at
>> >
>> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
>> > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>> > at
>> >
>> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
>> > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>> > at
>> >
>> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
>> > [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>> >
>> > On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <ma...@hotmail.com>
>> wrote:
>> >
>> >> Ryan, Elli,
>> >>
>> >> Do you by chance have any error messages in your logs from the
>> >> FileSystemRepository?
>> >>
>> >> I.e., if you perform:
>> >>
>> >> grep FileSystemRepository logs/*
>> >>
>> >> Do you get anything interesting in there?
>> >>
>> >> Thanks
>> >> -Mark
>> >>
>> >>
>> >>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
>> >> <el...@yahoo.com.INVALID> wrote:
>> >>>
>> >>> I've been working with Ryan. There appear to be a few issues here:
>> >>>
>> >>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
>> >> content_repository archive is now true by default. In 0.2.0 it was
>> false,
>> >> and the documentation still states it is false by default.
>> >>> - When we ran out of disk space overnight, the problem was solved by
>> >> me simply restarting nifi, and that cleared out the archive by itself.
>> >>>
>> >>> - In order to clear up the archive, I had to set archive to true, and
>> >> set max usage to 1%, and restart nifi. That cleared it up, and then I
>> set
>> >> archive to false and restarted again so we don't run out of space.
>> >>> - Based on the above, it appears that something happened yesterday
>> >> that prevented Nifi from clearing out the archive even though disk
>> usage
>> >> reached 100%. However, restarting nifi apparently enabled it to
>> perform the
>> >> clearing of the archive. So apparently the max usage setting doesn't
>> work
>> >> under some conditions, but we don't know what conditions occurred
>> overnight
>> >> to cause this problem.
>> >>>
>> >>> Thanks!-Elli
>> >>>
>> >>>
>> >>>
>> >>> On Friday, October 23, 2015 2:29 PM, Ryan H <
>> >> rhendrickson.work@gmail.com> wrote:
>> >>>
>> >>>
>> >>>
>> >>> Agree, they concern the archive... although it sounds like there are 2
>> >>> archives?
>> >>>
>> >>> Within the content_repository folder, there are subfolders with the
>> name
>> >>> 'archive' and files inside them.
>> >>>
>> >>> Example:
>> >>> ./nfii/content_repository/837/archive/1445611320767-837
>> >>>
>> >>> Settings:
>> >>> nifi.content.repository.archive.max.retention.period=12 hours
>> >>> nifi.content.repository.archive.max.usage.percentage=50%
>> >>> nifi.content.repository.archive.enabled=true
>> >>>
>> >>> Last night, our server ran out of disk space because the
>> >> content_repository
>> >>> grew too large. Nifi didn't crash, but the log file contained errors
>> >>> saying the disk was full.
>> >>>
>> >>> We're not sure how, but the content_repository did not respect the
>> above
>> >>> settings.
>> >>>
>> >>> We restarted Nifi, and it only then started to remove files, such as:
>> >>> ./nfii/content_repository/837/archive/1445611320767-837
>> >>>
>> >>> We've turned off archiving for now.
>> >>>
>> >>> Ryan
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com>
>> >> wrote:
>> >>>
>> >>>> Ryan,
>> >>>>
>> >>>> Those items only concern the archive. Did you have data enqueued in
>> >>>> connections in your flow? If so, these items are not eligible and
>> could
>> >>>> explain why your disk was filled. Otherwise, can you please provide
>> >> some
>> >>>> additional information so we can dig into why this may have arisen.
>> >>>>
>> >>>> Thanks!
>> >>>>
>> >>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <
>> rhendrickson.work@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> I've got the following set:
>> >>>>>
>> >>>>> nifi.content.repository.archive.max.retention.period=12 hours
>> >>>>> nifi.content.repository.archive.max.usage.percentage=50%
>> >>>>> nifi.content.repository.archive.enabled=true
>> >>>>>
>> >>>>> Yet, the content repo filled my disk last night...
>> >>>>>
>> >>>>>
>> >>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>>> Ryan,
>> >>>>>>
>> >>>>>> Those archive folders map to the
>> >>>> nifi.content.repository.archive.enabled
>> >>>>>> property.
>> >>>>>>
>> >>>>>> What this property provides is a retention of files no longer in
>> the
>> >>>>> system
>> >>>>>> for historical context of your flow's processing and the ability
>> for
>> >>>>>> viewing this in conjunction with provenance events as well as
>> allowing
>> >>>>>> replay. The amount of the archive when enabled is bounded by the
>> >>>>>> properties nifi.content.repository.archive.max.retention.period and
>> >>>>>> nifi.content.repository.archive.max.usage.percentage.
>> >>>>>>
>> >>>>>> Additional detail is available in the system properties of our
>> >>>>>> Administration Guide [1]
>> >>>>>>
>> >>>>>> Let us know if you have additional questions.
>> >>>>>>
>> >>>>>> --aldrin
>> >>>>>>
>> >>>>>> [1]
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>> >>>>>>
>> >>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <
>> rhendrickson.work@gmail.com
>> >>>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Interesting.. So what would
>> >>>>>>>
>> >>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>> >>>>>>>
>> >>>>>>> typically be?
>> >>>>>>>
>> >>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
>> >>>>> agrande@hortonworks.com
>> >>>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Attachments don't go through, view at imagebin:
>> >>>>>>>> http://ibin.co/2K3SwR0z8yWX
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
>> >>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Ryan,
>> >>>>>>>>>
>> >>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
>> >>>> the
>> >>>>>>>> content repository data. See the attached screenshot (Settings
>> menu
>> >>>>> on
>> >>>>>>> the
>> >>>>>>>> right).
>> >>>>>>>>>
>> >>>>>>>>> Andrew
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
>> >>>> behalf
>> >>>>>> of
>> >>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
>> >>>>>>>> rhendrickson.work@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Hi,
>> >>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
>> >>>> of
>> >>>>>>>> files...
>> >>>>>>>>>>
>> >>>>>>>>>> content_repo/837/archive/144...-837
>> >>>>>>>>>>
>> >>>>>>>>>> Is this new in 3.0? My conf file says any archiving should be
>> >>>>>> going
>> >>>>>>>>>> into ./conf/archive, but i don't see anything in there.
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>> Ryan
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>>
>>
>>
>>
>>
>
>
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
Here's our provenance settings:
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
AlternateIdentifierURI, ContentType, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when
retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it
will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz <
eliezer_schwarz@yahoo.com.invalid> wrote:
> We had max storage size of 1GB, but that's for provenance repo and our
> problem was with content_repo. Our disk was 60GB, all on one partition, and
> 55GB were taken up by content_repo. Now, it only contains 233MB.
>
>
>
> On Friday, October 23, 2015 2:50 PM, Mark Payne <ma...@hotmail.com>
> wrote:
>
>
>
> OK, so this is interesting. Do you have your content repository and
> provenance repository
> both pointing to the same partition? What do you have the
> "nifi.provenance.repository.max.storage.size"
> property set to? How large is the actual disk?
>
> Thanks
> -Mark
>
>
> > On Oct 23, 2015, at 2:45 PM, Ryan H <rh...@gmail.com> wrote:
> >
> > I've got this one... let me look for that
> >
> > 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
> > o.a.n.p.PersistentProvenanceRepository
> > java.io.IOException: No space left on device
> > at java.io.FileOutputStream.writeBytes(Native Method)
> ~[na:1.8.0_51]
> > at java.io.FileOutputStream.write(FileOutputStream.java:326)
> > ~[na:1.8.0_51]
> > at
> >
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> > ~[na:1.8.0_51]
> > at
> > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> > ~[na:1.8.0_51]
> > at
> >
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
> > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> > at
> >
> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
> > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> > at
> >
> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
> > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> > at
> >
> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
> > [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> >
> > On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <ma...@hotmail.com>
> wrote:
> >
> >> Ryan, Elli,
> >>
> >> Do you by chance have any error messages in your logs from the
> >> FileSystemRepository?
> >>
> >> I.e., if you perform:
> >>
> >> grep FileSystemRepository logs/*
> >>
> >> Do you get anything interesting in there?
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
> >> <el...@yahoo.com.INVALID> wrote:
> >>>
> >>> I've been working with Ryan. There appear to be a few issues here:
> >>>
> >>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
> >> content_repository archive is now true by default. In 0.2.0 it was
> false,
> >> and the documentation still states it is false by default.
> >>> - When we ran out of disk space overnight, the problem was solved by
> >> me simply restarting nifi, and that cleared out the archive by itself.
> >>>
> >>> - In order to clear up the archive, I had to set archive to true, and
> >> set max usage to 1%, and restart nifi. That cleared it up, and then I
> set
> >> archive to false and restarted again so we don't run out of space.
> >>> - Based on the above, it appears that something happened yesterday
> >> that prevented Nifi from clearing out the archive even though disk usage
> >> reached 100%. However, restarting nifi apparently enabled it to perform
> the
> >> clearing of the archive. So apparently the max usage setting doesn't
> work
> >> under some conditions, but we don't know what conditions occurred
> overnight
> >> to cause this problem.
> >>>
> >>> Thanks!-Elli
> >>>
> >>>
> >>>
> >>> On Friday, October 23, 2015 2:29 PM, Ryan H <
> >> rhendrickson.work@gmail.com> wrote:
> >>>
> >>>
> >>>
> >>> Agree, they concern the archive... although it sounds like there are 2
> >>> archives?
> >>>
> >>> Within the content_repository folder, there are subfolders with the
> name
> >>> 'archive' and files inside them.
> >>>
> >>> Example:
> >>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>
> >>> Settings:
> >>> nifi.content.repository.archive.max.retention.period=12 hours
> >>> nifi.content.repository.archive.max.usage.percentage=50%
> >>> nifi.content.repository.archive.enabled=true
> >>>
> >>> Last night, our server ran out of disk space because the
> >> content_repository
> >>> grew too large. Nifi didn't crash, but the log file contained errors
> >>> saying the disk was full.
> >>>
> >>> We're not sure how, but the content_repository did not respect the
> above
> >>> settings.
> >>>
> >>> We restarted Nifi, and it only then started to remove files, such as:
> >>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>
> >>> We've turned off archiving for now.
> >>>
> >>> Ryan
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com>
> >> wrote:
> >>>
> >>>> Ryan,
> >>>>
> >>>> Those items only concern the archive. Did you have data enqueued in
> >>>> connections in your flow? If so, these items are not eligible and
> could
> >>>> explain why your disk was filled. Otherwise, can you please provide
> >> some
> >>>> additional information so we can dig into why this may have arisen.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rhendrickson.work@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>>> I've got the following set:
> >>>>>
> >>>>> nifi.content.repository.archive.max.retention.period=12 hours
> >>>>> nifi.content.repository.archive.max.usage.percentage=50%
> >>>>> nifi.content.repository.archive.enabled=true
> >>>>>
> >>>>> Yet, the content repo filled my disk last night...
> >>>>>
> >>>>>
> >>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Ryan,
> >>>>>>
> >>>>>> Those archive folders map to the
> >>>> nifi.content.repository.archive.enabled
> >>>>>> property.
> >>>>>>
> >>>>>> What this property provides is a retention of files no longer in the
> >>>>> system
> >>>>>> for historical context of your flow's processing and the ability for
> >>>>>> viewing this in conjunction with provenance events as well as
> allowing
> >>>>>> replay. The amount of the archive when enabled is bounded by the
> >>>>>> properties nifi.content.repository.archive.max.retention.period and
> >>>>>> nifi.content.repository.archive.max.usage.percentage.
> >>>>>>
> >>>>>> Additional detail is available in the system properties of our
> >>>>>> Administration Guide [1]
> >>>>>>
> >>>>>> Let us know if you have additional questions.
> >>>>>>
> >>>>>> --aldrin
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> >>>>>>
> >>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <
> rhendrickson.work@gmail.com
> >>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Interesting.. So what would
> >>>>>>>
> >>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>>>>>
> >>>>>>> typically be?
> >>>>>>>
> >>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> >>>>> agrande@hortonworks.com
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Attachments don't go through, view at imagebin:
> >>>>>>>> http://ibin.co/2K3SwR0z8yWX
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Ryan,
> >>>>>>>>>
> >>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
> >>>> the
> >>>>>>>> content repository data. See the attached screenshot (Settings
> menu
> >>>>> on
> >>>>>>> the
> >>>>>>>> right).
> >>>>>>>>>
> >>>>>>>>> Andrew
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
> >>>> behalf
> >>>>>> of
> >>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> >>>>>>>> rhendrickson.work@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
> >>>> of
> >>>>>>>> files...
> >>>>>>>>>>
> >>>>>>>>>> content_repo/837/archive/144...-837
> >>>>>>>>>>
> >>>>>>>>>> Is this new in 3.0? My conf file says any archiving should be
> >>>>>> going
> >>>>>>>>>> into ./conf/archive, but i don't see anything in there.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Ryan
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
>
>
>
>
>
Re: Content Repo Large.. Archive in there?
Posted by Elli Schwarz <el...@yahoo.com.INVALID>.
We had max storage size of 1GB, but that's for provenance repo and our problem was with content_repo. Our disk was 60GB, all on one partition, and 55GB were taken up by content_repo. Now, it only contains 233MB.
On Friday, October 23, 2015 2:50 PM, Mark Payne <ma...@hotmail.com> wrote:
OK, so this is interesting. Do you have your content repository and provenance repository
both pointing to the same partition? What do you have the "nifi.provenance.repository.max.storage.size"
property set to? How large is the actual disk?
Thanks
-Mark
> On Oct 23, 2015, at 2:45 PM, Ryan H <rh...@gmail.com> wrote:
>
> I've got this one... let me look for that
>
> 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
> o.a.n.p.PersistentProvenanceRepository
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_51]
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> ~[na:1.8.0_51]
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> ~[na:1.8.0_51]
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> ~[na:1.8.0_51]
> at
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
> [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>
> On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Ryan, Elli,
>>
>> Do you by chance have any error messages in your logs from the
>> FileSystemRepository?
>>
>> I.e., if you perform:
>>
>> grep FileSystemRepository logs/*
>>
>> Do you get anything interesting in there?
>>
>> Thanks
>> -Mark
>>
>>
>>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
>> <el...@yahoo.com.INVALID> wrote:
>>>
>>> I've been working with Ryan. There appear to be a few issues here:
>>>
>>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
>> content_repository archive is now true by default. In 0.2.0 it was false,
>> and the documentation still states it is false by default.
>>> - When we ran out of disk space overnight, the problem was solved by
>> me simply restarting nifi, and that cleared out the archive by itself.
>>>
>>> - In order to clear up the archive, I had to set archive to true, and
>> set max usage to 1%, and restart nifi. That cleared it up, and then I set
>> archive to false and restarted again so we don't run out of space.
>>> - Based on the above, it appears that something happened yesterday
>> that prevented Nifi from clearing out the archive even though disk usage
>> reached 100%. However, restarting nifi apparently enabled it to perform the
>> clearing of the archive. So apparently the max usage setting doesn't work
>> under some conditions, but we don't know what conditions occurred overnight
>> to cause this problem.
>>>
>>> Thanks!-Elli
>>>
>>>
>>>
>>> On Friday, October 23, 2015 2:29 PM, Ryan H <
>> rhendrickson.work@gmail.com> wrote:
>>>
>>>
>>>
>>> Agree, they concern the archive... although it sounds like there are 2
>>> archives?
>>>
>>> Within the content_repository folder, there are subfolders with the name
>>> 'archive' and files inside them.
>>>
>>> Example:
>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>
>>> Settings:
>>> nifi.content.repository.archive.max.retention.period=12 hours
>>> nifi.content.repository.archive.max.usage.percentage=50%
>>> nifi.content.repository.archive.enabled=true
>>>
>>> Last night, our server ran out of disk space because the
>> content_repository
>>> grew too large. Nifi didn't crash, but the log file contained errors
>>> saying the disk was full.
>>>
>>> We're not sure how, but the content_repository did not respect the above
>>> settings.
>>>
>>> We restarted Nifi, and it only then started to remove files, such as:
>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>
>>> We've turned off archiving for now.
>>>
>>> Ryan
>>>
>>>
>>>
>>>
>>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com>
>> wrote:
>>>
>>>> Ryan,
>>>>
>>>> Those items only concern the archive. Did you have data enqueued in
>>>> connections in your flow? If so, these items are not eligible and could
>>>> explain why your disk was filled. Otherwise, can you please provide
>> some
>>>> additional information so we can dig into why this may have arisen.
>>>>
>>>> Thanks!
>>>>
>>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rh...@gmail.com>
>>>> wrote:
>>>>
>>>>> I've got the following set:
>>>>>
>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>> nifi.content.repository.archive.enabled=true
>>>>>
>>>>> Yet, the content repo filled my disk last night...
>>>>>
>>>>>
>>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> Ryan,
>>>>>>
>>>>>> Those archive folders map to the
>>>> nifi.content.repository.archive.enabled
>>>>>> property.
>>>>>>
>>>>>> What this property provides is a retention of files no longer in the
>>>>> system
>>>>>> for historical context of your flow's processing and the ability for
>>>>>> viewing this in conjunction with provenance events as well as allowing
>>>>>> replay. The amount of the archive when enabled is bounded by the
>>>>>> properties nifi.content.repository.archive.max.retention.period and
>>>>>> nifi.content.repository.archive.max.usage.percentage.
>>>>>>
>>>>>> Additional detail is available in the system properties of our
>>>>>> Administration Guide [1]
>>>>>>
>>>>>> Let us know if you have additional questions.
>>>>>>
>>>>>> --aldrin
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>
>>>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>>>>>>
>>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.work@gmail.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Interesting.. So what would
>>>>>>>
>>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>>>>>
>>>>>>> typically be?
>>>>>>>
>>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
>>>>> agrande@hortonworks.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Attachments don't go through, view at imagebin:
>>>>>>>> http://ibin.co/2K3SwR0z8yWX
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ryan,
>>>>>>>>>
>>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
>>>> the
>>>>>>>> content repository data. See the attached screenshot (Settings menu
>>>>> on
>>>>>>> the
>>>>>>>> right).
>>>>>>>>>
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
>>>> behalf
>>>>>> of
>>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
>>>>>>>> rhendrickson.work@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
>>>> of
>>>>>>>> files...
>>>>>>>>>>
>>>>>>>>>> content_repo/837/archive/144...-837
>>>>>>>>>>
>>>>>>>>>> Is this new in 3.0? My conf file says any archiving should be
>>>>>> going
>>>>>>>>>> into ./conf/archive, but i don't see anything in there.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ryan
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>>
Re: Content Repo Large.. Archive in there?
Posted by Mark Payne <ma...@hotmail.com>.
OK, so this is interesting. Do you have your content repository and provenance repository
both pointing to the same partition? What do you have the "nifi.provenance.repository.max.storage.size"
property set to? How large is the actual disk?
Thanks
-Mark
> On Oct 23, 2015, at 2:45 PM, Ryan H <rh...@gmail.com> wrote:
>
> I've got this one... let me look for that
>
> 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
> o.a.n.p.PersistentProvenanceRepository
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_51]
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> ~[na:1.8.0_51]
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> ~[na:1.8.0_51]
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> ~[na:1.8.0_51]
> at
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
> at
> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
> [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>
> On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Ryan, Elli,
>>
>> Do you by chance have any error messages in your logs from the
>> FileSystemRepository?
>>
>> I.e., if you perform:
>>
>> grep FileSystemRepository logs/*
>>
>> Do you get anything interesting in there?
>>
>> Thanks
>> -Mark
>>
>>
>>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
>> <el...@yahoo.com.INVALID> wrote:
>>>
>>> I've been working with Ryan. There appear to be a few issues here:
>>>
>>> - We upgraded from 0.2.0 to 0.3.0 and it appears that
>> content_repository archive is now true by default. In 0.2.0 it was false,
>> and the documentation still states it is false by default.
>>> - When we ran out of disk space overnight, the problem was solved by
>> me simply restarting nifi, and that cleared out the archive by itself.
>>>
>>> - In order to clear up the archive, I had to set archive to true, and
>> set max usage to 1%, and restart nifi. That cleared it up, and then I set
>> archive to false and restarted again so we don't run out of space.
>>> - Based on the above, it appears that something happened yesterday
>> that prevented Nifi from clearing out the archive even though disk usage
>> reached 100%. However, restarting nifi apparently enabled it to perform the
>> clearing of the archive. So apparently the max usage setting doesn't work
>> under some conditions, but we don't know what conditions occurred overnight
>> to cause this problem.
>>>
>>> Thanks!-Elli
>>>
>>>
>>>
>>> On Friday, October 23, 2015 2:29 PM, Ryan H <
>> rhendrickson.work@gmail.com> wrote:
>>>
>>>
>>>
>>> Agree, they concern the archive... although it sounds like there are 2
>>> archives?
>>>
>>> Within the content_repository folder, there are subfolders with the name
>>> 'archive' and files inside them.
>>>
>>> Example:
>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>
>>> Settings:
>>> nifi.content.repository.archive.max.retention.period=12 hours
>>> nifi.content.repository.archive.max.usage.percentage=50%
>>> nifi.content.repository.archive.enabled=true
>>>
>>> Last night, our server ran out of disk space because the
>> content_repository
>>> grew too large. Nifi didn't crash, but the log file contained errors
>>> saying the disk was full.
>>>
>>> We're not sure how, but the content_repository did not respect the above
>>> settings.
>>>
>>> We restarted Nifi, and it only then started to remove files, such as:
>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>
>>> We've turned off archiving for now.
>>>
>>> Ryan
>>>
>>>
>>>
>>>
>>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com>
>> wrote:
>>>
>>>> Ryan,
>>>>
>>>> Those items only concern the archive. Did you have data enqueued in
>>>> connections in your flow? If so, these items are not eligible and could
>>>> explain why your disk was filled. Otherwise, can you please provide
>> some
>>>> additional information so we can dig into why this may have arisen.
>>>>
>>>> Thanks!
>>>>
>>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rh...@gmail.com>
>>>> wrote:
>>>>
>>>>> I've got the following set:
>>>>>
>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>> nifi.content.repository.archive.enabled=true
>>>>>
>>>>> Yet, the content repo filled my disk last night...
>>>>>
>>>>>
>>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> Ryan,
>>>>>>
>>>>>> Those archive folders map to the
>>>> nifi.content.repository.archive.enabled
>>>>>> property.
>>>>>>
>>>>>> What this property provides is a retention of files no longer in the
>>>>> system
>>>>>> for historical context of your flow's processing and the ability for
>>>>>> viewing this in conjunction with provenance events as well as allowing
>>>>>> replay. The amount of the archive when enabled is bounded by the
>>>>>> properties nifi.content.repository.archive.max.retention.period and
>>>>>> nifi.content.repository.archive.max.usage.percentage.
>>>>>>
>>>>>> Additional detail is available in the system properties of our
>>>>>> Administration Guide [1]
>>>>>>
>>>>>> Let us know if you have additional questions.
>>>>>>
>>>>>> --aldrin
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>
>>>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>>>>>>
>>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.work@gmail.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Interesting.. So what would
>>>>>>>
>>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>>>>>
>>>>>>> typically be?
>>>>>>>
>>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
>>>>> agrande@hortonworks.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Attachments don't go through, view at imagebin:
>>>>>>>> http://ibin.co/2K3SwR0z8yWX
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ryan,
>>>>>>>>>
>>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
>>>> the
>>>>>>>> content repository data. See the attached screenshot (Settings menu
>>>>> on
>>>>>>> the
>>>>>>>> right).
>>>>>>>>>
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
>>>> behalf
>>>>>> of
>>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
>>>>>>>> rhendrickson.work@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I'm noticing my Content Repo growing large. There's a number
>>>> of
>>>>>>>> files...
>>>>>>>>>>
>>>>>>>>>> content_repo/837/archive/144...-837
>>>>>>>>>>
>>>>>>>>>> Is this new in 3.0? My conf file says any archiving should be
>>>>>> going
>>>>>>>>>> into ./conf/archive, but i don't see anything in there.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ryan
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>>
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
I've got this one... let me look for that
2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
o.a.n.p.PersistentProvenanceRepository
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_51]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_51]
at
org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
~[na:1.8.0_51]
at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
~[na:1.8.0_51]
at
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
at
org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <ma...@hotmail.com> wrote:
> Ryan, Elli,
>
> Do you by chance have any error messages in your logs from the
> FileSystemRepository?
>
> I.e., if you perform:
>
> grep FileSystemRepository logs/*
>
> Do you get anything interesting in there?
>
> Thanks
> -Mark
>
>
> > On Oct 23, 2015, at 2:38 PM, Elli Schwarz
> <el...@yahoo.com.INVALID> wrote:
> >
> > I've been working with Ryan. There appear to be a few issues here:
> >
> > - We upgraded from 0.2.0 to 0.3.0 and it appears that
> content_repository archive is now true by default. In 0.2.0 it was false,
> and the documentation still states it is false by default.
> > - When we ran out of disk space overnight, the problem was solved by
> me simply restarting nifi, and that cleared out the archive by itself.
> >
> > - In order to clear up the archive, I had to set archive to true, and
> set max usage to 1%, and restart nifi. That cleared it up, and then I set
> archive to false and restarted again so we don't run out of space.
> > - Based on the above, it appears that something happened yesterday
> that prevented Nifi from clearing out the archive even though disk usage
> reached 100%. However, restarting nifi apparently enabled it to perform the
> clearing of the archive. So apparently the max usage setting doesn't work
> under some conditions, but we don't know what conditions occurred overnight
> to cause this problem.
> >
> > Thanks!-Elli
> >
> >
> >
> > On Friday, October 23, 2015 2:29 PM, Ryan H <
> rhendrickson.work@gmail.com> wrote:
> >
> >
> >
> > Agree, they concern the archive... although it sounds like there are 2
> > archives?
> >
> > Within the content_repository folder, there are subfolders with the name
> > 'archive' and files inside them.
> >
> > Example:
> > ./nfii/content_repository/837/archive/1445611320767-837
> >
> > Settings:
> > nifi.content.repository.archive.max.retention.period=12 hours
> > nifi.content.repository.archive.max.usage.percentage=50%
> > nifi.content.repository.archive.enabled=true
> >
> > Last night, our server ran out of disk space because the
> content_repository
> > grew too large. Nifi didn't crash, but the log file contained errors
> > saying the disk was full.
> >
> > We're not sure how, but the content_repository did not respect the above
> > settings.
> >
> > We restarted Nifi, and it only then started to remove files, such as:
> > ./nfii/content_repository/837/archive/1445611320767-837
> >
> > We've turned off archiving for now.
> >
> > Ryan
> >
> >
> >
> >
> > On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com>
> wrote:
> >
> >> Ryan,
> >>
> >> Those items only concern the archive. Did you have data enqueued in
> >> connections in your flow? If so, these items are not eligible and could
> >> explain why your disk was filled. Otherwise, can you please provide
> some
> >> additional information so we can dig into why this may have arisen.
> >>
> >> Thanks!
> >>
> >> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rh...@gmail.com>
> >> wrote:
> >>
> >>> I've got the following set:
> >>>
> >>> nifi.content.repository.archive.max.retention.period=12 hours
> >>> nifi.content.repository.archive.max.usage.percentage=50%
> >>> nifi.content.repository.archive.enabled=true
> >>>
> >>> Yet, the content repo filled my disk last night...
> >>>
> >>>
> >>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
> >> wrote:
> >>>
> >>>> Ryan,
> >>>>
> >>>> Those archive folders map to the
> >> nifi.content.repository.archive.enabled
> >>>> property.
> >>>>
> >>>> What this property provides is a retention of files no longer in the
> >>> system
> >>>> for historical context of your flow's processing and the ability for
> >>>> viewing this in conjunction with provenance events as well as allowing
> >>>> replay. The amount of the archive when enabled is bounded by the
> >>>> properties nifi.content.repository.archive.max.retention.period and
> >>>> nifi.content.repository.archive.max.usage.percentage.
> >>>>
> >>>> Additional detail is available in the system properties of our
> >>>> Administration Guide [1]
> >>>>
> >>>> Let us know if you have additional questions.
> >>>>
> >>>> --aldrin
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> >>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> >>>>
> >>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.work@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>>> Interesting.. So what would
> >>>>>
> >>>>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>>>
> >>>>> typically be?
> >>>>>
> >>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> >>> agrande@hortonworks.com
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Attachments don't go through, view at imagebin:
> >>>>>> http://ibin.co/2K3SwR0z8yWX
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> Ryan,
> >>>>>>>
> >>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
> >> the
> >>>>>> content repository data. See the attached screenshot (Settings menu
> >>> on
> >>>>> the
> >>>>>> right).
> >>>>>>>
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
> >> behalf
> >>>> of
> >>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> >>>>>> rhendrickson.work@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>> I'm noticing my Content Repo growing large. There's a number
> >> of
> >>>>>> files...
> >>>>>>>>
> >>>>>>>> content_repo/837/archive/144...-837
> >>>>>>>>
> >>>>>>>> Is this new in 3.0? My conf file says any archiving should be
> >>>> going
> >>>>>>>> into ./conf/archive, but i don't see anything in there.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Ryan
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> >
> >
>
>
Re: Content Repo Large.. Archive in there?
Posted by Mark Payne <ma...@hotmail.com>.
Ryan, Elli,
Do you by chance have any error messages in your logs from the FileSystemRepository?
I.e., if you perform:
grep FileSystemRepository logs/*
Do you get anything interesting in there?
Thanks
-Mark
> On Oct 23, 2015, at 2:38 PM, Elli Schwarz <el...@yahoo.com.INVALID> wrote:
>
> I've been working with Ryan. There appear to be a few issues here:
>
> - We upgraded from 0.2.0 to 0.3.0 and it appears that content_repository archive is now true by default. In 0.2.0 it was false, and the documentation still states it is false by default.
> - When we ran out of disk space overnight, the problem was solved by me simply restarting nifi, and that cleared out the archive by itself.
>
> - In order to clear up the archive, I had to set archive to true, and set max usage to 1%, and restart nifi. That cleared it up, and then I set archive to false and restarted again so we don't run out of space.
> - Based on the above, it appears that something happened yesterday that prevented Nifi from clearing out the archive even though disk usage reached 100%. However, restarting nifi apparently enabled it to perform the clearing of the archive. So apparently the max usage setting doesn't work under some conditions, but we don't know what conditions occurred overnight to cause this problem.
>
> Thanks!-Elli
>
>
>
> On Friday, October 23, 2015 2:29 PM, Ryan H <rh...@gmail.com> wrote:
>
>
>
> Agree, they concern the archive... although it sounds like there are 2
> archives?
>
> Within the content_repository folder, there are subfolders with the name
> 'archive' and files inside them.
>
> Example:
> ./nfii/content_repository/837/archive/1445611320767-837
>
> Settings:
> nifi.content.repository.archive.max.retention.period=12 hours
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=true
>
> Last night, our server ran out of disk space because the content_repository
> grew too large. Nifi didn't crash, but the log file contained errors
> saying the disk was full.
>
> We're not sure how, but the content_repository did not respect the above
> settings.
>
> We restarted Nifi, and it only then started to remove files, such as:
> ./nfii/content_repository/837/archive/1445611320767-837
>
> We've turned off archiving for now.
>
> Ryan
>
>
>
>
> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com> wrote:
>
>> Ryan,
>>
>> Those items only concern the archive. Did you have data enqueued in
>> connections in your flow? If so, these items are not eligible and could
>> explain why your disk was filled. Otherwise, can you please provide some
>> additional information so we can dig into why this may have arisen.
>>
>> Thanks!
>>
>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rh...@gmail.com>
>> wrote:
>>
>>> I've got the following set:
>>>
>>> nifi.content.repository.archive.max.retention.period=12 hours
>>> nifi.content.repository.archive.max.usage.percentage=50%
>>> nifi.content.repository.archive.enabled=true
>>>
>>> Yet, the content repo filled my disk last night...
>>>
>>>
>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
>> wrote:
>>>
>>>> Ryan,
>>>>
>>>> Those archive folders map to the
>> nifi.content.repository.archive.enabled
>>>> property.
>>>>
>>>> What this property provides is a retention of files no longer in the
>>> system
>>>> for historical context of your flow's processing and the ability for
>>>> viewing this in conjunction with provenance events as well as allowing
>>>> replay. The amount of the archive when enabled is bounded by the
>>>> properties nifi.content.repository.archive.max.retention.period and
>>>> nifi.content.repository.archive.max.usage.percentage.
>>>>
>>>> Additional detail is available in the system properties of our
>>>> Administration Guide [1]
>>>>
>>>> Let us know if you have additional questions.
>>>>
>>>> --aldrin
>>>>
>>>> [1]
>>>>
>>>>
>>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>>>>
>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Interesting.. So what would
>>>>>
>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>>>
>>>>> typically be?
>>>>>
>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
>>> agrande@hortonworks.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>> Attachments don't go through, view at imagebin:
>>>>>> http://ibin.co/2K3SwR0z8yWX
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
>>>> wrote:
>>>>>>
>>>>>>> Ryan,
>>>>>>>
>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
>> the
>>>>>> content repository data. See the attached screenshot (Settings menu
>>> on
>>>>> the
>>>>>> right).
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
>> behalf
>>>> of
>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
>>>>>> rhendrickson.work@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I'm noticing my Content Repo growing large. There's a number
>> of
>>>>>> files...
>>>>>>>>
>>>>>>>> content_repo/837/archive/144...-837
>>>>>>>>
>>>>>>>> Is this new in 3.0? My conf file says any archiving should be
>>>> going
>>>>>>>> into ./conf/archive, but i don't see anything in there.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ryan
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
>
Re: Content Repo Large.. Archive in there?
Posted by Elli Schwarz <el...@yahoo.com.INVALID>.
I've been working with Ryan. There appear to be a few issues here:
- We upgraded from 0.2.0 to 0.3.0 and it appears that content_repository archive is now true by default. In 0.2.0 it was false, and the documentation still states it is false by default.
- When we ran out of disk space overnight, the problem was solved by me simply restarting nifi, and that cleared out the archive by itself.
- In order to clear up the archive, I had to set archive to true, and set max usage to 1%, and restart nifi. That cleared it up, and then I set archive to false and restarted again so we don't run out of space.
- Based on the above, it appears that something happened yesterday that prevented Nifi from clearing out the archive even though disk usage reached 100%. However, restarting nifi apparently enabled it to perform the clearing of the archive. So apparently the max usage setting doesn't work under some conditions, but we don't know what conditions occurred overnight to cause this problem.
Thanks!-Elli
On Friday, October 23, 2015 2:29 PM, Ryan H <rh...@gmail.com> wrote:
Agree, they concern the archive... although it sounds like there are 2
archives?
Within the content_repository folder, there are subfolders with the name
'archive' and files inside them.
Example:
./nfii/content_repository/837/archive/1445611320767-837
Settings:
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
Last night, our server ran out of disk space because the content_repository
grew too large. Nifi didn't crash, but the log file contained errors
saying the disk was full.
We're not sure how, but the content_repository did not respect the above
settings.
We restarted Nifi, and it only then started to remove files, such as:
./nfii/content_repository/837/archive/1445611320767-837
We've turned off archiving for now.
Ryan
On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com> wrote:
> Ryan,
>
> Those items only concern the archive. Did you have data enqueued in
> connections in your flow? If so, these items are not eligible and could
> explain why your disk was filled. Otherwise, can you please provide some
> additional information so we can dig into why this may have arisen.
>
> Thanks!
>
> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rh...@gmail.com>
> wrote:
>
> > I've got the following set:
> >
> > nifi.content.repository.archive.max.retention.period=12 hours
> > nifi.content.repository.archive.max.usage.percentage=50%
> > nifi.content.repository.archive.enabled=true
> >
> > Yet, the content repo filled my disk last night...
> >
> >
> > On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
> wrote:
> >
> > > Ryan,
> > >
> > > Those archive folders map to the
> nifi.content.repository.archive.enabled
> > > property.
> > >
> > > What this property provides is a retention of files no longer in the
> > system
> > > for historical context of your flow's processing and the ability for
> > > viewing this in conjunction with provenance events as well as allowing
> > > replay. The amount of the archive when enabled is bounded by the
> > > properties nifi.content.repository.archive.max.retention.period and
> > > nifi.content.repository.archive.max.usage.percentage.
> > >
> > > Additional detail is available in the system properties of our
> > > Administration Guide [1]
> > >
> > > Let us know if you have additional questions.
> > >
> > > --aldrin
> > >
> > > [1]
> > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> > >
> > > On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rh...@gmail.com>
> > > wrote:
> > >
> > > > Interesting.. So what would
> > > >
> > > > ./nfii/content_repository/837/archive/1445611320767-837
> > > >
> > > > typically be?
> > > >
> > > > On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> > agrande@hortonworks.com
> > > >
> > > > wrote:
> > > >
> > > > > Attachments don't go through, view at imagebin:
> > > > > http://ibin.co/2K3SwR0z8yWX
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
> > > wrote:
> > > > >
> > > > > >Ryan,
> > > > > >
> > > > > >./conf/archive is to create a snapshot of your entire flow, not
> the
> > > > > content repository data. See the attached screenshot (Settings menu
> > on
> > > > the
> > > > > right).
> > > > > >
> > > > > >Andrew
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
> behalf
> > > of
> > > > > Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> > > > > rhendrickson.work@gmail.com> wrote:
> > > > > >
> > > > > >>Hi,
> > > > > >> I'm noticing my Content Repo growing large. There's a number
> of
> > > > > files...
> > > > > >>
> > > > > >>content_repo/837/archive/144...-837
> > > > > >>
> > > > > >> Is this new in 3.0? My conf file says any archiving should be
> > > going
> > > > > >>into ./conf/archive, but i don't see anything in there.
> > > > > >>
> > > > > >>Thanks,
> > > > > >>Ryan
> > > > >
> > > >
> > >
> >
>
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
Agree, they concern the archive... although it sounds like there are 2
archives?
Within the content_repository folder, there are subfolders with the name
'archive' and files inside them.
Example:
./nfii/content_repository/837/archive/1445611320767-837
Settings:
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
Last night, our server ran out of disk space because the content_repository
grew too large. Nifi didn't crash, but the log file contained errors
saying the disk was full.
We're not sure how, but the content_repository did not respect the above
settings.
We restarted Nifi, and it only then started to remove files, such as:
./nfii/content_repository/837/archive/1445611320767-837
We've turned off archiving for now.
Ryan
On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <al...@gmail.com> wrote:
> Ryan,
>
> Those items only concern the archive. Did you have data enqueued in
> connections in your flow? If so, these items are not eligible and could
> explain why your disk was filled. Otherwise, can you please provide some
> additional information so we can dig into why this may have arisen.
>
> Thanks!
>
> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rh...@gmail.com>
> wrote:
>
> > I've got the following set:
> >
> > nifi.content.repository.archive.max.retention.period=12 hours
> > nifi.content.repository.archive.max.usage.percentage=50%
> > nifi.content.repository.archive.enabled=true
> >
> > Yet, the content repo filled my disk last night...
> >
> >
> > On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com>
> wrote:
> >
> > > Ryan,
> > >
> > > Those archive folders map to the
> nifi.content.repository.archive.enabled
> > > property.
> > >
> > > What this property provides is a retention of files no longer in the
> > system
> > > for historical context of your flow's processing and the ability for
> > > viewing this in conjunction with provenance events as well as allowing
> > > replay. The amount of the archive when enabled is bounded by the
> > > properties nifi.content.repository.archive.max.retention.period and
> > > nifi.content.repository.archive.max.usage.percentage.
> > >
> > > Additional detail is available in the system properties of our
> > > Administration Guide [1]
> > >
> > > Let us know if you have additional questions.
> > >
> > > --aldrin
> > >
> > > [1]
> > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> > >
> > > On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rh...@gmail.com>
> > > wrote:
> > >
> > > > Interesting.. So what would
> > > >
> > > > ./nfii/content_repository/837/archive/1445611320767-837
> > > >
> > > > typically be?
> > > >
> > > > On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> > agrande@hortonworks.com
> > > >
> > > > wrote:
> > > >
> > > > > Attachments don't go through, view at imagebin:
> > > > > http://ibin.co/2K3SwR0z8yWX
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
> > > wrote:
> > > > >
> > > > > >Ryan,
> > > > > >
> > > > > >./conf/archive is to create a snapshot of your entire flow, not
> the
> > > > > content repository data. See the attached screenshot (Settings menu
> > on
> > > > the
> > > > > right).
> > > > > >
> > > > > >Andrew
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on
> behalf
> > > of
> > > > > Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> > > > > rhendrickson.work@gmail.com> wrote:
> > > > > >
> > > > > >>Hi,
> > > > > >> I'm noticing my Content Repo growing large. There's a number
> of
> > > > > files...
> > > > > >>
> > > > > >>content_repo/837/archive/144...-837
> > > > > >>
> > > > > >> Is this new in 3.0? My conf file says any archiving should be
> > > going
> > > > > >>into ./conf/archive, but i don't see anything in there.
> > > > > >>
> > > > > >>Thanks,
> > > > > >>Ryan
> > > > >
> > > >
> > >
> >
>
Re: Content Repo Large.. Archive in there?
Posted by Aldrin Piri <al...@gmail.com>.
Ryan,
Those items only concern the archive. Did you have data enqueued in
connections in your flow? If so, these items are not eligible and could
explain why your disk was filled. Otherwise, can you please provide some
additional information so we can dig into why this may have arisen.
Thanks!
On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rh...@gmail.com>
wrote:
> I've got the following set:
>
> nifi.content.repository.archive.max.retention.period=12 hours
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=true
>
> Yet, the content repo filled my disk last night...
>
>
> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com> wrote:
>
> > Ryan,
> >
> > Those archive folders map to the nifi.content.repository.archive.enabled
> > property.
> >
> > What this property provides is a retention of files no longer in the
> system
> > for historical context of your flow's processing and the ability for
> > viewing this in conjunction with provenance events as well as allowing
> > replay. The amount of the archive when enabled is bounded by the
> > properties nifi.content.repository.archive.max.retention.period and
> > nifi.content.repository.archive.max.usage.percentage.
> >
> > Additional detail is available in the system properties of our
> > Administration Guide [1]
> >
> > Let us know if you have additional questions.
> >
> > --aldrin
> >
> > [1]
> >
> >
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> >
> > On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rh...@gmail.com>
> > wrote:
> >
> > > Interesting.. So what would
> > >
> > > ./nfii/content_repository/837/archive/1445611320767-837
> > >
> > > typically be?
> > >
> > > On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> agrande@hortonworks.com
> > >
> > > wrote:
> > >
> > > > Attachments don't go through, view at imagebin:
> > > > http://ibin.co/2K3SwR0z8yWX
> > > >
> > > >
> > > >
> > > >
> > > > On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
> > wrote:
> > > >
> > > > >Ryan,
> > > > >
> > > > >./conf/archive is to create a snapshot of your entire flow, not the
> > > > content repository data. See the attached screenshot (Settings menu
> on
> > > the
> > > > right).
> > > > >
> > > > >Andrew
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on behalf
> > of
> > > > Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> > > > rhendrickson.work@gmail.com> wrote:
> > > > >
> > > > >>Hi,
> > > > >> I'm noticing my Content Repo growing large. There's a number of
> > > > files...
> > > > >>
> > > > >>content_repo/837/archive/144...-837
> > > > >>
> > > > >> Is this new in 3.0? My conf file says any archiving should be
> > going
> > > > >>into ./conf/archive, but i don't see anything in there.
> > > > >>
> > > > >>Thanks,
> > > > >>Ryan
> > > >
> > >
> >
>
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
I've got the following set:
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
Yet, the content repo filled my disk last night...
On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <al...@gmail.com> wrote:
> Ryan,
>
> Those archive folders map to the nifi.content.repository.archive.enabled
> property.
>
> What this property provides is a retention of files no longer in the system
> for historical context of your flow's processing and the ability for
> viewing this in conjunction with provenance events as well as allowing
> replay. The amount of the archive when enabled is bounded by the
> properties nifi.content.repository.archive.max.retention.period and
> nifi.content.repository.archive.max.usage.percentage.
>
> Additional detail is available in the system properties of our
> Administration Guide [1]
>
> Let us know if you have additional questions.
>
> --aldrin
>
> [1]
>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>
> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rh...@gmail.com>
> wrote:
>
> > Interesting.. So what would
> >
> > ./nfii/content_repository/837/archive/1445611320767-837
> >
> > typically be?
> >
> > On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <agrande@hortonworks.com
> >
> > wrote:
> >
> > > Attachments don't go through, view at imagebin:
> > > http://ibin.co/2K3SwR0z8yWX
> > >
> > >
> > >
> > >
> > > On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com>
> wrote:
> > >
> > > >Ryan,
> > > >
> > > >./conf/archive is to create a snapshot of your entire flow, not the
> > > content repository data. See the attached screenshot (Settings menu on
> > the
> > > right).
> > > >
> > > >Andrew
> > > >
> > > >
> > > >
> > > >
> > > >On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on behalf
> of
> > > Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> > > rhendrickson.work@gmail.com> wrote:
> > > >
> > > >>Hi,
> > > >> I'm noticing my Content Repo growing large. There's a number of
> > > files...
> > > >>
> > > >>content_repo/837/archive/144...-837
> > > >>
> > > >> Is this new in 3.0? My conf file says any archiving should be
> going
> > > >>into ./conf/archive, but i don't see anything in there.
> > > >>
> > > >>Thanks,
> > > >>Ryan
> > >
> >
>
Re: Content Repo Large.. Archive in there?
Posted by Aldrin Piri <al...@gmail.com>.
Ryan,
Those archive folders map to the nifi.content.repository.archive.enabled
property.
What this property provides is a retention of files no longer in the system
for historical context of your flow's processing and the ability for
viewing this in conjunction with provenance events as well as allowing
replay. The amount of the archive when enabled is bounded by the
properties nifi.content.repository.archive.max.retention.period and
nifi.content.repository.archive.max.usage.percentage.
Additional detail is available in the system properties of our
Administration Guide [1]
Let us know if you have additional questions.
--aldrin
[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rh...@gmail.com>
wrote:
> Interesting.. So what would
>
> ./nfii/content_repository/837/archive/1445611320767-837
>
> typically be?
>
> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <ag...@hortonworks.com>
> wrote:
>
> > Attachments don't go through, view at imagebin:
> > http://ibin.co/2K3SwR0z8yWX
> >
> >
> >
> >
> > On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com> wrote:
> >
> > >Ryan,
> > >
> > >./conf/archive is to create a snapshot of your entire flow, not the
> > content repository data. See the attached screenshot (Settings menu on
> the
> > right).
> > >
> > >Andrew
> > >
> > >
> > >
> > >
> > >On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on behalf of
> > Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> > rhendrickson.work@gmail.com> wrote:
> > >
> > >>Hi,
> > >> I'm noticing my Content Repo growing large. There's a number of
> > files...
> > >>
> > >>content_repo/837/archive/144...-837
> > >>
> > >> Is this new in 3.0? My conf file says any archiving should be going
> > >>into ./conf/archive, but i don't see anything in there.
> > >>
> > >>Thanks,
> > >>Ryan
> >
>
Re: Content Repo Large.. Archive in there?
Posted by Ryan H <rh...@gmail.com>.
Interesting.. So what would
./nfii/content_repository/837/archive/1445611320767-837
typically be?
On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <ag...@hortonworks.com>
wrote:
> Attachments don't go through, view at imagebin:
> http://ibin.co/2K3SwR0z8yWX
>
>
>
>
> On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com> wrote:
>
> >Ryan,
> >
> >./conf/archive is to create a snapshot of your entire flow, not the
> content repository data. See the attached screenshot (Settings menu on the
> right).
> >
> >Andrew
> >
> >
> >
> >
> >On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on behalf of
> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
> rhendrickson.work@gmail.com> wrote:
> >
> >>Hi,
> >> I'm noticing my Content Repo growing large. There's a number of
> files...
> >>
> >>content_repo/837/archive/144...-837
> >>
> >> Is this new in 3.0? My conf file says any archiving should be going
> >>into ./conf/archive, but i don't see anything in there.
> >>
> >>Thanks,
> >>Ryan
>
Re: Content Repo Large.. Archive in there?
Posted by Andrew Grande <ag...@hortonworks.com>.
Attachments don't go through, view at imagebin: http://ibin.co/2K3SwR0z8yWX
On 10/23/15, 12:52 PM, "Andrew Grande" <ag...@hortonworks.com> wrote:
>Ryan,
>
>./conf/archive is to create a snapshot of your entire flow, not the content repository data. See the attached screenshot (Settings menu on the right).
>
>Andrew
>
>
>
>
>On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on behalf of Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of rhendrickson.work@gmail.com> wrote:
>
>>Hi,
>> I'm noticing my Content Repo growing large. There's a number of files...
>>
>>content_repo/837/archive/144...-837
>>
>> Is this new in 3.0? My conf file says any archiving should be going
>>into ./conf/archive, but i don't see anything in there.
>>
>>Thanks,
>>Ryan
Re: Content Repo Large.. Archive in there?
Posted by Andrew Grande <ag...@hortonworks.com>.
Ryan,
./conf/archive is to create a snapshot of your entire flow, not the content repository data. See the attached screenshot (Settings menu on the right).
Andrew
On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com on behalf of Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of rhendrickson.work@gmail.com> wrote:
>Hi,
> I'm noticing my Content Repo growing large. There's a number of files...
>
>content_repo/837/archive/144...-837
>
> Is this new in 3.0? My conf file says any archiving should be going
>into ./conf/archive, but i don't see anything in there.
>
>Thanks,
>Ryan