You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Pablo Lopez <pa...@integrado.com.au> on 2016/12/14 05:43:30 UTC
Data Provenance - Disk Full
Hi,
We've been loading quite a few big files through NiFi, but now it looks
that the file disk is full and we get this error in most places and nothing
gets processed.
"Failed to rollover Provenance Event Log due to java.io.IOException no
space left on device"
Is there a way to manually delete all the provenance entries from the
repository so we can continue processing?
Also, is there a way to disable data provenance for big files?
Thanks for your help,
Pablo.
Re: Data Provenance - Disk Full
Posted by Lee Laim <le...@gmail.com>.
Pablo,
The nifi.provenance.repository.max.storage.size parameter will delete
oldest provenance events once the 1 GB limit is reached.
The provenance_repo footprint is small compared to the content_repo. While
you're seeing a provenance error, the culprit may be excessive data in
content_repo filling the disk.
Check the following parameters on the content repo. You can disable
archiving or reduce max retention period to ease your disk usage.
*nifi.content.repository.archive.max.retention.period*
*If archiving is enabled (see nifi.content.repository.archive.enabled
below), then this property specifies the maximum amount of time to keep the
archived data. It is 12 hours by default.*
*nifi.content.repository.archive.max.usage.percentage*
*If archiving is enabled (see nifi.content.repository.archive.enabled
below), then this property also must have a value to indicate the maximum
percentage of disk space that may be used before archive data is removed.
If this value is already met even before archiving then arhival will not be
of much use. It is 50% by default.*
*nifi.content.repository.archive.enabled*
*To enable archiving, set this to true and specify a value for the
nifi.content.repository.archive.max.usage.percentage property above. By
default, archiving is enabled.*
Lastly, you might want to try to schedule smaller batches of files into the
flow, especially if there are hardware limitations. This will take some of
the pressure off the content and provenance repository during a data surge.
This is a reference I find helpful:
https://community.hortonworks.com/questions/22517/nifi-repository-typical-disk-usage-ratios-among-th.html
Thanks,
Lee
On Wed, Dec 14, 2016 at 4:20 AM, Pablo Lopez <pa...@integrado.com.au>
wrote:
> Hi,
>
> I've deleted the files as suggested by James, but I would also like to
> fine tune so this doesn't happen again.
> However, I'm not sure what exactly happens with parameter: nifi.provenance.
> repository.max.storage.size
>
> I haven't changed the default (1 GB), so what would happen when the limit
> is reached? Does NiFi stops storing the provenance and someone has to
> manually delete the files or does it cycle again over the 1 GB overwriting
> the existing files?
>
> Thanks,
> Pablo.
>
> On Wed, Dec 14, 2016 at 5:31 PM, James Wing <jv...@gmail.com> wrote:
>
>> Pablo,
>>
>> An extreme fix is to stop stop NiFi and completely delete the provenance
>> repository directory. You will get a new, empty provenance repository when
>> you restart NiFi. All data will be lost, but NiFi will work again.
>>
>> Obviously, Lee's advice is much better, but there are options if you need
>> them.
>>
>> James
>>
>> On Tue, Dec 13, 2016 at 10:23 PM, Lee Laim <le...@gmail.com> wrote:
>>
>>> Pablo,
>>>
>>> You can decrease the following attributes in conf/nifi.properties.
>>>
>>> nifi.provenance.repository.max.storage.time
>>>
>>> The maximum amount of time to keep data provenance information. The
>>> default value is 24 hours.
>>>
>>> nifi.provenance.repository.max.storage.size
>>>
>>> The maximum amount of data provenance information to store at a time.
>>> The default is 1 GB.
>>> https://nifi.apache.org/docs/nifi-docs/html/administration-g
>>> uide.html#flowfile-repository
>>>
>>>
>>> Thanks,
>>> Lee
>>>
>>>
>>>
>>> On Tue, Dec 13, 2016 at 10:43 PM, Pablo Lopez <
>>> pablo.lopez@integrado.com.au> wrote:
>>>
>>>> Hi,
>>>>
>>>> We've been loading quite a few big files through NiFi, but now it looks
>>>> that the file disk is full and we get this error in most places and nothing
>>>> gets processed.
>>>>
>>>> "Failed to rollover Provenance Event Log due to java.io.IOException no
>>>> space left on device"
>>>>
>>>> Is there a way to manually delete all the provenance entries from the
>>>> repository so we can continue processing?
>>>> Also, is there a way to disable data provenance for big files?
>>>>
>>>> Thanks for your help,
>>>> Pablo.
>>>>
>>>
>>>
>>
>
>
> --
> Pablo Lopez.
> Integration Architect
> Integrado Pty Ltd
> M: 044 84 52 479
> pablo.lopez@integrado.com.au
>
Re: Data Provenance - Disk Full
Posted by Pablo Lopez <pa...@integrado.com.au>.
Hi,
I've deleted the files as suggested by James, but I would also like to fine
tune so this doesn't happen again.
However, I'm not sure what exactly happens with
parameter: nifi.provenance.repository.max.storage.size
I haven't changed the default (1 GB), so what would happen when the limit
is reached? Does NiFi stops storing the provenance and someone has to
manually delete the files or does it cycle again over the 1 GB overwriting
the existing files?
Thanks,
Pablo.
On Wed, Dec 14, 2016 at 5:31 PM, James Wing <jv...@gmail.com> wrote:
> Pablo,
>
> An extreme fix is to stop stop NiFi and completely delete the provenance
> repository directory. You will get a new, empty provenance repository when
> you restart NiFi. All data will be lost, but NiFi will work again.
>
> Obviously, Lee's advice is much better, but there are options if you need
> them.
>
> James
>
> On Tue, Dec 13, 2016 at 10:23 PM, Lee Laim <le...@gmail.com> wrote:
>
>> Pablo,
>>
>> You can decrease the following attributes in conf/nifi.properties.
>>
>> nifi.provenance.repository.max.storage.time
>>
>> The maximum amount of time to keep data provenance information. The
>> default value is 24 hours.
>>
>> nifi.provenance.repository.max.storage.size
>>
>> The maximum amount of data provenance information to store at a time. The
>> default is 1 GB.
>> https://nifi.apache.org/docs/nifi-docs/html/administration-g
>> uide.html#flowfile-repository
>>
>>
>> Thanks,
>> Lee
>>
>>
>>
>> On Tue, Dec 13, 2016 at 10:43 PM, Pablo Lopez <
>> pablo.lopez@integrado.com.au> wrote:
>>
>>> Hi,
>>>
>>> We've been loading quite a few big files through NiFi, but now it looks
>>> that the file disk is full and we get this error in most places and nothing
>>> gets processed.
>>>
>>> "Failed to rollover Provenance Event Log due to java.io.IOException no
>>> space left on device"
>>>
>>> Is there a way to manually delete all the provenance entries from the
>>> repository so we can continue processing?
>>> Also, is there a way to disable data provenance for big files?
>>>
>>> Thanks for your help,
>>> Pablo.
>>>
>>
>>
>
--
Pablo Lopez.
Integration Architect
Integrado Pty Ltd
M: 044 84 52 479
pablo.lopez@integrado.com.au
Re: Data Provenance - Disk Full
Posted by James Wing <jv...@gmail.com>.
Pablo,
An extreme fix is to stop stop NiFi and completely delete the provenance
repository directory. You will get a new, empty provenance repository when
you restart NiFi. All data will be lost, but NiFi will work again.
Obviously, Lee's advice is much better, but there are options if you need
them.
James
On Tue, Dec 13, 2016 at 10:23 PM, Lee Laim <le...@gmail.com> wrote:
> Pablo,
>
> You can decrease the following attributes in conf/nifi.properties.
>
> nifi.provenance.repository.max.storage.time
>
> The maximum amount of time to keep data provenance information. The
> default value is 24 hours.
>
> nifi.provenance.repository.max.storage.size
>
> The maximum amount of data provenance information to store at a time. The
> default is 1 GB.
> https://nifi.apache.org/docs/nifi-docs/html/administration-
> guide.html#flowfile-repository
>
>
> Thanks,
> Lee
>
>
>
> On Tue, Dec 13, 2016 at 10:43 PM, Pablo Lopez <
> pablo.lopez@integrado.com.au> wrote:
>
>> Hi,
>>
>> We've been loading quite a few big files through NiFi, but now it looks
>> that the file disk is full and we get this error in most places and nothing
>> gets processed.
>>
>> "Failed to rollover Provenance Event Log due to java.io.IOException no
>> space left on device"
>>
>> Is there a way to manually delete all the provenance entries from the
>> repository so we can continue processing?
>> Also, is there a way to disable data provenance for big files?
>>
>> Thanks for your help,
>> Pablo.
>>
>
>
Re: Data Provenance - Disk Full
Posted by Lee Laim <le...@gmail.com>.
Pablo,
You can decrease the following attributes in conf/nifi.properties.
nifi.provenance.repository.max.storage.time
The maximum amount of time to keep data provenance information. The default
value is 24 hours.
nifi.provenance.repository.max.storage.size
The maximum amount of data provenance information to store at a time. The
default is 1 GB.
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#flowfile-repository
Thanks,
Lee
On Tue, Dec 13, 2016 at 10:43 PM, Pablo Lopez <pa...@integrado.com.au>
wrote:
> Hi,
>
> We've been loading quite a few big files through NiFi, but now it looks
> that the file disk is full and we get this error in most places and nothing
> gets processed.
>
> "Failed to rollover Provenance Event Log due to java.io.IOException no
> space left on device"
>
> Is there a way to manually delete all the provenance entries from the
> repository so we can continue processing?
> Also, is there a way to disable data provenance for big files?
>
> Thanks for your help,
> Pablo.
>