You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Wiktor Kubicki (Jira)" <ji...@apache.org> on 2021/12/09 12:28:00 UTC
[jira] [Updated] (NIFI-9464) Provenance Events files corrupted
[ https://issues.apache.org/jira/browse/NIFI-9464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wiktor Kubicki updated NIFI-9464:
---------------------------------
Description:
In my logs i found:
{code:java}
SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544] Failed to retrieve Provenance Events from repository due to: Attempted to skip to byte offset 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC Reader=StandardTocReader[file=/..../provenance_repository/toc/1125432890.toc, compressed=false]): java.io.EOFException: Attempted to skip to byte offset 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc, compressed=false])
{code}
It happened about 100 times in last 1 year for 15 *.prov.gz files. It is criticaly important for me to have 100% sure of my logs.
My config for provenance (BTW if you see posibbility for tune it, please tell me):
{code:java}
nifi.provenance.repository.directory.default=/....../provenance_repository
nifi.provenance.repository.max.storage.time=730 days
nifi.provenance.repository.max.storage.size=512 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.index.shard.size=1 GB
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=1
nifi.provenance.repository.buffer.size=100000
{code}
Now my provenance repo has 140GB of data.
was:
In my logs i found:
{code:java}
SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544] Failed to retrieve Provenance Events from repository due to: Attempted to skip to byte offset 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC Reader=StandardTocReader[file=/..../provenance_repository/toc/1125432890.toc, compressed=false]): java.io.EOFException: Attempted to skip to byte offset 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc, compressed=false])
{code}
It happened about 100 times in last 1 year. It is criticaly important for me to have 100% sure of my logs.
My config for provenance (BTW if you see posibbility for tune it, tell me):
{code:java}
nifi.provenance.repository.directory.default=/....../provenance_repository
nifi.provenance.repository.max.storage.time=730 days
nifi.provenance.repository.max.storage.size=512 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.index.shard.size=1 GB
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=1
nifi.provenance.repository.buffer.size=100000
{code}
Now my provenance repo has 140GB of data.
> Provenance Events files corrupted
> ---------------------------------
>
> Key: NIFI-9464
> URL: https://issues.apache.org/jira/browse/NIFI-9464
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.11.0, 1.15.0
> Environment: java 11, centos 7, nifi standalone
> Reporter: Wiktor Kubicki
> Priority: Minor
>
> In my logs i found:
> {code:java}
> SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544] Failed to retrieve Provenance Events from repository due to: Attempted to skip to byte offset 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC Reader=StandardTocReader[file=/..../provenance_repository/toc/1125432890.toc, compressed=false]): java.io.EOFException: Attempted to skip to byte offset 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc, compressed=false])
> {code}
> It happened about 100 times in last 1 year for 15 *.prov.gz files. It is criticaly important for me to have 100% sure of my logs.
> My config for provenance (BTW if you see posibbility for tune it, please tell me):
> {code:java}
> nifi.provenance.repository.directory.default=/....../provenance_repository
> nifi.provenance.repository.max.storage.time=730 days
> nifi.provenance.repository.max.storage.size=512 GB
> nifi.provenance.repository.rollover.time=10 mins
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID
> nifi.provenance.repository.indexed.attributes=
> nifi.provenance.repository.index.shard.size=1 GB
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=1
> nifi.provenance.repository.buffer.size=100000
> {code}
> Now my provenance repo has 140GB of data.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)