You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Mark Bean <ma...@gmail.com> on 2021/09/22 14:26:31 UTC
Content repo not cleaning up with RocksDB flowfile repo
We are experiencing an issue where content claims are not being cleaned up.
This causes the content repo to grow, and eventually fill the disk. We only
see this behavior when using the RocksDBFlowFileRepository, but not when
using the WriteAheadFlowFileRepository.
Unfortunately, we have not been able to create a consistently reproducible
test case to recreate the problem. It is only observed on a cluster that is
heavily loaded.
Evidence of the problem (besides the content repo filling up) is that upon
restarting NiFi, there are hundreds of messages:
2021-09-22 11:39:50,052 INFO [main] 0.a.n.c.repository.FileSystemRepository
Found unknown file /content_repository/60/1632307980953-60 (814776 bytes)
in File System Repository; removing file
The crux of the problem is that the claimant count for a given claim is not
being updated correctly. The count never reaches zero even when all
flowfiles which had been in the given claim have been dropped from NiFi.
(This is why the claim is able to be removed on restart.) Therefore, the
claim is considered "in use" and is not cleaned up. (Content archiving is
disabled as well, so no claims are advanced to the archive.)
The issue is not seen in 1.11.4, but is seen in 1.12.1 and 1.14.0. I
suspect it may be related to changes made in NIFI-6849.
https://issues.apache.org/jira/browse/NIFI-6849
Any insights or suggestions are greatly appreciated.
Thanks!
-Mark
Re: Content repo not cleaning up with RocksDB flowfile repo
Posted by Eric R <el...@gmail.com>.
https://issues.apache.org/jira/browse/NIFI-9056
On Wed, Sep 22, 2021 at 10:26 AM Mark Bean <ma...@gmail.com> wrote:
> We are experiencing an issue where content claims are not being cleaned up.
> This causes the content repo to grow, and eventually fill the disk. We only
> see this behavior when using the RocksDBFlowFileRepository, but not when
> using the WriteAheadFlowFileRepository.
>
> Unfortunately, we have not been able to create a consistently reproducible
> test case to recreate the problem. It is only observed on a cluster that is
> heavily loaded.
>
> Evidence of the problem (besides the content repo filling up) is that upon
> restarting NiFi, there are hundreds of messages:
>
> 2021-09-22 11:39:50,052 INFO [main] 0.a.n.c.repository.FileSystemRepository
> Found unknown file /content_repository/60/1632307980953-60 (814776 bytes)
> in File System Repository; removing file
>
> The crux of the problem is that the claimant count for a given claim is not
> being updated correctly. The count never reaches zero even when all
> flowfiles which had been in the given claim have been dropped from NiFi.
> (This is why the claim is able to be removed on restart.) Therefore, the
> claim is considered "in use" and is not cleaned up. (Content archiving is
> disabled as well, so no claims are advanced to the archive.)
>
> The issue is not seen in 1.11.4, but is seen in 1.12.1 and 1.14.0. I
> suspect it may be related to changes made in NIFI-6849.
> https://issues.apache.org/jira/browse/NIFI-6849
>
> Any insights or suggestions are greatly appreciated.
>
> Thanks!
> -Mark
>
Re: Content repo not cleaning up with RocksDB flowfile repo
Posted by Phillip Lord <ph...@gmail.com>.
Unfortunately this is also happening with the WriteAhead repo implementation. The amount of claims in the content-repo keeps growing uncontrollably.
On 2021/09/22 14:26:31, Mark Bean <ma...@gmail.com> wrote:
> We are experiencing an issue where content claims are not being cleaned up.
> This causes the content repo to grow, and eventually fill the disk. We only
> see this behavior when using the RocksDBFlowFileRepository, but not when
> using the WriteAheadFlowFileRepository.
>
> Unfortunately, we have not been able to create a consistently reproducible
> test case to recreate the problem. It is only observed on a cluster that is
> heavily loaded.
>
> Evidence of the problem (besides the content repo filling up) is that upon
> restarting NiFi, there are hundreds of messages:
>
> 2021-09-22 11:39:50,052 INFO [main] 0.a.n.c.repository.FileSystemRepository
> Found unknown file /content_repository/60/1632307980953-60 (814776 bytes)
> in File System Repository; removing file
>
> The crux of the problem is that the claimant count for a given claim is not
> being updated correctly. The count never reaches zero even when all
> flowfiles which had been in the given claim have been dropped from NiFi.
> (This is why the claim is able to be removed on restart.) Therefore, the
> claim is considered "in use" and is not cleaned up. (Content archiving is
> disabled as well, so no claims are advanced to the archive.)
>
> The issue is not seen in 1.11.4, but is seen in 1.12.1 and 1.14.0. I
> suspect it may be related to changes made in NIFI-6849.
> https://issues.apache.org/jira/browse/NIFI-6849
>
> Any insights or suggestions are greatly appreciated.
>
> Thanks!
> -Mark
>