[jira] [Commented] (NIFI-16238) Allow specifying a max journal size for flowfile repo and local state

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106564#comment-18106564 ] 

ASF subversion and git services commented on NIFI-16238:
--------------------------------------------------------

Commit db732669a521b3df05b4ee1f2ef1cb1d37c78846 in nifi's branch refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=db732669a52 ]

NIFI-16238: Checkpoint write-ahead logs when the journal reaches a co… (#11575)

Journals could grow without bound between interval-based checkpoints. A maximum journal size now triggers a checkpoint so storage is reclaimed even when updates keep arriving.

> Allow specifying a max journal size for flowfile repo and local state
> ---------------------------------------------------------------------
>
>                 Key: NIFI-16238
>                 URL: https://issues.apache.org/jira/browse/NIFI-16238
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Both the local state provider and the FlowFile Repository make use of the {{SequentialAccessWriteAheadLog}} and checkpoint periodically based on a time threshold. But under heavy load we can write significant amounts of data, resulting on {{IOException: No space left on device}} issues. We should allow configuring a max journal size and if configured checkpoint when the journal exceeds that threshold. This allows us to avoid overfilling the disk even under heavy load.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)