You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "markap14 (via GitHub)" <gi...@apache.org> on 2026/10/23 00:02:00 UTC

[PR] NIFI-16494 Implemented partial component state updates in framework [nifi]

markap14 opened a new pull request, #11837:
URL: https://github.com/apache/nifi/pull/11837

   <!-- SPDX-License-Identifier: Apache-2.0 -->
   
   # Summary
   
   [NIFI-16494](https://issues.apache.org/jira/browse/NIFI-16494)
   
   Implements the framework portion of [NIP-50](https://issues.apache.org/jira/browse/NIP-50). This depends on apache/nifi-api#112 ([NIFI-16493](https://issues.apache.org/jira/browse/NIFI-16493)) and sets `nifi-api.version` to `2.15.0-SNAPSHOT`, so it stays a draft until that PR is merged and published.
   
   - `StateProvider` gets a default `patchState` that retrieves the current state, applies the changes, and calls `replace` until it succeeds. Providers that can update individual keys can override it.
   - `StandardProcessSession` remembers whether the pending state update for each scope is a whole-map update (from `setState`) or a patch (from `patchState`). On commit, a patch is sent to the `StateManager` as a `patchState` call containing only the patched keys. The patch is applied even if another writer changed the stored state since the session read it, and keys outside the patch keep their stored values. Whole-map updates keep the existing version check. Calling `setState` after `patchState` in the same session replaces the patch with the new whole map.
   - `StandardStateManager`, `TaskTerminationAwareStateManager`, `StandardStateManagerProvider`, `ManagedStateProvider`, `BatchingSessionFactory`, `WeakHashMapProcessSessionFactory`, the Groovyx `ProcessSessionWrap`, and `MockProcessSession` pass `patchState` through instead of falling back to a whole-map replacement.
   - `WriteAheadLocalStateProvider` writes only the patched keys to its journal. `StateMapSerDe` moves to version 1, which can record either a full state map or a patch applied on top of the previous record for that component. Snapshots still hold full state maps. Journals written by version 0 can still be read. NiFi versions without this change cannot read a version 1 journal, so going back to an older NiFi after state has been written is not supported.
   - The `VerifyLocalClusterStateService` and `MultiKeyState` system test extensions use `patchState`, and the Developer Guide describes it.
   
   Tests:
   
   - `StandardProcessSessionIT` checks that a session patch reaches the `StateManager` as a `patchState` call, that several patches in one session are combined, that `setState` after `patchState` replaces the patch, and that a patch is applied on commit even after another writer updated the state.
   - `TestStateMapSerDe` checks that a patch journal entry round-trips on top of a previous record.
   - `TestWriteAheadLocalStateProvider` checks that a patch is written to the journal as only the patched keys and that state recovers correctly after restart.
   - `AbstractTestStateProvider`, `KubernetesConfigMapStateProviderTest`, `TestMockProcessSession`, and `ClusterStateKeyDropIT` cover `patchState` for the other providers, the mock session, and a cluster.
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
   
   ### Pull Request Tracking
   
   - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
   - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
   - [x] Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status
   
   ### Pull Request Formatting
   
   - [x] Pull Request based on current revision of the `main` branch
   - [x] Pull Request refers to a feature branch with one commit containing changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request creation.
   
   ### Build
   
   - [ ] Build completed using `./mvnw clean install -P contrib-check`
     - [ ] JDK 21
     - [ ] JDK 25
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
   
   ### Documentation
   
   - [x] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-16494 Implemented partial component state updates in framework [nifi]

Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #11837:
URL: https://github.com/apache/nifi/pull/11837#discussion_r4401948462


##########
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java:
##########
@@ -1405,21 +1418,64 @@ public ProvenanceReporter getProvenanceReporter() {
     @Override
     public void setState(final Map<String, String> state, final Scope scope) throws IOException {
         stateManager.setState(state, scope);
+        pendingStatePatches.remove(scope);
     }
 
     @Override
     public StateMap getState(final Scope scope) throws IOException {
-        return stateManager.getState(scope);
+        final StateMap storedState = stateManager.getState(scope);
+        final Map<String, String> patches = pendingStatePatches.get(scope);
+        if (patches == null) {
+            return storedState;
+        }
+
+        final long version = storedState.getStateVersion().map(Long::parseLong).orElse(-1L);

Review Comment:
   [Claude Opus 5.5] Good catch. `MockProcessSession` now counts the patches it is holding for each scope, and `getState` returns a version equal to the stored version plus that count, the same way `StandardProcessSession` gives the session's state a new version on each patch. A `StateMap` obtained before a patch therefore no longer matches in `replaceState`, even if the patch wrote the same values. `TestMockProcessSession.RegardingState.canSetLocalStateWhenDeclaredStateful` now checks this case.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java:
##########
@@ -2597,12 +2682,14 @@ public void testRollbackAfterCheckpointStoresState() throws IOException {
         session.checkpoint();
 
         session.setState(Collections.singletonMap("abc", "2"), Scope.LOCAL);
+        session.patchState(Map.of("def", "2"), Scope.LOCAL);

Review Comment:
   [Claude Opus 5.5] Done. `testRollbackAfterCheckpointStoresState` now patches `abc`, checkpoints, patches `def`, rolls back, and then commits. It verifies that `patchState` is called on the state manager with only `{abc=1}`, and that the stored state is `{abc=1}`, so the patch made after the checkpoint is discarded and the checkpointed patch is kept.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/StateMapSerDe.java:
##########
@@ -28,14 +28,33 @@
 import java.util.Map;
 import java.util.Optional;
 
+/**
+ * Serializes component state for the write-ahead log. Full records, which are used for snapshots, always hold the whole state map.
+ * Starting with version 1, an edit in the journal holds either the whole state map or only the patches. An edit that holds
+ * only the patches is applied on top of the component's previous record when the journal is recovered. Version 0 edits always
+ * hold the whole state map, in the same layout as full records.
+ */
 public class StateMapSerDe implements SerDe<StateMapUpdate> {
     private static final long EMPTY_VERSION = -1;
 
-    private static final int VERSION = 0;
+    private static final int PATCHES_VERSION = 1;

Review Comment:
   [Claude Opus 5.5] Agreed that this needs a migration note. I don't have edit access to the wiki, so here is suggested text for the migration guide. Could you or someone with access add it?
   
   > Local component state written by this version uses a newer write-ahead log format (serializer version 1). Older versions of NiFi cannot read it and fail to start with an error saying the journal was written by a newer serializer version. To downgrade, restore a backup of the `state/local` directory taken before the upgrade, or remove that directory, which discards all local component state.
   



##########
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java:
##########
@@ -1405,21 +1418,70 @@ public ProvenanceReporter getProvenanceReporter() {
     @Override
     public void setState(final Map<String, String> state, final Scope scope) throws IOException {
         stateManager.setState(state, scope);
+        pendingStatePatches.remove(scope);
     }
 
+    /**
+     * Returns the stored state with any patches that have not yet been committed applied. Each patch gives the returned state a new version,
+     * as the framework does, so that a state obtained before a patch no longer matches in {@link #replaceState(StateMap, Map, Scope)}.
+     */
     @Override
     public StateMap getState(final Scope scope) throws IOException {
-        return stateManager.getState(scope);
+        final StateMap storedState = stateManager.getState(scope);
+        final PendingStatePatches patches = pendingStatePatches.get(scope);
+        if (patches == null) {
+            return storedState;
+        }
+
+        final long storedVersion = storedState.getStateVersion().map(Long::parseLong).orElse(-1L);
+        return new MockStateMap(storedState.toMap(patches.values), storedVersion + patches.count);
     }
 
+    /**
+     * Replaces the state if the given old value matches the session's view of the state. When the session holds patches that have not yet
+     * been committed, the old value must match the stored state with those patches applied, and a successful replacement discards the patches.
+     */
     @Override
     public boolean replaceState(final StateMap oldValue, final Map<String, String> newValue, final Scope scope) throws IOException {
-        return stateManager.replace(oldValue, newValue, scope);
+        if (!pendingStatePatches.containsKey(scope)) {
+            return stateManager.replace(oldValue, newValue, scope);
+        }
+
+        final StateMap currentState = getState(scope);
+        if (oldValue == null || !oldValue.getStateVersion().equals(currentState.getStateVersion()) || !oldValue.toMap().equals(currentState.toMap())) {
+            return false;
+        }
+
+        final boolean replaced = stateManager.replace(stateManager.getState(scope), newValue, scope);

Review Comment:
   [Claude Opus 5.5] You're right. `replaceState` compared against one read of the stored state and then passed a second read to `stateManager.replace`, so an update stored between the two reads would be overwritten. It now reads the stored state once, applies the pending patches to that copy for the comparison, and passes the same copy to `stateManager.replace`. If anything else updates the state in between, the replacement fails. I extended `canSetLocalStateWhenDeclaredStateful` with a state manager that stores an update right after the session reads the state. That check failed before this change and passes now.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-16494 Implemented partial component state updates in framework [nifi]

Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 merged PR #11837:
URL: https://github.com/apache/nifi/pull/11837


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-16494 Implemented partial component state updates in framework [nifi]

Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on code in PR #11837:
URL: https://github.com/apache/nifi/pull/11837#discussion_r4401243866


##########
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java:
##########
@@ -1405,21 +1418,70 @@ public ProvenanceReporter getProvenanceReporter() {
     @Override
     public void setState(final Map<String, String> state, final Scope scope) throws IOException {
         stateManager.setState(state, scope);
+        pendingStatePatches.remove(scope);
     }
 
+    /**
+     * Returns the stored state with any patches that have not yet been committed applied. Each patch gives the returned state a new version,
+     * as the framework does, so that a state obtained before a patch no longer matches in {@link #replaceState(StateMap, Map, Scope)}.
+     */
     @Override
     public StateMap getState(final Scope scope) throws IOException {
-        return stateManager.getState(scope);
+        final StateMap storedState = stateManager.getState(scope);
+        final PendingStatePatches patches = pendingStatePatches.get(scope);
+        if (patches == null) {
+            return storedState;
+        }
+
+        final long storedVersion = storedState.getStateVersion().map(Long::parseLong).orElse(-1L);
+        return new MockStateMap(storedState.toMap(patches.values), storedVersion + patches.count);
     }
 
+    /**
+     * Replaces the state if the given old value matches the session's view of the state. When the session holds patches that have not yet
+     * been committed, the old value must match the stored state with those patches applied, and a successful replacement discards the patches.
+     */
     @Override
     public boolean replaceState(final StateMap oldValue, final Map<String, String> newValue, final Scope scope) throws IOException {
-        return stateManager.replace(oldValue, newValue, scope);
+        if (!pendingStatePatches.containsKey(scope)) {
+            return stateManager.replace(oldValue, newValue, scope);
+        }
+
+        final StateMap currentState = getState(scope);
+        if (oldValue == null || !oldValue.getStateVersion().equals(currentState.getStateVersion()) || !oldValue.toMap().equals(currentState.toMap())) {
+            return false;
+        }
+
+        final boolean replaced = stateManager.replace(stateManager.getState(scope), newValue, scope);

Review Comment:
   This race was already present before the latest commit, but looks cheap to address: a concurrent update between validation and the second state read could be overwritten. Could we capture the stored state once and use that same snapshot for both validation and `replace`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-16494 Implemented partial component state updates in framework [nifi]

Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on code in PR #11837:
URL: https://github.com/apache/nifi/pull/11837#discussion_r4392555602


##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/StateMapSerDe.java:
##########
@@ -28,14 +28,33 @@
 import java.util.Map;
 import java.util.Optional;
 
+/**
+ * Serializes component state for the write-ahead log. Full records, which are used for snapshots, always hold the whole state map.
+ * Starting with version 1, an edit in the journal holds either the whole state map or only the patches. An edit that holds
+ * only the patches is applied on top of the component's previous record when the journal is recovered. Version 0 edits always
+ * hold the whole state map, in the same layout as full records.
+ */
 public class StateMapSerDe implements SerDe<StateMapUpdate> {
     private static final long EMPTY_VERSION = -1;
 
-    private static final int VERSION = 0;
+    private static final int PATCHES_VERSION = 1;

Review Comment:
   Can we add a note to the [Migration Guidance](https://cwiki.apache.org/confluence/spaces/NIFI/pages/57905503/Migration+Guidance) that older NiFi versions cannot read state storage written with serializer version 1?



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java:
##########
@@ -2597,12 +2682,14 @@ public void testRollbackAfterCheckpointStoresState() throws IOException {
         session.checkpoint();
 
         session.setState(Collections.singletonMap("abc", "2"), Scope.LOCAL);
+        session.patchState(Map.of("def", "2"), Scope.LOCAL);

Review Comment:
   Can we extend this test to checkpoint a patch, apply another patch, then roll back and commit, verifying that only the checkpointed patch is stored?



##########
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java:
##########
@@ -1405,21 +1418,64 @@ public ProvenanceReporter getProvenanceReporter() {
     @Override
     public void setState(final Map<String, String> state, final Scope scope) throws IOException {
         stateManager.setState(state, scope);
+        pendingStatePatches.remove(scope);
     }
 
     @Override
     public StateMap getState(final Scope scope) throws IOException {
-        return stateManager.getState(scope);
+        final StateMap storedState = stateManager.getState(scope);
+        final Map<String, String> patches = pendingStatePatches.get(scope);
+        if (patches == null) {
+            return storedState;
+        }
+
+        final long version = storedState.getStateVersion().map(Long::parseLong).orElse(-1L);

Review Comment:
   Should pending patches advance the mock session's state version as they do in `StandardProcessSession`, so a stale state handle after a same-value patch is rejected consistently?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-16494 Implemented partial component state updates in framework [nifi]

Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #11837:
URL: https://github.com/apache/nifi/pull/11837#issuecomment-6473532969

   [Claude Opus 5.5] The `integration-tests / Ubuntu Java 21` check failed in [job 122893261196](https://github.com/apache/nifi/actions/runs/40708043626/job/122893261196) on `ConsumeKinesisIT.testRecordOrientedStrategy`:
   
   ```
   software.amazon.awssdk.services.kinesis.model.ResourceInUseException: Stream record-oriented-test already exists (Service: Kinesis, Status Code: 400, Request ID: 917db076-1c83-408f-b8a3-3bebc428b076) (SDK Attempt Count: 4)
   	at software.amazon.awssdk.services.kinesis.DefaultKinesisClient.createStream(DefaultKinesisClient.java:621)
   	at org.apache.nifi.processors.aws.kinesis.ConsumeKinesisIT.createStream(ConsumeKinesisIT.java:1041)
   	at org.apache.nifi.processors.aws.kinesis.ConsumeKinesisIT.createStream(ConsumeKinesisIT.java:1033)
   	at org.apache.nifi.processors.aws.kinesis.ConsumeKinesisIT.testRecordOrientedStrategy(ConsumeKinesisIT.java:208)
   	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Read timed out
   	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Read timed out
   	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: Read timed out
   ```
   
   This is unrelated to this PR, which does not touch the AWS bundle. The first request to create the stream in the local Kinesis test container timed out but did create the stream, so the SDK's fourth retry was rejected because the stream already existed. I have rerun the failed job.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org