You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/09/09 11:37:00 UTC

[jira] [Commented] (NIFI-16319) nifi.security.autoreload.enabled does not reload keystore/truststore when files are Kubernetes Secret mounts (..data atomic symlink swap)

    [ https://issues.apache.org/jira/browse/NIFI-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113289#comment-18113289 ] 

ASF subversion and git services commented on NIFI-16319:
--------------------------------------------------------

Commit b6a04999e8aea33a44dde95299cc04456567b66c in nifi's branch refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b6a04999e8a ]

NIFI-16319 Fixed reloading of symlinked Security Stores (#11653)

- Replaced WatchService approach with file digest comparison

> nifi.security.autoreload.enabled does not reload keystore/truststore when files are Kubernetes Secret mounts (..data atomic symlink swap)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-16319
>                 URL: https://issues.apache.org/jira/browse/NIFI-16319
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.9.0, 2.10.0, 2.11.0
>            Reporter: Bryan Bende
>            Assignee: David Handermann
>            Priority: Major
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> With {{{}nifi.security.autoreload.enabled=true{}}}, NiFi is supposed to pick up keystore and truststore changes without a restart. That fails when the stores are Kubernetes Secret or projected-volume mounts, which is the normal way to inject certs on Kubernetes.
> Kubelet does not overwrite {{keystore.jks}} / {{truststore.jks}} in place. It writes a new timestamped directory, atomically retargets the {{..data}} symlink, and deletes the old directory:
> {{/nifi/certs/truststore.jks -> ..data/truststore.jks -> ..2026_07_21_12_00_00.123/truststore.jks}}
> after Secret update:
> {{/nifi/certs/truststore.jks -> ..data/truststore.jks -> ..2026_08_30_12_00_00.456/truststore.jks}}
> The stable names never change. Only the directory behind {{..data}} changes.
> h3. What 2.x does
> NIFI-12125 replaced the Jetty {{StoreScanner}} with {{java.nio.file.WatchService}} in {{{}SslContextConfiguration{}}}. On startup it:
>  * registers {{ENTRY_CREATE}} / {{ENTRY_MODIFY}} on the store parent directory _and_ on {{storePath.toRealPath()}} (the timestamped kubelet dir)
>  * reloads only when an event filename is {{keystore.jks}} or {{truststore.jks}} ({{{}SecurityStoreChangedPathListener{}}})
>  * polls one {{WatchKey}} per {{nifi.security.autoreload.interval}} ({{{}WatchServiceMonitorCommand{}}})
> The timestamped directory is never modified; it is replaced. The watch key on that directory becomes invalid ({{{}Watch Key reset failed: Watch Service no longer valid{}}}). Events on the parent are for {{..data}} or {{{}..<timestamp>{}}}, which do not match the filename filter. After the first Secret update, autoreload never fires again for the life of the JVM.
> The wrapping {{StandardX509ExtendedKeyManager}} / {{StandardX509ExtendedTrustManager}} from NIFI-12125 are fine: new sockets (including cluster-protocol heartbeats) would use reloaded material _if_ a reload happened. The bug is that the watcher never re-reads the files.
> h3. Regression
> This is NIFI-10425 again. That ticket described the same kubelet layout. The 1.x fix was Jetty {{reportRealPaths=false}} so the scanner tracked the {_}symlink{_}, not the real file. NIFI-12125 dropped that scanner. {{toRealPath()}} in {{SslContextConfiguration.addStorePath}} / {{addStoreFileName}} is the opposite of that fix.
> h3. Expected
> Updating the Secret that backs {{nifi.security.keystore}} / {{nifi.security.truststore}} causes NiFi, within about one autoreload interval, to log that the Key Manager and Trust Manager were reloaded and to use the new material on the next TLS handshake (Jetty and cluster protocol).
> h3. Actual
> The process keeps the Key Manager / Trust Manager loaded at startup. Cluster protocol and HTTPS keep working on the old in-memory trust anchors. A later pod restart loads the on-disk stores and can fail (for example PKIX path validation if the CA rotated and the leaf did not).
> h3. How to reproduce
>  * Run NiFi 2.x in Kubernetes with {{nifi.security.autoreload.enabled=true}} and keystore/truststore coming from a Secret or projected volume (not a copy into a regular directory).
>  * Confirm startup log: {{{}Scheduled Security Store Monitor with Duration [...]{}}}.
>  * Rotate the Secret (new {{truststore.jks}} / {{keystore.jks}} bytes)
>  * Wait several autoreload intervals.
> Expected: {{{}Key Manager and Trust Manager Reloaded from Changed Path{}}}.
>  Actual: no reload; often {{Watch Key reset failed: Watch Service no longer valid}} when the old timestamped dir is removed. {{openssl s_client}} / a new cluster handshake still presents or trusts the original material until the pod restarts.
> A local stand-in without a cluster: point the stores at symlinks whose targets are replaced by swapping a {{{}..data{}}}-style link the way kubelet does, rather than overwriting the file in place.
> h3. Suggested fix
> Do not pin {{WatchService}} to {{toRealPath()}} of a kubelet timestamped directory.
> Reload from the _configured_ paths in {{nifi.properties}} (follow the symlink at read time). The most reliable approach is to poll those paths every {{{}nifi.security.autoreload.interval{}}}: compare digest or {{{}lastModified{}}}, then rebuild with {{FrameworkKeyManagerBuilder}} / {{FrameworkTrustManagerBuilder}} and {{setKeyManager}} / {{setTrustManager}} as today.
> Watching the parent for {{..data}} changes can be an extra trigger, but polling is what actually works with projected Secrets. In-place file writes on a normal filesystem should keep working.
> h3. Related
>  * NIFI-10425 — same kubelet symlink layout; Jetty scanner
>  * NIFI-11536 — autoreload broken after 1.17.0
>  * NIFI-12125 — WatchService + shared SSLContext / cluster protocol; introduced this watcher



--
This message was sent by Atlassian Jira
(v8.20.10#820010)