You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/07/28 08:23:00 UTC
[jira] [Commented] (NIFI-7679) Flow fingerprinting became extremely
slow
[ https://issues.apache.org/jira/browse/NIFI-7679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166251#comment-17166251 ]
ASF subversion and git services commented on NIFI-7679:
-------------------------------------------------------
Commit 90c9db8208d1713b05526d9e3f91f9a484382263 in nifi's branch refs/heads/main from Andy LoPresto
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=90c9db8 ]
NIFI-7679 Changed flow fingerprint masking logic to derive a static key once (slowly) from nifi.sensitive.props.key and use a (fast) HMAC/SHA-256 operation during fingerprinting to resolve unacceptable delays.
Added unit tests.
Signed-off-by: Pierre Villard <pi...@gmail.com>
This closes #4434.
> Flow fingerprinting became extremely slow
> -----------------------------------------
>
> Key: NIFI-7679
> URL: https://issues.apache.org/jira/browse/NIFI-7679
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Pierre Villard
> Assignee: Andy LoPresto
> Priority: Blocker
> Fix For: 1.12.0
>
>
> With recent commits (likely related to NIFI-7122 or NIFI-7638), NiFi will be extremely slow to start with relatively large flows definitions and will be extremely CPU intensive. According to a thread dump, it seems to be related to the fingerprinting logic:
> {code:java}
> "main" Id=1 RUNNABLE
> at org.bouncycastle.crypto.generators.Argon2BytesGenerator.fillBlock(Unknown Source)
> at org.bouncycastle.crypto.generators.Argon2BytesGenerator.fillSegment(Unknown Source)
> at org.bouncycastle.crypto.generators.Argon2BytesGenerator.fillMemoryBlocks(Unknown Source)
> at org.bouncycastle.crypto.generators.Argon2BytesGenerator.generateBytes(Unknown Source)
> at org.bouncycastle.crypto.generators.Argon2BytesGenerator.generateBytes(Unknown Source)
> at org.apache.nifi.security.util.crypto.Argon2SecureHasher.hash(Argon2SecureHasher.java:290)
> at org.apache.nifi.security.util.crypto.Argon2SecureHasher.hash(Argon2SecureHasher.java:258)
> at org.apache.nifi.security.util.crypto.AbstractSecureHasher.hashBase64(AbstractSecureHasher.java:188)
> at org.apache.nifi.security.util.crypto.CipherUtility.getLoggableRepresentationOfSensitiveValue(CipherUtility.java:450)
> at org.apache.nifi.security.util.crypto.CipherUtility.getLoggableRepresentationOfSensitiveValue(CipherUtility.java:435)
> at org.apache.nifi.fingerprint.FingerprintFactory.getLoggableRepresentationOfSensitiveValue(FingerprintFactory.java:552)
> at org.apache.nifi.fingerprint.FingerprintFactory.access$200(FingerprintFactory.java:70)
> at org.apache.nifi.fingerprint.FingerprintFactory$6.compare(FingerprintFactory.java:846)
> at org.apache.nifi.fingerprint.FingerprintFactory$6.compare(FingerprintFactory.java:831)
> at java.util.TimSort.binarySort(TimSort.java:296)
> at java.util.TimSort.sort(TimSort.java:221)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1464)
> at java.util.Collections.sort(Collections.java:177)
> at org.apache.nifi.fingerprint.FingerprintFactory.sortElements(FingerprintFactory.java:880)
> at org.apache.nifi.fingerprint.FingerprintFactory.addFlowFileProcessorFingerprint(FingerprintFactory.java:488)
> at org.apache.nifi.fingerprint.FingerprintFactory.addProcessGroupFingerprint(FingerprintFactory.java:370)
> at org.apache.nifi.fingerprint.FingerprintFactory.addProcessGroupFingerprint(FingerprintFactory.java:398)
> at org.apache.nifi.fingerprint.FingerprintFactory.addProcessGroupFingerprint(FingerprintFactory.java:398)
> at org.apache.nifi.fingerprint.FingerprintFactory.addProcessGroupFingerprint(FingerprintFactory.java:398)
> at org.apache.nifi.fingerprint.FingerprintFactory.addFlowControllerFingerprint(FingerprintFactory.java:232)
> at org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:172)
> - waiting on org.apache.nifi.fingerprint.FingerprintFactory@6ae4b0f3
> at org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:145)
> - waiting on org.apache.nifi.fingerprint.FingerprintFactory@6ae4b0f3
> at org.apache.nifi.controller.inheritance.FlowFingerprintCheck.checkInheritability(FlowFingerprintCheck.java:43)
> at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:206)
> at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1425)
> at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
> - waiting on org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO@48106119
> at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:792)
> at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:537)
> at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:930)
> at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
> at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:889)
> at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:356)
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
> at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> - waiting on java.lang.Object@1c1d9b1e
> - waiting on java.lang.Object@6c3bb93d
> - waiting on java.lang.Object@2317ff61
> - waiting on java.lang.Object@68444ab3
> - waiting on java.lang.Object@18f1e9ce
> - waiting on java.lang.Object@29b249bc
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> - waiting on java.lang.Object@1c1d9b1e
> - waiting on java.lang.Object@6c3bb93d
> - waiting on java.lang.Object@2317ff61
> - waiting on java.lang.Object@68444ab3
> - waiting on java.lang.Object@18f1e9ce
> - waiting on java.lang.Object@29b249bc
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:425)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> - waiting on java.lang.Object@1c1d9b1e
> - waiting on java.lang.Object@6c3bb93d
> - waiting on java.lang.Object@2317ff61
> - waiting on java.lang.Object@68444ab3
> - waiting on java.lang.Object@18f1e9ce
> - waiting on java.lang.Object@29b249bc
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> - waiting on java.lang.Object@1c1d9b1e
> - waiting on java.lang.Object@6c3bb93d
> - waiting on java.lang.Object@2317ff61
> - waiting on java.lang.Object@68444ab3
> - waiting on java.lang.Object@18f1e9ce
> - waiting on java.lang.Object@29b249bc
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> - waiting on java.lang.Object@1c1d9b1e
> - waiting on java.lang.Object@6c3bb93d
> - waiting on java.lang.Object@2317ff61
> - waiting on java.lang.Object@68444ab3
> - waiting on java.lang.Object@18f1e9ce
> - waiting on java.lang.Object@29b249bc
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.server.Server.start(Server.java:407)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.server.Server.doStart(Server.java:371)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> - waiting on java.lang.Object@1c1d9b1e
> - waiting on java.lang.Object@6c3bb93d
> - waiting on java.lang.Object@2317ff61
> - waiting on java.lang.Object@68444ab3
> - waiting on java.lang.Object@18f1e9ce
> - waiting on java.lang.Object@29b249bc
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1058)
> at org.apache.nifi.NiFi.<init>(NiFi.java:158)
> at org.apache.nifi.NiFi.<init>(NiFi.java:72)
> at org.apache.nifi.NiFi.main(NiFi.java:301)
> Number of Locked Synchronizers: 2
> - java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@35c2471c
> - java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@7f9982c5
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)