You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Kevin Doran (JIRA)" <ji...@apache.org> on 2019/02/08 01:20:00 UTC
[jira] [Commented] (NIFI-5950) Error updating flow version if a
port is removed and another port is given the deleted port's name
[ https://issues.apache.org/jira/browse/NIFI-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16763206#comment-16763206 ]
Kevin Doran commented on NIFI-5950:
-----------------------------------
Hey [~joewitt], sure, happy to provide an update.
*_TL;DR: I think it's important to fix this if possible, but it's not quite ready at the moment, another day or two will likely allow this to be closed._*
*Regarding the severity of this bug and urgency to fix:*
It is something that a user could encounter relatively easily in normal use. There are actually a few ways to trigger it – I updated the summary and description accordingly. Basically, the user is able to apply a valid set of changes to ports (of any type, input/output/remote/etc) in one NiFi environment, save the flow to registry, and fail to upgrade to that flow version in another environment. It's confusing when it happens and hard to troubleshoot with the given error message/state.
Mitigating factors for severity:
# There is a known, reliable workaround documented above.
# We've not heard of this impacting many users in practice. That said, impact will likely grow over time as Registry adoption increases, so fixing it sooner rather than later would be good I think.
*Regarding progress and the timeliness of a patch:*
Since that time, I've attempted the original "proposed fix" in the description above. In testing that fix, I realized that while it fixed this case, it broke other cases, ie, other flow diffs that worked previously were broken.
Looking at it closer, with [~markap14], the ordering of this bit of code (the method that applies a flow diff) is pretty important and tricky to change. There is no single order that changes can be applied that works for all flow updates due to circular "change dependencies" in the *general* case. For example, adding ports requires deleting ports (to free the port namespace) but deleting ports requires updating connections, and updating connections requires adding ports.
While there is not a single order of changes that works for all flow updates, for any given flow update there is an order of changes that will work. That is the order that was used in the original NiFi authoring environment, but unfortunately we don't retain that information when saving new flow versions to Registry. We could try to compute that order after the fact by building a dependency graph of change operations and doing a topological sort. Likely simpler, but not without complexity, would be doing something like allow a flow to go into an invalid state (ie, allow duplicate names) _during_ a flow update, so long as it _ends_ in a valid state. The complexity with that approach comes with could we reliably either apply the full update or rollback/revert on failure.
I originally tagged this for inclusion in 1.9.0 when I thought the fix would be simple. There is still likely a viable bug fix approach, but I'd like to take more time on it to make sure we don't introduce a new bug.
> Error updating flow version if a port is removed and another port is given the deleted port's name
> --------------------------------------------------------------------------------------------------
>
> Key: NIFI-5950
> URL: https://issues.apache.org/jira/browse/NIFI-5950
> Project: Apache NiFi
> Issue Type: Bug
> Components: Flow Versioning
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1
> Reporter: Kevin Doran
> Priority: Major
> Fix For: 1.9.0
>
>
> Originally reported on the users@nifi mailing list on this thread:
> [https://lists.apache.org/thread.html/cf1206d65109bd35e07dfa9b7a24684a31e0580ddc2d351d2cb6eeaa@%3Cusers.nifi.apache.org%3E]
> h2. Steps to reproduce (add)
> # Create a PG. We will call this PG1 in these instructions
> # Add a port to PG1 named "my port"
> # Start version control on PG1 to save to registry
> # Create a new PG (PG2) by importing PG1 at version 1 from Registry.
> # In PG1, delete the port named "my port"
> # In PG1, add a port with the same name as the port you just deleted, "my port"
> # Commit a new version of PG1 to registry
> # Change/update PG2 to version 2. This will trigger the failure.
> h2. Steps to reproduce (update)
> # Create a PG. We will call this PG1 in these instructions
> # Add a port to PG1 named "Port 1"
> # Add a port to PG1 named "Port 2"
> # Start version control on PG1 to save to registry
> # Create a new PG (PG2) by importing PG1 at version 1 from Registry.
> # In PG1, delete "Port 1"
> # In PG1, rename "Port 2" to "Port 1"
> # Commit a new version of PG1 to registry
> # Change/update PG2 to version 2. This will trigger the failure.
> h2. Analysis
> During the update, we are attempting to remove an input port and and add an input port with the same name (but different IDs). Here is an example diff that makes this apparent:
> {noformat}
> {
> "bucketId": "898c9714-f3fa-456a-ac81-b63b7b0e7ff9",
> "componentDifferenceGroups": [
> {
> "componentId": "956f4819-77d3-3044-a6df-cf79c9be0646",
> "componentName": "my port",
> "componentType": "Input Port",
> "differences": [
> {
> "changeDescription": "Input Port was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "956f4819-77d3-3044-a6df-cf79c9be0646"
> }
> ],
> "processGroupId": "6cca3284-5caf-3555-9813-48f2dd913782"
> },
> {
> "componentId": "3434b1ce-e5be-348e-897e-cb8f79fdc42a",
> "componentName": "my port",
> "componentType": "Input Port",
> "differences": [
> {
> "changeDescription": "Input Port was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "3434b1ce-e5be-348e-897e-cb8f79fdc42a"
> }
> ],
> "processGroupId": "6cca3284-5caf-3555-9813-48f2dd913782"
> }
> ],
> "flowId": "4f5a691c-428d-4439-9013-9a729b5f5d37",
> "versionA": 1,
> "versionB": 2
> }
> {noformat}
> *It is likely that the update flow logic in [StandardProcessGroup|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java] is attempting to add the new input port before deleting the existing one. As input ports are required to have unique names, it fails with the exception/message we see in the stack trace.*
> h3. Stack trace:
> {noformat}
> 2019-01-10 21:18:38,032 ERROR [Version Control Update Thread-1] org.apache.nifi.web.api.VersionsResource Failed to update flow to new version
> java.lang.IllegalStateException: The input port name or identifier is not available to be added.
> at org.apache.nifi.groups.StandardProcessGroup.addInputPort(StandardProcessGroup.java:512)
> at org.apache.nifi.groups.StandardProcessGroup.addInputPort(StandardProcessGroup.java:4142)
> at org.apache.nifi.groups.StandardProcessGroup.updateProcessGroup(StandardProcessGroup.java:3597)
> at org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3367)
> at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:358)
> at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke(<generated>)
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
> at org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
> at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
> at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
> at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$EnhancerBySpringCGLIB$$9d72ce1d.updateProcessGroupFlow(<generated>)
> at org.apache.nifi.web.StandardNiFiServiceFacade$14.update(StandardNiFiServiceFacade.java:4380)
> at org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:117)
> at org.apache.nifi.web.StandardNiFiServiceFacade.updateProcessGroupContents(StandardNiFiServiceFacade.java:4376)
> at org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke(<generated>)
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
> at org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
> at org.apache.nifi.web.NiFiServiceFacadeLock.updateLock(NiFiServiceFacadeLock.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
> at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
> at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
> at org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$f366f297.updateProcessGroupContents(<generated>)
> at org.apache.nifi.web.api.VersionsResource.updateFlowVersion(VersionsResource.java:1526)
> at org.apache.nifi.web.api.VersionsResource.lambda$null$19(VersionsResource.java:1186)
> at org.apache.nifi.web.api.concurrent.AsyncRequestManager$2.run(AsyncRequestManager.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. Workaround for affected versions of NiFi:
> If you are encountering this issue in your usage of NiFi, it is recommended to rename the port that is causing the issue to give it a name that has not been previously used by a deleted port, save a new version of your flow to Registry, and then do an update/change version that skips ahead to the new version of the flow with the new name. Basically, avoid deleting and adding/updating a port with the same name in a single update of the flow.
> h2. Proposed fix:
> -Change the logic in {{StandardProcessGroup}} so that when flow diffs are being applied, all {{COMPONENT_REMOVED}} changes are applied before {{COMPONENT_ADDED}} changes.-
> Updated in comments below.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)