You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Eric Secules <es...@gmail.com> on 2022/04/05 22:25:50 UTC
Cannot Delete Process Group Because Source of Connection is "Running"
Hello,
I have this program which stops and deletes flows from NiFi, when we're
done with them and once in a long while we fail this operation because of
this:
2022-04-05 13:03:18,569 WARN [NiFi Web Server-281]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is
> running. Returning Conflict response.
> java.lang.IllegalStateException: Destination of Connection
> (ce5a7658-59db-360f-f978-f275e69c072e) is running
> at
> org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke(<generated>)
We run version 1.14.0 right now and I couldn't find any relevant bug
reports in Nifi's Jira solved between this version and the current. The
closest I got was https://issues.apache.org/jira/browse/NIFI-1777
When deactivating and removing flows from the canvas we follow this
procedure:
- Stop the top level process group (which might contain 400-2000
processors), using the equivalent API of right clicking on the process
group and selecting stop
- Wait for stopped state on that process group (we assume this means
that the stopped state is persisted on all contained processors and process
groups)
- Traverse inner process groups, for each:
- deactivate controller services and wait for deactivated state
- delete templates associated with the process group
- delete parameter context
- Delete top level process group <---- error happens here
Upon noticing the error I go into the NiFi canvas and see that all
processors are either stopped or in error.
I can cause the same IllegalStateException through the canvas and get the
same error message, but when I go search for Connection
(ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input ports
are rightfully in the error state (missing parameters, deactivated
controller services) I can work around and delete all the components
individually, and then continue to remove the rest of the flow, it's almost
as if there's some lingering state that the processor is running and that's
what's preventing me from removing a connection that stems from it.
Thanks,
Eric
Re: Cannot Delete Process Group Because Source of Connection is "Running"
Posted by Mark Payne <ma...@hotmail.com>.
Eric,
The endpoint you’d want is /nifi-api/flow/process-groups/{id}/status
So for the root group you’d use /nifi-api/flow/process-groups/root/status
That will give you a big JSON blurb. From there you’d want to look at the element at
$.processGroupStatus.aggregateSnapshot.activeThreadCount
Hope that helps!
-Mark
On Apr 5, 2022, at 7:07 PM, Eric Secules <es...@gmail.com>> wrote:
Hi Mark,
Is there an API for this that can filter by processors within a process group? Would it be the one that provides the data for this portion of the UI?
<image.png>
Thanks,
Eric
On Tue, Apr 5, 2022 at 3:29 PM Mark Payne <ma...@hotmail.com>> wrote:
Eric,
Once a processor state transitions to stopped, it may still have active threads that haven’t completed yet. You’ll need to wait until the processor is stopped and the active threads on the processor reach 0.
Thanks
-Mark
On Apr 5, 2022, at 6:25 PM, Eric Secules <es...@gmail.com>> wrote:
Hello,
I have this program which stops and deletes flows from NiFi, when we're done with them and once in a long while we fail this operation because of this:
2022-04-05 13:03:18,569 WARN [NiFi Web Server-281] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running. Returning Conflict response.
java.lang.IllegalStateException: Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running
at org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
at org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
at org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
at org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
at org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke(<generated>)
We run version 1.14.0 right now and I couldn't find any relevant bug reports in Nifi's Jira solved between this version and the current. The closest I got was https://issues.apache.org/jira/browse/NIFI-1777
When deactivating and removing flows from the canvas we follow this procedure:
* Stop the top level process group (which might contain 400-2000 processors), using the equivalent API of right clicking on the process group and selecting stop
* Wait for stopped state on that process group (we assume this means that the stopped state is persisted on all contained processors and process groups)
* Traverse inner process groups, for each:
* deactivate controller services and wait for deactivated state
* delete templates associated with the process group
* delete parameter context
* Delete top level process group <---- error happens here
Upon noticing the error I go into the NiFi canvas and see that all processors are either stopped or in error.
I can cause the same IllegalStateException through the canvas and get the same error message, but when I go search for Connection (ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input ports are rightfully in the error state (missing parameters, deactivated controller services) I can work around and delete all the components individually, and then continue to remove the rest of the flow, it's almost as if there's some lingering state that the processor is running and that's what's preventing me from removing a connection that stems from it.
Thanks,
Eric
Re: Cannot Delete Process Group Because Source of Connection is "Running"
Posted by Mark Payne <ma...@hotmail.com>.
Eric,
I hear you, this can be a bit confusing.
The idea is that “stopped” is the “scheduled state”. I.e., it’s scheduled to be stopped. It may still have active threads, though.
It’s one of those things where it made a lot of sense when it was done, but maybe wasn’t the best decision in the long run :)
But we try to remain pretty strict about breaking backward compatibility in our API’s, both REST API and the extension API.
So we’re a bit stuck with that oddity. At least until a 2.0 comes out.
Thanks
-Mark
On Apr 5, 2022, at 7:16 PM, Eric Secules <es...@gmail.com>> wrote:
Side note, that's the behavior I'd expect from a "stopping" processor but not one that's in the "stopped" state.
On Tue., Apr. 5, 2022, 4:07 p.m. Eric Secules, <es...@gmail.com>> wrote:
Hi Mark,
Is there an API for this that can filter by processors within a process group? Would it be the one that provides the data for this portion of the UI?
<image.png>
Thanks,
Eric
On Tue, Apr 5, 2022 at 3:29 PM Mark Payne <ma...@hotmail.com>> wrote:
Eric,
Once a processor state transitions to stopped, it may still have active threads that haven’t completed yet. You’ll need to wait until the processor is stopped and the active threads on the processor reach 0.
Thanks
-Mark
On Apr 5, 2022, at 6:25 PM, Eric Secules <es...@gmail.com>> wrote:
Hello,
I have this program which stops and deletes flows from NiFi, when we're done with them and once in a long while we fail this operation because of this:
2022-04-05 13:03:18,569 WARN [NiFi Web Server-281] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running. Returning Conflict response.
java.lang.IllegalStateException: Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running
at org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
at org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
at org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
at org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
at org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke(<generated>)
We run version 1.14.0 right now and I couldn't find any relevant bug reports in Nifi's Jira solved between this version and the current. The closest I got was https://issues.apache.org/jira/browse/NIFI-1777
When deactivating and removing flows from the canvas we follow this procedure:
* Stop the top level process group (which might contain 400-2000 processors), using the equivalent API of right clicking on the process group and selecting stop
* Wait for stopped state on that process group (we assume this means that the stopped state is persisted on all contained processors and process groups)
* Traverse inner process groups, for each:
* deactivate controller services and wait for deactivated state
* delete templates associated with the process group
* delete parameter context
* Delete top level process group <---- error happens here
Upon noticing the error I go into the NiFi canvas and see that all processors are either stopped or in error.
I can cause the same IllegalStateException through the canvas and get the same error message, but when I go search for Connection (ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input ports are rightfully in the error state (missing parameters, deactivated controller services) I can work around and delete all the components individually, and then continue to remove the rest of the flow, it's almost as if there's some lingering state that the processor is running and that's what's preventing me from removing a connection that stems from it.
Thanks,
Eric
Re: Cannot Delete Process Group Because Source of Connection is "Running"
Posted by Eric Secules <es...@gmail.com>.
Side note, that's the behavior I'd expect from a "stopping" processor but
not one that's in the "stopped" state.
On Tue., Apr. 5, 2022, 4:07 p.m. Eric Secules, <es...@gmail.com> wrote:
> Hi Mark,
>
> Is there an API for this that can filter by processors within a process
> group? Would it be the one that provides the data for this portion of the
> UI?
> [image: image.png]
> Thanks,
> Eric
>
> On Tue, Apr 5, 2022 at 3:29 PM Mark Payne <ma...@hotmail.com> wrote:
>
>> Eric,
>>
>> Once a processor state transitions to stopped, it may still have active
>> threads that haven’t completed yet. You’ll need to wait until the processor
>> is stopped and the active threads on the processor reach 0.
>>
>> Thanks
>> -Mark
>>
>> On Apr 5, 2022, at 6:25 PM, Eric Secules <es...@gmail.com> wrote:
>>
>> Hello,
>>
>> I have this program which stops and deletes flows from NiFi, when we're
>> done with them and once in a long while we fail this operation because of
>> this:
>>
>> 2022-04-05 13:03:18,569 WARN [NiFi Web Server-281]
>>> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
>>> Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is
>>> running. Returning Conflict response.
>>> java.lang.IllegalStateException: Destination of Connection
>>> (ce5a7658-59db-360f-f978-f275e69c072e) is running
>>> at
>>> org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
>>> at
>>> org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
>>> at
>>> org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
>>> at
>>> org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
>>> at
>>> org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
>>> at
>>> org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
>>> at
>>> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
>>> at
>>> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke(<generated>)
>>
>>
>> We run version 1.14.0 right now and I couldn't find any relevant bug
>> reports in Nifi's Jira solved between this version and the current. The
>> closest I got was https://issues.apache.org/jira/browse/NIFI-1777
>>
>> When deactivating and removing flows from the canvas we follow this
>> procedure:
>>
>>
>> - Stop the top level process group (which might contain 400-2000
>> processors), using the equivalent API of right clicking on the process
>> group and selecting stop
>> - Wait for stopped state on that process group (we assume this means
>> that the stopped state is persisted on all contained processors and process
>> groups)
>> - Traverse inner process groups, for each:
>> - deactivate controller services and wait for deactivated state
>> - delete templates associated with the process group
>> - delete parameter context
>> - Delete top level process group <---- error happens here
>>
>> Upon noticing the error I go into the NiFi canvas and see that all
>> processors are either stopped or in error.
>>
>> I can cause the same IllegalStateException through the canvas and get the
>> same error message, but when I go search for Connection
>> (ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input
>> ports are rightfully in the error state (missing parameters, deactivated
>> controller services) I can work around and delete all the components
>> individually, and then continue to remove the rest of the flow, it's almost
>> as if there's some lingering state that the processor is running and that's
>> what's preventing me from removing a connection that stems from it.
>>
>>
>> Thanks,
>> Eric
>>
>>
>>
Re: Cannot Delete Process Group Because Source of Connection is "Running"
Posted by Eric Secules <es...@gmail.com>.
Hi Mark,
Is there an API for this that can filter by processors within a process
group? Would it be the one that provides the data for this portion of the
UI?
[image: image.png]
Thanks,
Eric
On Tue, Apr 5, 2022 at 3:29 PM Mark Payne <ma...@hotmail.com> wrote:
> Eric,
>
> Once a processor state transitions to stopped, it may still have active
> threads that haven’t completed yet. You’ll need to wait until the processor
> is stopped and the active threads on the processor reach 0.
>
> Thanks
> -Mark
>
> On Apr 5, 2022, at 6:25 PM, Eric Secules <es...@gmail.com> wrote:
>
> Hello,
>
> I have this program which stops and deletes flows from NiFi, when we're
> done with them and once in a long while we fail this operation because of
> this:
>
> 2022-04-05 13:03:18,569 WARN [NiFi Web Server-281]
>> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
>> Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is
>> running. Returning Conflict response.
>> java.lang.IllegalStateException: Destination of Connection
>> (ce5a7658-59db-360f-f978-f275e69c072e) is running
>> at
>> org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
>> at
>> org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
>> at
>> org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
>> at
>> org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
>> at
>> org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
>> at
>> org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
>> at
>> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
>> at
>> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke(<generated>)
>
>
> We run version 1.14.0 right now and I couldn't find any relevant bug
> reports in Nifi's Jira solved between this version and the current. The
> closest I got was https://issues.apache.org/jira/browse/NIFI-1777
>
> When deactivating and removing flows from the canvas we follow this
> procedure:
>
>
> - Stop the top level process group (which might contain 400-2000
> processors), using the equivalent API of right clicking on the process
> group and selecting stop
> - Wait for stopped state on that process group (we assume this means
> that the stopped state is persisted on all contained processors and process
> groups)
> - Traverse inner process groups, for each:
> - deactivate controller services and wait for deactivated state
> - delete templates associated with the process group
> - delete parameter context
> - Delete top level process group <---- error happens here
>
> Upon noticing the error I go into the NiFi canvas and see that all
> processors are either stopped or in error.
>
> I can cause the same IllegalStateException through the canvas and get the
> same error message, but when I go search for Connection
> (ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input
> ports are rightfully in the error state (missing parameters, deactivated
> controller services) I can work around and delete all the components
> individually, and then continue to remove the rest of the flow, it's almost
> as if there's some lingering state that the processor is running and that's
> what's preventing me from removing a connection that stems from it.
>
>
> Thanks,
> Eric
>
>
>
Re: Cannot Delete Process Group Because Source of Connection is "Running"
Posted by Mark Payne <ma...@hotmail.com>.
Eric,
Once a processor state transitions to stopped, it may still have active threads that haven’t completed yet. You’ll need to wait until the processor is stopped and the active threads on the processor reach 0.
Thanks
-Mark
On Apr 5, 2022, at 6:25 PM, Eric Secules <es...@gmail.com>> wrote:
Hello,
I have this program which stops and deletes flows from NiFi, when we're done with them and once in a long while we fail this operation because of this:
2022-04-05 13:03:18,569 WARN [NiFi Web Server-281] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running. Returning Conflict response.
java.lang.IllegalStateException: Destination of Connection (ce5a7658-59db-360f-f978-f275e69c072e) is running
at org.apache.nifi.connectable.StandardConnection.verifyCanDelete(StandardConnection.java:508)
at org.apache.nifi.groups.StandardProcessGroup.removeConnection(StandardProcessGroup.java:1270)
at org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:864)
at org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at org.apache.nifi.groups.StandardProcessGroup.removeComponents(StandardProcessGroup.java:897)
at org.apache.nifi.groups.StandardProcessGroup.removeProcessGroup(StandardProcessGroup.java:851)
at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.deleteProcessGroup(StandardProcessGroupDAO.java:528)
at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke(<generated>)
We run version 1.14.0 right now and I couldn't find any relevant bug reports in Nifi's Jira solved between this version and the current. The closest I got was https://issues.apache.org/jira/browse/NIFI-1777
When deactivating and removing flows from the canvas we follow this procedure:
* Stop the top level process group (which might contain 400-2000 processors), using the equivalent API of right clicking on the process group and selecting stop
* Wait for stopped state on that process group (we assume this means that the stopped state is persisted on all contained processors and process groups)
* Traverse inner process groups, for each:
* deactivate controller services and wait for deactivated state
* delete templates associated with the process group
* delete parameter context
* Delete top level process group <---- error happens here
Upon noticing the error I go into the NiFi canvas and see that all processors are either stopped or in error.
I can cause the same IllegalStateException through the canvas and get the same error message, but when I go search for Connection (ce5a7658-59db-360f-f978-f275e69c072e), all the processors and input ports are rightfully in the error state (missing parameters, deactivated controller services) I can work around and delete all the components individually, and then continue to remove the rest of the flow, it's almost as if there's some lingering state that the processor is running and that's what's preventing me from removing a connection that stems from it.
Thanks,
Eric