[jira] [Created] (NIFI-3564) Deadlock on startup

["Brandon DeVries (JIRA)" <ji...@apache.org>]

Brandon DeVries created NIFI-3564:
-------------------------------------

             Summary: Deadlock on startup
                 Key: NIFI-3564
                 URL: https://issues.apache.org/jira/browse/NIFI-3564
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 1.1.1, 0.7.1
            Reporter: Brandon DeVries


We have uncovered an issue in the way that ControllerServices and Processors are started that can result in a deadlock.  Basically, a ControllerService that is reported by the framework as ENABLING might not actually be.  This is because of how they are scheduled to be started in StandardControllerServiceNode.enable()\[1].  This changes the state from DISABLED to ENABLING, and *then* actually schedules the OnEnabled method to be called.  However, it is scheduled with a ScheduledExecutorService that is limited to 8 threads\[2], and is *also used to start Processors*\[3].  

The situation that exposed the bug was a Processor that attempted to wait for a ControllerService to become ENABLED in its customValidate() method.  The ControllerService must be at least in the ENABLING state to pass framework validation, and since the ControllerService was neccessary to do the custom validation, waiting for it to become ENABLED seems reasonable.  However, there were several (more than 8) instances of this custom Processor on the graph, and the ControllerService being waited on was one of dozens.  This led to the situation where all 8 of the executor threads were held by our Processor's customValidate() method waiting for a service that will never transition from ENABLING to ENABLED because to do so it needs one of those same 8 threads.  This deadlocks the instance, preventing startup.

My first thought as to a fix was to not set the ENABLING state until the OnEnabled method was actually being called (as opposed to scheduled to be called).  However, this could result in a Processor attempting to start with a dependent ControllerService in a DISABLED state (even though the ControllerService will eventually be ENABLED), which would cause the processor to not start\[4](as opposed to being retried as is the case when OnScheduled throws an Exception).  My feeling is that ultimately we're going to need to wait for all ControllerServices to be ENABLED before moving on to Processors, possibly using schedule(Callable) instead of execute(Runnable).  


\[1] https://github.com/apache/nifi/blob/rel/nifi-0.7.1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java#L299-L304
\[2] https://github.com/apache/nifi/blob/rel/nifi-0.7.1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/scheduling/StandardProcessScheduler.java#L83
\[3] https://github.com/apache/nifi/blob/rel/nifi-0.7.1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java#L1219-L1228
\[4] https://github.com/apache/nifi/blob/rel/nifi-0.7.1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java#L1221-L1223



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)