[GitHub] dubee commented on a change in pull request #3704: Invoker graceful shutdown and drain mode

[GitBox <gi...@apache.org>]

dubee commented on a change in pull request #3704: Invoker graceful shutdown and drain mode
URL: https://github.com/apache/incubator-openwhisk/pull/3704#discussion_r194101322
 
 

 ##########
 File path: core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
 ##########
 @@ -284,4 +290,93 @@ class InvokerReactive(
       })
   }
 
+  val healthScheduler = Scheduler.scheduleWaitAtMost(1.seconds)(() => {
+    producer.send("health", PingMessage(instance)).andThen {
+      case Failure(t) => logging.error(this, s"failed to ping the controller(s): $t")
+    }
+  })
+
+  /** Polls the pool's status and returns a future which completes once the pool is idle. */
+  def waitForContainerPoolIdle(pool: ActorRef): Future[Unit] = {
+    implicit val timeout = Timeout(5 seconds)
+    val delay = 1.second
+
+    (pool ? Busy)
+      .mapTo[Boolean]
+      .flatMap {
+        case true =>
+          logging.info(this, "Container pool is not idle.")
+          after(delay, actorSystem.scheduler)(waitForContainerPoolIdle(pool))
+        case false =>
+          Future.successful(())
+      }
+      .recoverWith { case _ => after(delay, actorSystem.scheduler)(waitForContainerPoolIdle(pool)) }
+  }
+
+  /** Polls the feed's status and returns a future which completes once the feed is idle. */
+  def waitForActivationFeedIdle(feed: ActorRef): Future[Unit] = {
+    implicit val timeout = Timeout(5 seconds)
+    val delay = 1.second
+
+    activationFeed ! MessageFeed.GracefulShutdown
+    (feed ? MessageFeed.Busy)
+      .mapTo[Boolean]
+      .flatMap {
+        case true =>
+          logging.info(this, "Activation feed is not idle.")
+          after(delay, actorSystem.scheduler)(waitForActivationFeedIdle(feed))
+        case false =>
+          Future.successful(())
+      }
+      .recoverWith { case _ => after(delay, actorSystem.scheduler)(waitForActivationFeedIdle(feed)) }
+  }
+
+  // Capture SIGTERM signals to gracefully shutdown the invoker. When gracefully shutting down, the health scheduler is
+  // shutdown preventing additional actions from being scheduler to the invoker, then the invoker processes its buffered
+  // messages from the activation feed, and waits for its user containers to finish running before the process exits.
+  Signal.handle(
 
 Review comment:
   From experimenting, order of shutdown hooks is not guaranteed. That means the actor system may shutdown before the graceful shutdown hook is executed. However, the actor system needs to be running in order to perform the graceful shutdown.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services