[GitHub] [openwhisk] rabbah commented on a change in pull request #4921: Add a proposal for a new scheduler

[GitBox <gi...@apache.org>]

rabbah commented on a change in pull request #4921:
URL: https://github.com/apache/openwhisk/pull/4921#discussion_r447714805



##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)

Review comment:
       ```suggestion
   Function Pulling Container Scheduler (FPCScheduler)
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)

Review comment:
       Small note: I'm not sure about this name - the existing scheduler is also a pull scheduler in that each invoker pulls from the kafka queue and schedules containers.

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.

Review comment:
       I don't understand this point. When/How is the decision to create more containers done?

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).

Review comment:
       ```suggestion
   2. Current scheduler does not consider the time taken for container operations (create/delete, pause/unpause).
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.

Review comment:
       ```suggestion
   - An added benefit is that schedulers don't need to track or consider the location of existing containers. Schedulers decide where and when to create more containers.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.

Review comment:
       ```suggestion
   As a result, controllers may throttle activations even if cluster-wide invocations are below limits (and when there are available resources and system capacity).
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.

Review comment:
       ```suggestion
   For the record, we (Naver) are already operating OpenWhisk with this scheduler in a production environment.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.
+We want to contribute the scheduler and hope it evolves with the community.
+
+There are many [new components](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design).
+
+#### Common components
+We store data in ETCD, there are many relevant components such as `EtcdClient`, `LeaseKeepAliveService`, `WatcherService`, `DataManagementService`, etc.
+
+* `EtcdClient`: An ETCD client offering basic CRUD operations.
+* `DataManagementService`: In charge of storing/recovering data in ETCD. It internally utilizes `LeaseKeepAliveService`, and `WatcherService`.
+* `LeaseKeepAliveService`: In charge of keeping the given lease alive.
+* `WatcherService`: Watches the given keys and receives events for the keys.
+
+
+#### Scheduler components
+
+In an abstract view, schedulers provide queueing, container scheduling, and activation routing.
+
+* `QueueManager`: The main entry point for queue creation request. It has references to all queues.
+* `MemoryQueue`: Dynamically created/deleted for each action. It watches the incoming/outgoing requests and triggers container creation.
+* `ContainerManager`: Schedule container creation requests to appropriate invokers.
+* `ActivationServiceImpl`: Provide API for containers to fetch activations via Akka-grpc. It works in a long-poll way to avoid busy-waiting.
+
+#### Controller components
+* `FPCPoolBalancer`: Create queues if not exist, and forward messages to them.
+* `FPCEntitlementProvider`: Throttle activations based on throttling information in ETCD.
+
+#### Invoker components.

Review comment:
       ```suggestion
   #### Invoker components
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),

Review comment:
       ```suggestion
   If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time (most of the serverless cases),
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.

Review comment:
       ```suggestion
   This is generally a good idea to increase the probability of container reuse, but it leads to performance degradation because of slow container operation.
   ```
   
   Note the reason for the home invoker is to limit the impact of any one user for a fixed capacity system. That is, it reduces the number of invokers that any one user can monopolize. You are right however that there is a pathology where a queue of actions on the home invoker cannot be reallocated (no work stealing) when other invokers that are accessible to the same user in general become idle.

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.

Review comment:
       Provide a link to etcd?

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.

Review comment:
       ```suggestion
   Second, since controllers schedule activations to invokers in a distributed environment, the invokers in the system are partitioned and assigned to specific controllers. This means that any one controller is assigned a fraction of the available resources in the system.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.

Review comment:
       I'm not sure if I understood the point you're trying to make here. My interpretation is this:
   
   ```suggestion
   It's not feasible to collect the status of all invokers so that scheduling decisions have a global view.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),

Review comment:
       it might be useful to provide a couple of references to back up the short duration - there are some studies from datadog and new relic that support the statement as examples.

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.

Review comment:
       ```suggestion
   When all resources in an invoker are taken by one action (busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.

Review comment:
       ```suggestion
   - Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.

Review comment:
       ```suggestion
   the heuristic severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and this results in poor performance.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.

Review comment:
       ```suggestion
   The scheduler differs from the existing scheduler (`ShardingPoolBalancer`) in the following ways:
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.

Review comment:
       Is the container specific to a function+user?

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.

Review comment:
       ```suggestion
   - It schedules containers rather than function requests. Each container will pull activations requests from the given action queue continuously.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of

Review comment:
       Is the queue specific to a function name or name + namespace?

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.

Review comment:
       How do you secure the queue interfaces from inside the container?

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of

Review comment:
       ```suggestion
   Each action queue is a dedicated queue for the given action and dynamically created/deleted by the FPCSchedulers. Each action has its own queue, so there is no interference among actions because of a shared queue.
   ```
   
   

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design

Review comment:
       ```suggestion
   ## Proposed changes: Architecture Diagram (optional) and Design
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.

Review comment:
       ```suggestion
   Controllers no longer schedule activation requests. Instead, controllers create action queues and forward activation requests to these queues.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.

Review comment:
       ```suggestion
   Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed via a lease timing out.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+

Review comment:
       ```suggestion
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.

Review comment:
       Curious why etcd not redis?

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.

Review comment:
       ```suggestion
   Each scheduler performs a transaction via ETCD when scheduling. The health status of each component (scheduler, invoker) is managed by a [lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.
+We want to contribute the scheduler and hope it evolves with the community.
+
+There are many [new components](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design).
+
+#### Common components
+We store data in ETCD, there are many relevant components such as `EtcdClient`, `LeaseKeepAliveService`, `WatcherService`, `DataManagementService`, etc.
+
+* `EtcdClient`: An ETCD client offering basic CRUD operations.
+* `DataManagementService`: In charge of storing/recovering data in ETCD. It internally utilizes `LeaseKeepAliveService`, and `WatcherService`.
+* `LeaseKeepAliveService`: In charge of keeping the given lease alive.
+* `WatcherService`: Watches the given keys and receives events for the keys.
+
+
+#### Scheduler components
+
+In an abstract view, schedulers provide queueing, container scheduling, and activation routing.
+
+* `QueueManager`: The main entry point for queue creation request. It has references to all queues.
+* `MemoryQueue`: Dynamically created/deleted for each action. It watches the incoming/outgoing requests and triggers container creation.
+* `ContainerManager`: Schedule container creation requests to appropriate invokers.
+* `ActivationServiceImpl`: Provide API for containers to fetch activations via Akka-grpc. It works in a long-poll way to avoid busy-waiting.
+
+#### Controller components
+* `FPCPoolBalancer`: Create queues if not exist, and forward messages to them.
+* `FPCEntitlementProvider`: Throttle activations based on throttling information in ETCD.
+
+#### Invoker components.
+
+* `FunctionPullingContainerPool`: A container pool for function pulling container. It handles the container creation requests.
+* `FunctionPullingContainerProxy`: A proxy for a container. It repeatedly fetches activations and invokes them.
+* `ActivationClientProxy`: The Akka-grpc client. It communicates with `ActivationServiceImpl` in schedulers.
+* `InvokerHealthManager`: Manages the health and resources information of invokers. The data is stored in ETCD. If an invoker becomes unhealthy, it invokes health activations.
+
+## Future work.
+
+#### Persistence
+
+We reached a conclusion that the persistence of activation requests is not that mandatory requirement along with the at-most-once nature and circuit-breaking of OpenWhisk.
+But if it is desired, we can implement a persistent queue rather than an in-memory queue.

Review comment:
       I think the at most once policy was a mistake. I am of the opinion that at least once is superior in practice.

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.

Review comment:
       ```suggestion
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)

Review comment:
       ```suggestion
   The design document along architecture diagram is already shared on the [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.

Review comment:
       ```suggestion
   - The scheduler can create more containers on invokers with enough resources. This enables distributed scheduling with all invoker resources among multiple schedulers.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.
+We want to contribute the scheduler and hope it evolves with the community.
+
+There are many [new components](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design).
+
+#### Common components
+We store data in ETCD, there are many relevant components such as `EtcdClient`, `LeaseKeepAliveService`, `WatcherService`, `DataManagementService`, etc.
+
+* `EtcdClient`: An ETCD client offering basic CRUD operations.
+* `DataManagementService`: In charge of storing/recovering data in ETCD. It internally utilizes `LeaseKeepAliveService`, and `WatcherService`.
+* `LeaseKeepAliveService`: In charge of keeping the given lease alive.
+* `WatcherService`: Watches the given keys and receives events for the keys.
+
+
+#### Scheduler components
+
+In an abstract view, schedulers provide queueing, container scheduling, and activation routing.
+
+* `QueueManager`: The main entry point for queue creation request. It has references to all queues.
+* `MemoryQueue`: Dynamically created/deleted for each action. It watches the incoming/outgoing requests and triggers container creation.
+* `ContainerManager`: Schedule container creation requests to appropriate invokers.
+* `ActivationServiceImpl`: Provide API for containers to fetch activations via Akka-grpc. It works in a long-poll way to avoid busy-waiting.
+
+#### Controller components
+* `FPCPoolBalancer`: Create queues if not exist, and forward messages to them.
+* `FPCEntitlementProvider`: Throttle activations based on throttling information in ETCD.
+
+#### Invoker components.
+
+* `FunctionPullingContainerPool`: A container pool for function pulling container. It handles the container creation requests.
+* `FunctionPullingContainerProxy`: A proxy for a container. It repeatedly fetches activations and invokes them.
+* `ActivationClientProxy`: The Akka-grpc client. It communicates with `ActivationServiceImpl` in schedulers.
+* `InvokerHealthManager`: Manages the health and resources information of invokers. The data is stored in ETCD. If an invoker becomes unhealthy, it invokes health activations.
+
+## Future work.

Review comment:
       ```suggestion
   ## Future work
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.

Review comment:
       ```suggestion
   Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referenced by corresponding components.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.
+We want to contribute the scheduler and hope it evolves with the community.
+
+There are many [new components](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design).
+
+#### Common components
+We store data in ETCD, there are many relevant components such as `EtcdClient`, `LeaseKeepAliveService`, `WatcherService`, `DataManagementService`, etc.
+
+* `EtcdClient`: An ETCD client offering basic CRUD operations.
+* `DataManagementService`: In charge of storing/recovering data in ETCD. It internally utilizes `LeaseKeepAliveService`, and `WatcherService`.
+* `LeaseKeepAliveService`: In charge of keeping the given lease alive.
+* `WatcherService`: Watches the given keys and receives events for the keys.
+
+
+#### Scheduler components
+
+In an abstract view, schedulers provide queueing, container scheduling, and activation routing.
+
+* `QueueManager`: The main entry point for queue creation request. It has references to all queues.
+* `MemoryQueue`: Dynamically created/deleted for each action. It watches the incoming/outgoing requests and triggers container creation.
+* `ContainerManager`: Schedule container creation requests to appropriate invokers.
+* `ActivationServiceImpl`: Provide API for containers to fetch activations via Akka-grpc. It works in a long-poll way to avoid busy-waiting.
+
+#### Controller components
+* `FPCPoolBalancer`: Create queues if not exist, and forward messages to them.
+* `FPCEntitlementProvider`: Throttle activations based on throttling information in ETCD.
+
+#### Invoker components.
+
+* `FunctionPullingContainerPool`: A container pool for function pulling container. It handles the container creation requests.
+* `FunctionPullingContainerProxy`: A proxy for a container. It repeatedly fetches activations and invokes them.
+* `ActivationClientProxy`: The Akka-grpc client. It communicates with `ActivationServiceImpl` in schedulers.
+* `InvokerHealthManager`: Manages the health and resources information of invokers. The data is stored in ETCD. If an invoker becomes unhealthy, it invokes health activations.
+
+## Future work.
+
+#### Persistence
+
+We reached a conclusion that the persistence of activation requests is not that mandatory requirement along with the at-most-once nature and circuit-breaking of OpenWhisk.
+But if it is desired, we can implement a persistent queue rather than an in-memory queue.
+
+#### Multiple partitions
+
+Currently, a queue only has one partition. Since there can be multiple queues for each action, cluster-wide RPS(request per second) linearly increases.

Review comment:
       ```suggestion
   Currently, a queue only has one partition. Since there can be multiple queues for each action, cluster-wide RPS (request per second) increases linearly.
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.
+We want to contribute the scheduler and hope it evolves with the community.
+
+There are many [new components](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design).
+
+#### Common components
+We store data in ETCD, there are many relevant components such as `EtcdClient`, `LeaseKeepAliveService`, `WatcherService`, `DataManagementService`, etc.
+
+* `EtcdClient`: An ETCD client offering basic CRUD operations.
+* `DataManagementService`: In charge of storing/recovering data in ETCD. It internally utilizes `LeaseKeepAliveService`, and `WatcherService`.
+* `LeaseKeepAliveService`: In charge of keeping the given lease alive.
+* `WatcherService`: Watches the given keys and receives events for the keys.
+
+
+#### Scheduler components
+
+In an abstract view, schedulers provide queueing, container scheduling, and activation routing.
+
+* `QueueManager`: The main entry point for queue creation request. It has references to all queues.
+* `MemoryQueue`: Dynamically created/deleted for each action. It watches the incoming/outgoing requests and triggers container creation.
+* `ContainerManager`: Schedule container creation requests to appropriate invokers.
+* `ActivationServiceImpl`: Provide API for containers to fetch activations via Akka-grpc. It works in a long-poll way to avoid busy-waiting.
+
+#### Controller components
+* `FPCPoolBalancer`: Create queues if not exist, and forward messages to them.
+* `FPCEntitlementProvider`: Throttle activations based on throttling information in ETCD.
+
+#### Invoker components.
+
+* `FunctionPullingContainerPool`: A container pool for function pulling container. It handles the container creation requests.
+* `FunctionPullingContainerProxy`: A proxy for a container. It repeatedly fetches activations and invokes them.
+* `ActivationClientProxy`: The Akka-grpc client. It communicates with `ActivationServiceImpl` in schedulers.
+* `InvokerHealthManager`: Manages the health and resources information of invokers. The data is stored in ETCD. If an invoker becomes unhealthy, it invokes health activations.
+
+## Future work.
+
+#### Persistence
+
+We reached a conclusion that the persistence of activation requests is not that mandatory requirement along with the at-most-once nature and circuit-breaking of OpenWhisk.
+But if it is desired, we can implement a persistent queue rather than an in-memory queue.
+
+#### Multiple partitions
+
+Currently, a queue only has one partition. Since there can be multiple queues for each action, cluster-wide RPS(request per second) linearly increases.
+If high RPS for one action is required, we can implement partitioning in queues to introduce parallel processing like what Kafka does.

Review comment:
       I don't understand the reference to Kafka here.

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.
+We want to contribute the scheduler and hope it evolves with the community.
+
+There are many [new components](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design).
+
+#### Common components
+We store data in ETCD, there are many relevant components such as `EtcdClient`, `LeaseKeepAliveService`, `WatcherService`, `DataManagementService`, etc.
+
+* `EtcdClient`: An ETCD client offering basic CRUD operations.
+* `DataManagementService`: In charge of storing/recovering data in ETCD. It internally utilizes `LeaseKeepAliveService`, and `WatcherService`.
+* `LeaseKeepAliveService`: In charge of keeping the given lease alive.
+* `WatcherService`: Watches the given keys and receives events for the keys.
+
+
+#### Scheduler components
+
+In an abstract view, schedulers provide queueing, container scheduling, and activation routing.
+
+* `QueueManager`: The main entry point for queue creation request. It has references to all queues.
+* `MemoryQueue`: Dynamically created/deleted for each action. It watches the incoming/outgoing requests and triggers container creation.
+* `ContainerManager`: Schedule container creation requests to appropriate invokers.
+* `ActivationServiceImpl`: Provide API for containers to fetch activations via Akka-grpc. It works in a long-poll way to avoid busy-waiting.
+
+#### Controller components
+* `FPCPoolBalancer`: Create queues if not exist, and forward messages to them.
+* `FPCEntitlementProvider`: Throttle activations based on throttling information in ETCD.
+
+#### Invoker components.
+
+* `FunctionPullingContainerPool`: A container pool for function pulling container. It handles the container creation requests.
+* `FunctionPullingContainerProxy`: A proxy for a container. It repeatedly fetches activations and invokes them.
+* `ActivationClientProxy`: The Akka-grpc client. It communicates with `ActivationServiceImpl` in schedulers.
+* `InvokerHealthManager`: Manages the health and resources information of invokers. The data is stored in ETCD. If an invoker becomes unhealthy, it invokes health activations.
+
+## Future work.
+
+#### Persistence
+
+We reached a conclusion that the persistence of activation requests is not that mandatory requirement along with the at-most-once nature and circuit-breaking of OpenWhisk.
+But if it is desired, we can implement a persistent queue rather than an in-memory queue.
+
+#### Multiple partitions
+
+Currently, a queue only has one partition. Since there can be multiple queues for each action, cluster-wide RPS(request per second) linearly increases.
+If high RPS for one action is required, we can implement partitioning in queues to introduce parallel processing like what Kafka does.
+But we already confirmed 100K RPS with 10 queues and 10 commodity invokers. And the performance would linearly increase with more invokers and queues.
+

Review comment:
       ```suggestion
   ```

##########
File path: proposals/POEM-2-function-pulling-container-scheduler.md
##########
@@ -0,0 +1,132 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+# Title
+Function Pulling Container Scheduler(FPCScheduler)
+
+## Status
+* Current state: Draft
+* Author(s): @style95, @ningyougang, @keonhee, @jiangpengcheng, @upgle
+
+## Summary and Motivation
+
+This POEM proposes a new scheduler for OpenWhisk, FPCScheduler.
+Previously we revealed [performance issues](https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduling+v1) in OpenWhisk.
+There are many reasons for them, but we can summarize them into the following problems.
+
+1. Multiple actions share the same `homeInvoker`.
+2. Current scheduler does not consider time taking for container operations(create/delete, pause/unpause).
+3. Resources are evenly divided by the number of controllers.
+
+First, multiple actions share the same `homeInvoker`. This is because the `homeInvoker` is statically decided by a hash function.
+While scheduling, resource status in the invoker side is not considered and this introduces busy hotspot invokers while the others are idle.
+This is generally a good idea to increase the probability of container reuse, but it becomes bad in the real scene because of slow container operation.
+When all resources in an invoker are taken by one action(busy/warm), an activation for another action will remove one of the existing containers and create a new one for it.
+If actions run for a long time this would be a good approach in terms of bin-packing. But when actions run for short time(most of the serverless cases),
+it severely increases the response time because container operations take one to two orders of magnitude longer time than invocation and it results in bad performance.
+
+Second, since controllers schedule activations to invokers in a distributed environment, controllers are assigned resource fraction of invokers.
+It's not feasible to collect the status of invokers, scheduling decisions made by all controllers and schedule based on them in real time so that controllers schedule activations within their own partial invoker resources.
+As a result, controllers throttle activations even if cluster-wide invocations are less than limits.
+
+We propose FPCScheduler to address the above issues.
+The scheduler differs from the existing scheduler(`ShardingPoolBalancer`) in many directions.
+It schedules containers rather than function requests. Each container is supposed to pull activations requests from the given action queue continuously.
+Whenever one execution is over, it fetches the next activation requests and invokes it repeatedly. In this way, we can maximize container reuse.
+One more benefit is schedulers don't need to track/consider the location of existing containers. Schedulers just decide where and when to create more containers.
+It can just create more containers in invokers with enough resources. And this enables distributed scheduling with all invoker resources among multiple schedulers.
+
+Controllers no longer schedule activation requests. They are in charge of creating action queues and forwarding activation requests to the queues.
+The action queue is a dedicated queue for the given action and dynamically created/deleted in schedulers. Each action has its own queue, so there is no interference among actions because of
+a shared queue.
+
+
+ETCD, a distributed and reliable key-value store is used for a transaction, health-check, cluster information sharing, etc.
+Each scheduler performs a transaction via ETCD when scheduling. The health status of each component(scheduler, invoker) is managed by [Lease](https://help.compose.com/docs/etcd-using-etcd3-features#leases) in ETCD.
+Whenever a component is failed, it no longer sends keepalive requests, and its health data is removed by lease timeout.
+Cluster-wide information such as scheduler endpoints, queue endpoints, containers in a namespace, and throttling is stored in ETCD and referred by corresponding components.
+Controllers throttle namespaces based on the throttling data in ETCD. So all controllers share the same view against the resources and manage them in the same way.
+
+One more benefit of having our own component for routing rather than utilizing open-source components such as Kafka is we can extend and implement any routing logic.
+We would want various routing policies at some point. For example, when we utilize multiple versions of an action in-flight in the future we might want to control the traffic ratio between the two versions,
+we can route activations only to invokers with a specific resource, we may want to have dedicated invokers for some namespaces and so on. And this POEM could be a baseline for such extensions.
+
+
+## Proposed changes: Architecture Diagram (optional), and Design
+The design document along architecture diagram is already shared in [OpenWhisk Wiki](https://cwiki.apache.org/confluence/display/OPENWHISK/Apache+OpenWhisk+Project+Wiki?src=sidebar)
+
+* [Design Consideration](https://cwiki.apache.org/confluence/display/OPENWHISK/Design+consideration?src=contextnavpagetreemode)
+* [Architecture](https://cwiki.apache.org/confluence/display/OPENWHISK/System+Architecture)
+* [Component Design](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design)
+
+### Implementation details
+
+For the record, we(Naver) are already operating OpenWhisk with this scheduler in a production environment.
+We want to contribute the scheduler and hope it evolves with the community.
+
+There are many [new components](https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design).
+
+#### Common components
+We store data in ETCD, there are many relevant components such as `EtcdClient`, `LeaseKeepAliveService`, `WatcherService`, `DataManagementService`, etc.
+
+* `EtcdClient`: An ETCD client offering basic CRUD operations.
+* `DataManagementService`: In charge of storing/recovering data in ETCD. It internally utilizes `LeaseKeepAliveService`, and `WatcherService`.
+* `LeaseKeepAliveService`: In charge of keeping the given lease alive.
+* `WatcherService`: Watches the given keys and receives events for the keys.
+
+
+#### Scheduler components
+
+In an abstract view, schedulers provide queueing, container scheduling, and activation routing.
+
+* `QueueManager`: The main entry point for queue creation request. It has references to all queues.
+* `MemoryQueue`: Dynamically created/deleted for each action. It watches the incoming/outgoing requests and triggers container creation.
+* `ContainerManager`: Schedule container creation requests to appropriate invokers.
+* `ActivationServiceImpl`: Provide API for containers to fetch activations via Akka-grpc. It works in a long-poll way to avoid busy-waiting.
+
+#### Controller components
+* `FPCPoolBalancer`: Create queues if not exist, and forward messages to them.
+* `FPCEntitlementProvider`: Throttle activations based on throttling information in ETCD.
+
+#### Invoker components.
+
+* `FunctionPullingContainerPool`: A container pool for function pulling container. It handles the container creation requests.
+* `FunctionPullingContainerProxy`: A proxy for a container. It repeatedly fetches activations and invokes them.
+* `ActivationClientProxy`: The Akka-grpc client. It communicates with `ActivationServiceImpl` in schedulers.
+* `InvokerHealthManager`: Manages the health and resources information of invokers. The data is stored in ETCD. If an invoker becomes unhealthy, it invokes health activations.
+
+## Future work.
+
+#### Persistence
+
+We reached a conclusion that the persistence of activation requests is not that mandatory requirement along with the at-most-once nature and circuit-breaking of OpenWhisk.
+But if it is desired, we can implement a persistent queue rather than an in-memory queue.
+
+#### Multiple partitions
+
+Currently, a queue only has one partition. Since there can be multiple queues for each action, cluster-wide RPS(request per second) linearly increases.
+If high RPS for one action is required, we can implement partitioning in queues to introduce parallel processing like what Kafka does.
+But we already confirmed 100K RPS with 10 queues and 10 commodity invokers. And the performance would linearly increase with more invokers and queues.
+
+

Review comment:
       ```suggestion
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org