You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2021/05/07 09:17:29 UTC
[GitHub] [openwhisk] style95 opened a new pull request #5110: Add memory queue for the new scheduler
style95 opened a new pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110
## Description
This PR adds one of the core implementations of the new scheduler.
This queue is dynamically created for each action and buffer activations in an internal queue.
It provisions more containers according to the number of activations in the queue.
This process is called "scheduling" and it is handled by another component `SchedulingDecisionMaker`.
This queue is designed in a way that activations are not persistent.
But it could be improved in the future by taking a similar approach with what Kafka does.
## My changes affect the following components
<!--- Select below all system components are affected by your change. -->
<!--- Enter an `x` in all applicable boxes. -->
- [ ] API
- [ ] Controller
- [x] Message Bus (e.g., Kafka)
- [ ] Loadbalancer
- [x] Scheduler
- [x] Invoker
- [ ] Intrinsic actions (e.g., sequences, conductors)
- [ ] Data stores (e.g., CouchDB)
- [x] Tests
- [ ] Deployment
- [ ] CLI
- [ ] General tooling
- [ ] Documentation
## Types of changes
<!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
- [ ] Bug fix (generally a non-breaking change which closes an issue).
- [x] Enhancement or new feature (adds new functionality).
- [x] Breaking change (a bug fix or enhancement which changes existing behavior).
## Checklist:
<!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
- [x] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
- [x] I reviewed the [style guides](https://github.com/apache/openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :).
- [x] I added tests to cover my changes.
- [x] My changes require further changes to the documentation.
- [] I updated the documentation where necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6661b33) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **increase** coverage by `20.29%`.
> The diff coverage is `0.86%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
+ Coverage 43.88% 64.18% +20.29%
===========================================
Files 231 234 +3
Lines 12807 13381 +574
Branches 528 513 -15
===========================================
+ Hits 5620 8588 +2968
+ Misses 7187 4793 -2394
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `68.49% <0.00%> (-3.98%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `93.78% <100.00%> (-1.87%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `60.00% <100.00%> (+37.77%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `88.00% <100.00%> (+20.65%)` | :arrow_up: |
| [...che/openwhisk/core/invoker/LogStoreCollector.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9Mb2dTdG9yZUNvbGxlY3Rvci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...hisk/core/scheduler/message/ContainerMessage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvbWVzc2FnZS9Db250YWluZXJNZXNzYWdlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../openwhisk/core/scheduler/FPCSchedulerServer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvRlBDU2NoZWR1bGVyU2VydmVyLnNjYWxh) | `0.00% <0.00%> (-95.84%)` | :arrow_down: |
| ... and [161 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...6661b33](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d2668a2) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **increase** coverage by `22.26%`.
> The diff coverage is `0.86%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
+ Coverage 43.73% 66.00% +22.26%
===========================================
Files 231 227 -4
Lines 12807 13121 +314
Branches 528 544 +16
===========================================
+ Hits 5601 8660 +3059
+ Misses 7206 4461 -2745
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `70.13% <0.00%> (-2.33%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.67% <100.00%> (+0.02%)` | :arrow_up: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `60.00% <100.00%> (+37.77%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `88.00% <100.00%> (+20.65%)` | :arrow_up: |
| [...hisk/core/scheduler/message/ContainerMessage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvbWVzc2FnZS9Db250YWluZXJNZXNzYWdlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../openwhisk/core/scheduler/FPCSchedulerServer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvRlBDU2NoZWR1bGVyU2VydmVyLnNjYWxh) | `0.00% <0.00%> (-95.84%)` | :arrow_down: |
| [...sk/core/scheduler/container/ContainerManager.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvY29udGFpbmVyL0NvbnRhaW5lck1hbmFnZXIuc2NhbGE=) | `0.00% <0.00%> (-88.72%)` | :arrow_down: |
| ... and [159 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...d2668a2](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r657495593
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/MemoryQueue.scala
##########
@@ -0,0 +1,1097 @@
+package org.apache.openwhisk.core.scheduler.queue
+
+import java.time.{Duration, Instant}
+
+import akka.actor.Status.{Failure => FailureMessage}
+import akka.actor.{ActorRef, ActorSystem, Cancellable, FSM, Props, Stash}
+import akka.util.Timeout
+import org.apache.openwhisk.common._
+import org.apache.openwhisk.core.ack.ActiveAck
+import org.apache.openwhisk.core.connector.ContainerCreationError.{TooManyConcurrentRequests, ZeroNamespaceLimit}
+import org.apache.openwhisk.core.connector._
+import org.apache.openwhisk.core.database.{NoDocumentException, UserContext}
+import org.apache.openwhisk.core.entity._
+import org.apache.openwhisk.core.entity.size._
+import org.apache.openwhisk.core.etcd.EtcdClient
+import org.apache.openwhisk.core.etcd.EtcdKV.ContainerKeys.containerPrefix
+import org.apache.openwhisk.core.etcd.EtcdKV.{ContainerKeys, QueueKeys, ThrottlingKeys}
+import org.apache.openwhisk.core.scheduler.SchedulerEndpoints
+import org.apache.openwhisk.core.scheduler.grpc.{GetActivation, ActivationResponse => GetActivationResponse}
+import org.apache.openwhisk.core.scheduler.message.{
+ ContainerCreation,
+ ContainerDeletion,
+ FailedCreationJob,
+ SuccessfulCreationJob
+}
+import org.apache.openwhisk.core.service._
+import org.apache.openwhisk.core.{ConfigKeys, WhiskConfig}
+import org.apache.openwhisk.http.Messages.{namespaceLimitUnderZero, tooManyConcurrentRequests}
+import pureconfig.generic.auto._
+import pureconfig.loadConfigOrThrow
+import spray.json._
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+import scala.collection.mutable
+import scala.concurrent.duration._
+import scala.concurrent.{duration, ExecutionContextExecutor, Future, Promise}
+import scala.language.postfixOps
+import scala.util.{Failure, Success}
+
+// States
+sealed trait MemoryQueueState
+case object Uninitialized extends MemoryQueueState
+case object Running extends MemoryQueueState
+case object Idle extends MemoryQueueState
+case object Flushing extends MemoryQueueState
+case object Removing extends MemoryQueueState
+case object Removed extends MemoryQueueState
+case object ActionThrottled extends MemoryQueueState
+case object NamespaceThrottled extends MemoryQueueState
+
+// Data
+sealed abstract class MemoryQueueData()
+case class NoData() extends MemoryQueueData()
+case class NoActors() extends MemoryQueueData()
+case class RunningData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class ThrottledData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class FlushingData(schedulerActor: ActorRef,
+ droppingActor: ActorRef,
+ error: ContainerCreationError,
+ reason: String,
+ activeDuringFlush: Boolean = false)
+ extends MemoryQueueData()
+case class RemovingData(schedulerActor: ActorRef, droppingActor: ActorRef, outdated: Boolean) extends MemoryQueueData()
+
+// Events sent by the actor
+case class QueueRemoved(invocationNamespace: String, action: DocInfo, leaderKey: Option[String])
+case class QueueReactivated(invocationNamespace: String, action: FullyQualifiedEntityName, docInfo: DocInfo)
+case class CancelPoll(promise: Promise[Either[MemoryQueueError, ActivationMessage]])
+case object QueueRemovedCompleted
+case object FlushPulse
+
+// Events received by the actor
+case object Start
+case object VersionUpdated
+case object StopSchedulingAsOutdated
+
+sealed trait RequiredAction
+case object Skip extends RequiredAction
+case object AddInitialContainer extends RequiredAction
+case object AddContainer extends RequiredAction
+case class EnableNamespaceThrottling(dropMsg: Boolean) extends RequiredAction
+case object DisableNamespaceThrottling extends RequiredAction
+case object EnableActionThrottling extends RequiredAction
+case object DisableActionThrottling extends RequiredAction
+case object Pausing extends RequiredAction
+case class DecisionResults(required: RequiredAction, num: Int)
+
+case class TimeSeriesActivationEntry(timestamp: Instant, msg: ActivationMessage)
+
+class MemoryQueue(private val etcdClient: EtcdClient,
+ private val durationChecker: DurationChecker,
+ private val action: FullyQualifiedEntityName,
+ messagingProducer: MessageProducer,
+ config: WhiskConfig,
+ invocationNamespace: String,
+ revision: DocRevision,
+ endpoints: SchedulerEndpoints,
+ actionMetaData: WhiskActionMetaData,
+ dataManagementService: ActorRef,
+ watcherService: ActorRef,
+ containerManager: ActorRef,
+ decisionMaker: ActorRef,
+ schedulerId: SchedulerInstanceId,
+ ack: ActiveAck,
+ store: (TransactionId, WhiskActivation, UserContext) => Future[Any],
+ getUserLimit: String => Future[Int],
+ checkToDropStaleActivation: (Queue[TimeSeriesActivationEntry],
+ Long,
+ String,
+ FullyQualifiedEntityName,
+ MemoryQueueState,
+ ActorRef) => Unit,
+ queueConfig: QueueConfig)(implicit logging: Logging)
+ extends FSM[MemoryQueueState, MemoryQueueData]
+ with Stash {
+
+ private implicit val ec: ExecutionContextExecutor = context.dispatcher
+ private implicit val actorSystem: ActorSystem = context.system
+ private implicit val timeout = Timeout(5.seconds)
+ private implicit val order: Ordering[BufferedRequest] = Ordering.by(_.containerId)
+
+ private val unversionedAction = action.copy(version = None)
+ private val checkInterval: FiniteDuration = 100 milliseconds
+ private val StaleThreshold: Double = 100.0
+ private val StaleDuration = Duration.ofMillis(StaleThreshold.toLong)
+ private val dropInterval: FiniteDuration = 10 seconds
+ private val leaderKey = QueueKeys.queue(invocationNamespace, unversionedAction, leader = true)
+ private val inProgressContainerPrefixKey =
+ containerPrefix(ContainerKeys.inProgressPrefix, invocationNamespace, action, Some(revision))
+ private val existingContainerPrefixKey =
+ containerPrefix(ContainerKeys.namespacePrefix, invocationNamespace, action, Some(revision))
+ private val namespaceThrottlingKey = ThrottlingKeys.namespace(EntityName(invocationNamespace))
+ private val actionThrottlingKey = ThrottlingKeys.action(invocationNamespace, unversionedAction)
+ private val pollTimeOut = 1.seconds
+ private var requestBuffer = mutable.PriorityQueue.empty[BufferedRequest]
+ private val memory = actionMetaData.limits.memory.megabytes.MB
+ private val queueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), Some(leaderKey))
+ private val staleQueueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), None)
+
+ private[queue] var containers = Set.empty[String]
+ private[queue] var creationIds = Set.empty[String]
+
+ private[queue] var queue = Queue.empty[TimeSeriesActivationEntry]
+ private[queue] var in: Int = 0
+ private[queue] val namespaceContainerCount = NamespaceContainerCount(invocationNamespace, etcdClient, watcherService)
+ private[queue] var averageDuration: Option[Double] = None
+ private[queue] var averageDurationBuffer = AverageRingBuffer(queueConfig.durationBufferSize)
+ private[queue] var limit: Option[Int] = None
+ private[queue] var initialized = false
+
+ private val logScheduler: Cancellable = context.system.scheduler.scheduleWithFixedDelay(0.seconds, 1.seconds) { () =>
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_QUEUE_WAITING_ACTIVATION(s"$invocationNamespace/$action"),
+ queue.size)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_CONTAINER(invocationNamespace),
+ namespaceContainerCount.existingContainerNumByNamespace)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_INPROGRESS_CONTAINER(invocationNamespace),
+ namespaceContainerCount.inProgressContainerNumByNamespace)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_CONTAINER(invocationNamespace, action.asString),
+ containers.size)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_INPROGRESS_CONTAINER(invocationNamespace, action.asString),
+ creationIds.size)
+ }
+
+ getAverageDuration()
+
+ private val watcherName = s"memory-queue-$action-$revision"
+ // watch existing containers for action and namespace
+ private val watchedKeys = Seq(inProgressContainerPrefixKey, existingContainerPrefixKey)
+
+ watchedKeys.foreach { key =>
+ watcherService ! WatchEndpoint(key, "", isPrefix = true, watcherName, Set(PutEvent, DeleteEvent))
+ }
+
+ startWith(Uninitialized, NoData())
+
+ when(Uninitialized) {
+ case Event(Start, _) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] a new queue is created.")
+ val (schedulerActor, droppingActor) = startMonitoring()
+ initializeThrottling()
+
+ watcherService ! WatchEndpoint(leaderKey, endpoints.serialize, isPrefix = false, watcherName, Set(DeleteEvent))
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // this is the case that the action version is updated, so no data needs to be stored
+ case Event(VersionUpdated, _) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // other messages should not be handled in this state.
+ case _ =>
+ stash()
+ stay
+ }
+
+ when(Running, stateTimeout = queueConfig.idleGrace) {
+ case Event(EnableNamespaceThrottling(dropMsg), data: RunningData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Enable namespace throttling.")
+ enableNamespaceThrottling()
+
+ // if no container could be created, it is same with Flushing state.
+ if (dropMsg) {
+ completeAllActivations(tooManyConcurrentRequests, isWhiskError = false)
+ goto(Flushing) using FlushingData(
+ data.schedulerActor,
+ data.droppingActor,
+ TooManyConcurrentRequests,
+ tooManyConcurrentRequests)
+ } else {
+ // if there are already some containers running, activations can still be processed so goto the NamespaceThrottled state.
+ goto(NamespaceThrottled) using ThrottledData(data.schedulerActor, data.droppingActor)
+ }
+
+ case Event(StateTimeout, data: RunningData) =>
+ if (queue.isEmpty && (containers.size + creationIds.size) <= 0) {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] No activations coming in ${queueConfig.idleGrace}")
+ actorSystem.stop(data.schedulerActor)
+ actorSystem.stop(data.droppingActor)
+
+ goto(Idle) using NoActors()
+ } else {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue is timed out but there are still ${queue.size} activation messages or (running: ${containers.size}, in-progress: ${creationIds.size}) containers")
+ stay
+ }
+
+ case Event(FailedCreationJob(creationId, _, _, _, error, message), RunningData(schedulerActor, droppingActor)) =>
+ creationIds -= creationId.asString
+ // when there is no container, it moves to the Flushing state as no activations can be invoked
+ if (containers.size <= 0) {
+ val isWhiskError = ContainerCreationError.whiskErrors.contains(error)
+ completeAllActivations(message, isWhiskError)
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Failed to create an initial container due to ${if (isWhiskError) "whiskError"
+ else "developerError"}, reason: $message.")
+
+ goto(Flushing) using FlushingData(schedulerActor, droppingActor, error, message)
+ } else
+ // if there are already some containers running, activations can be handled anyway.
+ stay
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(NamespaceThrottled) {
+ case Event(msg: ActivationMessage, _: ThrottledData) =>
+ handleActivationMessage(msg)
+ stay
+
+ case Event(DisableNamespaceThrottling, data: ThrottledData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Disable namespace throttling.")
+ disableNamespaceThrottling()
+ goto(Running) using RunningData(data.schedulerActor, data.schedulerActor)
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(ActionThrottled) {
+ // since there are already too many activation messages, it drops the new messages
+ case Event(msg: ActivationMessage, ThrottledData(_, _)) =>
+ completeErrorActivation(msg, tooManyConcurrentRequests, isWhiskError = false)
+ stay
+ }
+
+ when(Idle, stateTimeout = queueConfig.stopGrace) {
+ case Event(msg: ActivationMessage, _: NoActors) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+ handleActivationMessage(msg)
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ case Event(request: GetActivation, _) if request.action == action =>
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ stay
+
+ case Event(StateTimeout, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] The queue is timed out, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(GracefulShutdown, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(StopSchedulingAsOutdated, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+
+ cleanUpWatcher()
+
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ // since the queue is outdated and there is no activation, delete all old containers.
+ containerManager ! ContainerDeletion(invocationNamespace, action, revision, actionMetaData)
+
+ goto(Removed) using NoData()
+ }
+
+ when(Flushing) {
+ // an initial container is successfully created.
+ case Event(SuccessfulCreationJob(creationId, _, _, _), FlushingData(schedulerActor, droppingActor, _, _, _)) =>
+ creationIds -= creationId.asString
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // log the failed information
+ case Event(FailedCreationJob(creationId, _, _, _, _, message), data: FlushingData) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Failed to create a container due to $message")
+
+ // keep updating the reason
+ stay using data.copy(reason = message)
+
+ // since there is no container, activations cannot be handled.
+ case Event(msg: ActivationMessage, data: FlushingData) =>
+ completeErrorActivation(msg, data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ stay() using data.copy(activeDuringFlush = true)
+
+ // Since SchedulingDecisionMaker keep sending a message to create a container, this state is not automatically timed out.
+ // Instead, StateTimeout message will be sent by a timer.
+ case Event(StateTimeout, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ if (data.activeDuringFlush)
+ stay using data.copy(activeDuringFlush = false)
+ else
+ cleanUpActorsAndGotoRemoved(data)
+
+ case Event(GracefulShutdown, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpActorsAndGotoRemoved(data)
+ }
+
+ // in case there is any activation in the queue, it waits until all of them are handled.
+ when(Removing, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // When there is no message in the queue, SchedulingDecisionMaker would stop sending any message
+ // So the queue can be timed out on every gracefulShutdownTimeout
+ case Event(QueueRemovedCompleted | StateTimeout, data: RemovingData) =>
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(GracefulShutdown, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received GracefulShutdown trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(StopSchedulingAsOutdated, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received StopSchedulingAsOutdated trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data.copy(outdated = true))
+ }
+
+ when(Removed, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // since this Queue will be terminated, rescheduling the msg
+ case Event(msg: ActivationMessage, _: NoData) =>
+ context.parent ! msg
+ stay()
+
+ // this queue is going to stop so let client connect to a new queue
+ case Event(request: GetActivation, _: NoData) if request.action == action =>
+ implicit val tid = request.transactionId
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Get activation request ${request.containerId}, let client connect to a new queue.")
+ forwardAllActivations(context.parent)
+ sender ! GetActivationResponse(Left(NoMemoryQueue()))
+
+ stay
+
+ // actors and data are already wiped
+ case Event(QueueRemovedCompleted, _: NoData) =>
+ stop()
+
+ // This is not supposed to happen. This will ensure the queue does not run forever.
+ // This can happen when QueueManager could not respond with QueueRemovedCompleted for some reason.
+ case Event(StateTimeout, _: NoData) =>
+ context.parent ! queueRemovedMsg
+
+ stop()
+
+ // This queue is going to stop, do nothing
+ case Event(msg @ (StopSchedulingAsOutdated | GracefulShutdown), _: NoData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received $msg but do nothing as it is going to stop.")
+ stay
+ }
+
+ whenUnhandled {
+ // The queue endpoint is removed, trying to restore it.
+ case Event(WatchEndpointRemoved(_, `leaderKey`, value, false), data) =>
+ data match {
+ case RemovingData(_, _, _) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] This queue is shutdown by `/disable` api, do nothing here.")
+ case _ =>
+ dataManagementService ! RegisterInitialData(leaderKey, value, failoverEnabled = false, Some(self)) // the watcher is already setup
+ }
+ stay
+
+ // we don't care the storage results for namespaceThrottlingKey
+ case Event(InitialDataStorageResults(`namespaceThrottlingKey`, _), _) =>
+ stay
+
+ // The queue endpoint is restored
+ case Event(InitialDataStorageResults(`leaderKey`, Right(_)), _) =>
+ stay
+
+ // this can be a case that there is another queue already running.
+ // it can happen if a node is segregated by the temporal network rupture and the queue endpoint is removed.
+ case Event(InitialDataStorageResults(`leaderKey`, Left(_)), data) =>
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] the queue is superseded by a new queue.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! queueRemovedMsg
+
+ // forward all activations to the parent queue manager.
+ // parent queue manager is supposed to removed the reference of this queue and forward messages to a new queue
+ forwardAllActivations(context.parent)
+
+ // only clean up actors because etcd data is already being used by another queue
+ cleanUpActors(data)
+
+ goto(Removed) using NoData()
+
+ case Event(WatchEndpointRemoved(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds -= key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers -= key.split("/").last
+ case _ =>
+ }
+ stay
+
+ case Event(WatchEndpointInserted(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds += key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers += key.split("/").last
+ case _ =>
+ }
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled
+ case Event(SuccessfulCreationJob(creationId, _, _, _), _) =>
+ creationIds -= creationId.asString
+ stay()
+
+ // for other cases
+ case Event(FailedCreationJob(creationId, invocationNamespace, action, revision, _, message), _) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Got failed creation job with revision $revision and error $message.")
+ stay()
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(cancel: CancelPoll, _) =>
+ cancel.promise.trySuccess(Left(NoActivationMessage()))
+
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(msg: ActivationMessage, _) =>
+ handleActivationMessage(msg)
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action == action =>
+ implicit val tid = request.transactionId
+ if (request.alive) {
+ containers += request.containerId
+ handleActivationRequest(request)
+ } else {
+ logging.info(this, s"Remove containerId because ${request.containerId} is not alive")
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ containers -= request.containerId
+ stay
+ }
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action != action =>
+ implicit val tid = request.transactionId
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] version mismatch ${request.action}")
+ sender ! GetActivationResponse(Left(ActionMismatch()))
+
+ stay
+
+ case Event(DropOld, _) =>
+ if (queue.nonEmpty && Duration
+ .between(queue.head.timestamp, Instant.now)
+ .compareTo(Duration.ofMillis(queueConfig.maxRetentionMs)) < 0) {
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Drop some stale activations for $revision, existing container is ${containers.size}, inProgress container is ${creationIds.size}, state data: $stateData, in is $in, current: ${queue.size}.")
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] the head stale message: ${queue.head.msg.activationId}")
+ }
+ queue = MemoryQueue.dropOld(queue, Duration.ofMillis(queueConfig.maxRetentionMs), completeErrorActivation)
+
+ stay
+
+ // common case for all statuses
+ case Event(StatusQuery, _) =>
+ sender ! StatusData(invocationNamespace, action.asString, queue.size, stateName.toString, stateData.toString)
+ stay
+
+ // Common case for all cases
+ case Event(GracefulShutdown, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Gracefully shutdown the memory queue.")
+ // delete relative data, e.g leaderKey, namespaceThrottlingKey, actionThrottlingKey
+ cleanUpData()
+
+ // let queue manager knows this queue is going to stop and let it forward incoming activations to a new queue
+ context.parent ! queueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = false)
+
+ // the version is updated. it's a shared case for all states
+ case Event(StopSchedulingAsOutdated, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = true)
+
+ case Event(t: FailureMessage, _) =>
+ logging.error(this, s"[$invocationNamespace:$action:$stateName] got an unexpected failure message: $t")
+
+ stay
+
+ case Event(msg: DecisionResults, _) =>
+ val DecisionResults(result, num) = msg
+ result match {
+ case AddInitialContainer if num > 0 =>
+ initialized = true
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+
+ case AddContainer if num > 0 =>
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+
+ case enable: EnableNamespaceThrottling =>
+ if (num > 0) {
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+ }
+ self ! enable
+
+ case DisableNamespaceThrottling =>
+ if (num > 0) {
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+ }
+ self ! DisableNamespaceThrottling
+
+ case Pausing =>
+ logging.warn(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The limit value is less than 0. No activation can be handled so the queue becomes the Flushing state.")
+ self ! FailedCreationJob(
+ CreationId.void,
+ invocationNamespace,
+ action,
+ revision,
+ ZeroNamespaceLimit,
+ namespaceLimitUnderZero)
+ }
+ stay
+
+ // this should not happen
+ case otherMsg =>
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] received unexpected message: $otherMsg")
+
+ stay
+ }
+
+ onTransition {
+ case Uninitialized -> _ => unstashAll()
+ case _ -> Flushing => startTimerWithFixedDelay("StopQueue", StateTimeout, queueConfig.flushGrace)
+ case Flushing -> _ => cancelTimer("StopQueue")
+ }
+
+ onTermination {
+ case _ =>
+ // logscheduler must be canceled when FSM is terminated
+ logScheduler.cancel()
+
+ // the lifecycle of DecisionMaker conforms to the one of MemoryQueue
+ actorSystem.stop(decisionMaker)
+ }
+
+ initialize()
+
+ private def cleanUpDataAndGotoRemoved() = {
+ cleanUpWatcher()
+ cleanUpData()
+ context.parent ! queueRemovedMsg
+
+ goto(Removed) using NoData()
+ }
+
+ private def cleanUpActorsAndGotoRemoved(data: FlushingData) = {
+ cleanUpActors(data)
+ cleanUpData()
+
+ context.parent ! queueRemovedMsg
+
+ goto(Removed) using NoData()
+ }
+
+ private def cleanUpActorsAndGotoRemovedIfPossible(data: RemovingData) = {
+ requestBuffer = requestBuffer.filter(!_.promise.isCompleted)
+ if (queue.isEmpty && requestBuffer.isEmpty) {
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] No activation exist. Shutdown the queue.")
+ // it can be safely called multiple times as it's idempotent
+ cleanUpActors(data)
+
+ // if the queue is outdated, remove old containers.
+ if (data.outdated) {
+ // let the container manager know this version of containers are outdated.
+ containerManager ! ContainerDeletion(invocationNamespace, action, revision, actionMetaData)
+ }
+ self ! QueueRemovedCompleted
+
+ goto(Removed) using NoData()
+ } else {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Queue is going to stop but there are still ${queue.size} activations and ${requestBuffer.size} request buffered.")
+ stay // waiting for next timeout
+ }
+ }
+
+ private def getRemovingData(data: MemoryQueueData, outdated: Boolean): MemoryQueueData = {
+ data match {
+ case RunningData(schedulerActor, droppingActor) =>
+ RemovingData(schedulerActor, droppingActor, outdated)
+ case ThrottledData(schedulerActor, droppingActor) =>
+ RemovingData(schedulerActor, droppingActor, outdated)
+ case FlushingData(schedulerActor, droppingActor, _, _, _) =>
+ RemovingData(schedulerActor, droppingActor, outdated)
+ case data: RemovingData =>
+ data.copy(outdated = outdated)
+ case _ =>
+ NoData()
+ }
+ }
+ private def cleanUpWatcher(): Unit = {
+ watchedKeys.foreach { key =>
+ watcherService ! UnwatchEndpoint(key, isPrefix = true, watcherName)
+ }
+ watcherService ! UnwatchEndpoint(leaderKey, isPrefix = false, watcherName)
+ namespaceContainerCount.close()
+ }
+
+ private def cleanUpActors(data: MemoryQueueData): Unit = {
+ cleanUpWatcher()
+
+ data match {
+ case RunningData(schedulerActor, droppingActor) =>
+ actorSystem.stop(schedulerActor)
+ actorSystem.stop(droppingActor)
+
+ case ThrottledData(schedulerActor, droppingActor) =>
+ actorSystem.stop(schedulerActor)
+ actorSystem.stop(droppingActor)
+
+ case FlushingData(schedulerActor, droppingActor, _, _, _) =>
+ actorSystem.stop(schedulerActor)
+ actorSystem.stop(droppingActor)
+
+ case RemovingData(schedulerActor, droppingActor, _) =>
+ actorSystem.stop(schedulerActor)
+ actorSystem.stop(droppingActor)
+
+ case _ => // do nothing
+ }
+ }
+
+ private def cleanUpData(): Unit = {
+ dataManagementService ! UnregisterData(leaderKey)
+ dataManagementService ! UnregisterData(namespaceThrottlingKey)
+ dataManagementService ! UnregisterData(actionThrottlingKey)
+ }
+
+ private def initializeThrottling() = {
+ dataManagementService ! RegisterInitialData(namespaceThrottlingKey, false.toString, failoverEnabled = false)
+ dataManagementService ! RegisterData(actionThrottlingKey, false.toString, failoverEnabled = false)
+ }
+
+ private def tryEnableActionThrottling() = {
+ if (queue.size >= queueConfig.maxRetentionSize && stateName != ActionThrottled) {
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Enable action throttling.")
+ dataManagementService ! RegisterData(actionThrottlingKey, true.toString, failoverEnabled = false)
+
+ stateData match {
+ case RunningData(schedulerActor, droppingActor) =>
+ goto(ActionThrottled) using ThrottledData(schedulerActor, droppingActor)
+ case _ =>
+ stay
+ }
+ } else {
+ stay
+ }
+ }
+
+ private def tryDisableActionThrottling()(implicit tid: TransactionId) = {
+ (stateName, stateData) match {
+ case (ActionThrottled, ThrottledData(schedulerActor, droppingActor))
+ if queue.size <= queueConfig.maxRetentionSize * queueConfig.throttlingFraction =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Disable action throttling.")
+ dataManagementService ! RegisterData(actionThrottlingKey, false.toString, failoverEnabled = false)
+
+ // at this point, namespace throttling might be enabled,
+ // then the state will be changed to NamespaceThrottled automatically at the next tick
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+ case _ => stay
+ }
+ }
+
+ private def disableNamespaceThrottling() = {
+ dataManagementService ! RegisterData(namespaceThrottlingKey, false.toString, failoverEnabled = false)
+ }
+
+ private def enableNamespaceThrottling() = {
+ dataManagementService ! RegisterData(namespaceThrottlingKey, true.toString, failoverEnabled = false)
+ }
+
+ private def completeErrorActivation(activation: ActivationMessage,
+ message: String,
+ isWhiskError: Boolean): Future[Any] = {
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] complete activation ${activation.activationId} with error $message")(
+ activation.transid)
+ val activationResponse =
+ if (isWhiskError)
+ generateFallbackActivation(activation, ActivationResponse.whiskError(message))
+ else
+ generateFallbackActivation(activation, ActivationResponse.developerError(message))
+
+ // TODO change scheduler instance id
+ val instance = InvokerInstanceId(0, userMemory = 0.MB)
+
+ val ackMsg = if (activation.blocking) {
+ CombinedCompletionAndResultMessage(activation.transid, activationResponse, instance)
+ } else {
+ CompletionMessage(activation.transid, activationResponse, instance)
+ }
+
+ if (!isWhiskError && message == tooManyConcurrentRequests) {
+ val metric = Metric("ConcurrentRateLimit", 1)
+ UserEvents.send(
+ messagingProducer,
+ EventMessage(
+ schedulerId.toString,
+ metric,
+ activation.user.subject,
+ invocationNamespace,
+ activation.user.namespace.uuid,
+ metric.typeName))
+ }
+
+ ack(
+ activation.transid,
+ activationResponse,
+ activation.blocking,
+ activation.rootControllerIndex,
+ activation.user.namespace.uuid,
+ ackMsg)
+ .andThen {
+ case Failure(t) =>
+ logging.error(this, s"[$invocationNamespace:$action:$stateName] failed to send ack due to $t")
+ }
+ store(activation.transid, activationResponse, UserContext(activation.user))
+ .andThen {
+ case Failure(t) =>
+ logging.error(this, s"[$invocationNamespace:$action:$stateName] failed to store activation due to $t")
+ }
+ }
+
+ private def forwardAllActivations(queueManager: ActorRef): Unit = {
+ while (queue.nonEmpty) {
+ val (TimeSeriesActivationEntry(_, msg), newQueue) = queue.dequeue
+ queue = newQueue
+ logging.info(this, s"Forward msg ${msg.activationId} to the queue manager")(msg.transid)
+ queueManager ! msg
+ }
+ }
+
+ private def completeAllActivations(reason: String, isWhiskError: Boolean): Unit = {
+ while (queue.nonEmpty) {
+ val (TimeSeriesActivationEntry(_, msg), newQueue) = queue.dequeue
+ queue = newQueue
+ completeErrorActivation(msg, reason, isWhiskError)
+ }
+ }
+
+ // since there is no initial delay, it will try to create a container at initialization time
+ // these schedulers will run forever and stop when the memory queue stops
+ private def startMonitoring(): (ActorRef, ActorRef) = {
+ val droppingScheduler = Scheduler.scheduleWaitAtLeast(dropInterval) { () =>
+ checkToDropStaleActivation(queue, queueConfig.maxRetentionMs, invocationNamespace, action, stateName, self)
+ Future.successful(())
+ }
+
+ val monitoringScheduler = Scheduler.scheduleWaitAtLeast(checkInterval) { () =>
+ // the average duration is updated every checkInterval
+ if (averageDurationBuffer.nonEmpty) {
+ averageDuration = Some(averageDurationBuffer.average)
+ }
+ getUserLimit(invocationNamespace).andThen {
+ case Success(limit) =>
+ decisionMaker ! QueueSnapshot(
+ initialized,
+ in,
+ queue.size,
+ containers.size,
+ creationIds.size,
+ getStaleActivationNum(0, queue),
+ namespaceContainerCount.existingContainerNumByNamespace,
+ namespaceContainerCount.inProgressContainerNumByNamespace,
+ averageDuration,
+ limit,
+ stateName,
+ self)
+ case Failure(_: NoDocumentException) =>
+ // no limit available for the namespace
+ self ! StopSchedulingAsOutdated
+ }
+ }
+ (monitoringScheduler, droppingScheduler)
+ }
+
+ private def getAverageDuration() = {
+ // check the duration only once
+ actorSystem.scheduler.scheduleOnce(duration.Duration.Zero) {
+ durationChecker.checkAverageDuration(invocationNamespace, actionMetaData) { durationCheckResult =>
+ if (durationCheckResult.hitCount > 0) {
+ averageDuration = durationCheckResult.averageDuration
+ }
+ durationCheckResult
+ }
+ }
+ }
+
+ @tailrec
+ private def getStaleActivationNum(count: Int, queue: Queue[TimeSeriesActivationEntry]): Int = {
+ if (queue.isEmpty || Duration
+ .between(queue.head.timestamp, Instant.now)
+ .compareTo(StaleDuration) < 0) count
+ else
+ getStaleActivationNum(count + 1, queue.tail)
+ }
+
+ private def generateContainerCreationMessages(num: Int) = {
+ (1 to num).map { _ =>
+ val msg = ContainerCreationMessage(
+ TransactionId.containerCreation,
+ invocationNamespace,
+ action,
+ revision,
+ actionMetaData,
+ schedulerId,
+ endpoints.host,
+ endpoints.rpcPort)
+ creationIds += msg.creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Try to create a new container with creationId ${msg.creationId.asString}")
+ msg
+ }.toList
+ }
+
+ /* take the first uncompleted request from requestBuffer. */
+ private def takeUncompletedRequest(): Option[Promise[Either[MemoryQueueError, ActivationMessage]]] = {
+ requestBuffer = requestBuffer.filter(!_.promise.isCompleted)
+ if (requestBuffer.nonEmpty) {
+ Some(requestBuffer.dequeue.promise)
+ } else None
+ }
+
+ private def handleActivationMessage(msg: ActivationMessage) = {
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] got a new activation message ${msg.activationId}")(
+ msg.transid)
+ in += 1
+ takeUncompletedRequest()
+ .map { res =>
+ res.trySuccess(Right(msg))
+ in -= 1
Review comment:
Pretty sure this isn't thread safe if `takeUncompletedRequest` is a promise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (954d4f8) into [master](https://codecov.io/gh/apache/openwhisk/commit/f1829e1160d0aca51fc3ff93a77621c488b148ad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1829e1) will **decrease** coverage by `12.87%`.
> The diff coverage is `2.22%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
- Coverage 81.26% 68.39% -12.88%
===========================================
Files 218 228 +10
Lines 11158 11994 +836
Branches 497 529 +32
===========================================
- Hits 9068 8203 -865
- Misses 2090 3791 +1701
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `71.88% <0.00%> (-13.76%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `93.18% <0.00%> (-1.08%)` | :arrow_down: |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `66.83% <66.66%> (-2.08%)` | :arrow_down: |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.59% <100.00%> (+0.02%)` | :arrow_up: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `30.00% <100.00%> (+7.77%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `90.00% <100.00%> (+0.20%)` | :arrow_up: |
| ... and [46 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1829e1...954d4f8](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6661b33) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **increase** coverage by `30.72%`.
> The diff coverage is `80.62%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
+ Coverage 43.88% 74.60% +30.72%
===========================================
Files 231 234 +3
Lines 12807 13381 +574
Branches 528 513 -15
===========================================
+ Hits 5620 9983 +4363
+ Misses 7187 3398 -3789
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `27.27% <27.27%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `73.17% <73.17%> (ø)` | |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `83.73% <83.73%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `78.99% <100.00%> (+6.53%)` | :arrow_up: |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.65% <100.00%> (ø)` | |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `60.00% <100.00%> (+37.77%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `88.00% <100.00%> (+20.65%)` | :arrow_up: |
| [...whisk/connector/kafka/KafkaProducerConnector.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYWZrYVByb2R1Y2VyQ29ubmVjdG9yLnNjYWxh) | `55.00% <0.00%> (-32.50%)` | :arrow_down: |
| [...whisk/connector/kafka/KafkaConsumerConnector.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYWZrYUNvbnN1bWVyQ29ubmVjdG9yLnNjYWxh) | `59.15% <0.00%> (-22.54%)` | :arrow_down: |
| [...pache/openwhisk/core/invoker/InvokerReactive.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9JbnZva2VyUmVhY3RpdmUuc2NhbGE=) | `57.75% <0.00%> (-21.56%)` | :arrow_down: |
| ... and [142 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...6661b33](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-867198560
This is probably the most complicated PR, but did the best I could to understand at a high level I think LGTM a part from my comments.
One comment on the persistence nature of the queue. I think queueing with kafka doesn't add anything to openwhisk if openwhisk does not provide at least once guarantee of activation execution. It's unnecessary overhead to queue up through kafka if the activation can get lost elsewhere anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (364cc8e) into [master](https://codecov.io/gh/apache/openwhisk/commit/ecb2a980659f28d0adbd9ef837afaf4cb2b695bf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ecb2a98) will **decrease** coverage by `24.03%`.
> The diff coverage is `1.40%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
==========================================
- Coverage 28.91% 4.88% -24.04%
==========================================
Files 219 231 +12
Lines 11775 12815 +1040
Branches 506 555 +49
==========================================
- Hits 3405 626 -2779
- Misses 8370 12189 +3819
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `41.01% <0.00%> (-28.79%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `34.83% <0.00%> (-57.22%)` | :arrow_down: |
| [...rpool/logging/DockerToActivationFileLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL0RvY2tlclRvQWN0aXZhdGlvbkZpbGVMb2dTdG9yZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...ainerpool/logging/DockerToActivationLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL0RvY2tlclRvQWN0aXZhdGlvbkxvZ1N0b3JlLnNjYWxh) | `0.00% <0.00%> (-96.97%)` | :arrow_down: |
| [.../containerpool/logging/ElasticSearchLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL0VsYXN0aWNTZWFyY2hMb2dTdG9yZS5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...sk/core/containerpool/logging/SplunkLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL1NwbHVua0xvZ1N0b3JlLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...penwhisk/core/database/ActivationFileStorage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQWN0aXZhdGlvbkZpbGVTdG9yYWdlLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...ache/openwhisk/core/database/ActivationStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQWN0aXZhdGlvblN0b3JlLnNjYWxh) | `0.00% <ø> (-92.31%)` | :arrow_down: |
| [...nwhisk/core/database/ArtifactActivationStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQXJ0aWZhY3RBY3RpdmF0aW9uU3RvcmUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| ... and [172 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [99fb2c5...364cc8e](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-867198560
This is probably the most complicated PR, but did the best I could to understand at a high level I think LGTM apart from my comments.
One comment on the persistence nature of the queue. I think queueing with kafka doesn't add anything to openwhisk if openwhisk does not provide at least once guarantee of activation execution. It's unnecessary overhead to queue up through kafka if the activation can get lost elsewhere anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (99c95cd) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **increase** coverage by `23.14%`.
> The diff coverage is `0.86%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
+ Coverage 43.88% 67.02% +23.14%
===========================================
Files 231 234 +3
Lines 12807 13381 +574
Branches 528 513 -15
===========================================
+ Hits 5620 8969 +3349
+ Misses 7187 4412 -2775
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `70.77% <0.00%> (-1.69%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.65% <100.00%> (ø)` | |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `60.00% <100.00%> (+37.77%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `90.00% <100.00%> (+22.65%)` | :arrow_up: |
| [...hisk/core/scheduler/message/ContainerMessage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvbWVzc2FnZS9Db250YWluZXJNZXNzYWdlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../openwhisk/core/scheduler/FPCSchedulerServer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvRlBDU2NoZWR1bGVyU2VydmVyLnNjYWxh) | `0.00% <0.00%> (-95.84%)` | :arrow_down: |
| [...sk/core/scheduler/container/ContainerManager.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvY29udGFpbmVyL0NvbnRhaW5lck1hbmFnZXIuc2NhbGE=) | `0.00% <0.00%> (-88.72%)` | :arrow_down: |
| ... and [148 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...99c95cd](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (954d4f8) into [master](https://codecov.io/gh/apache/openwhisk/commit/f1829e1160d0aca51fc3ff93a77621c488b148ad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1829e1) will **decrease** coverage by `51.57%`.
> The diff coverage is `0.51%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
- Coverage 81.26% 29.68% -51.58%
===========================================
Files 218 228 +10
Lines 11158 11994 +836
Branches 497 529 +32
===========================================
- Hits 9068 3561 -5507
- Misses 2090 8433 +6343
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `63.59% <0.00%> (-22.05%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `90.90% <0.00%> (-3.35%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `41.96% <0.00%> (-26.95%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.59% <100.00%> (+0.02%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `66.00% <100.00%> (-23.80%)` | :arrow_down: |
| ... and [173 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1829e1...954d4f8](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (90464ba) into [master](https://codecov.io/gh/apache/openwhisk/commit/f1829e1160d0aca51fc3ff93a77621c488b148ad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1829e1) will **decrease** coverage by `50.21%`.
> The diff coverage is `0.51%`.
> :exclamation: Current head 90464ba differs from pull request most recent head bc6e18a. Consider uploading reports for the commit bc6e18a to get more accurate results
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
- Coverage 81.26% 31.04% -50.22%
===========================================
Files 218 224 +6
Lines 11158 11749 +591
Branches 497 517 +20
===========================================
- Hits 9067 3648 -5419
- Misses 2091 8101 +6010
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `64.81% <0.00%> (-20.83%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `90.69% <0.00%> (-3.56%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `41.96% <0.00%> (-26.95%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.56% <100.00%> (ø)` | |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `68.00% <100.00%> (-20.78%)` | :arrow_down: |
| ... and [174 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1829e1...bc6e18a](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r628061145
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/common/AverageRingBuffer.scala
##########
@@ -0,0 +1,34 @@
+package org.apache.openwhisk.common
+
+object AverageRingBuffer {
+ def apply(maxSize: Int) = new AverageRingBuffer(maxSize)
+}
+
+/**
+ * This buffer provides the average of the given elements.
+ * The number of elements are limited and the first element is removed if the maximum size is reached.
+ * Since it is based on the Vector, its operation takes effectively constant time.
+ * For more details, please visit https://docs.scala-lang.org/overviews/collections/performance-characteristics.html
+ *
+ * @param maxSize the maximum size of the buffer
+ */
+class AverageRingBuffer(private val maxSize: Int) {
Review comment:
This circular buffer is used to calculate the average execution time of recent N activations for a given action.
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/SchedulingDecisionMaker.scala
##########
@@ -0,0 +1,244 @@
+package org.apache.openwhisk.core.scheduler.queue
+
+import akka.actor.{Actor, ActorSystem, Props}
+import org.apache.openwhisk.common.Logging
+import org.apache.openwhisk.core.entity.FullyQualifiedEntityName
+
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Failure, Success}
+
+class SchedulingDecisionMaker(
Review comment:
This component decides whether to add more containers or not for the given action.
It uses the average execution time of an action.
There are three cases to calculate the average execution.
1. Initial execution.
In this case, we just assume the execution time as tick time(100ms by default).
2. Queue is newly created.
In this case, activations exist in the DB, but no data in the AverageRingBuffer.
So we fetch the average execution time from DB.
3. An action is continuously executed.
Each container sends the last execution time along with `FetchRequest` when it pulls a new activation from a queue.
And our scheduler keeps N execution times in the AverageRingBuffer.
So we can easily calculate the average execution time of recent N executions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 merged pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 merged pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (954d4f8) into [master](https://codecov.io/gh/apache/openwhisk/commit/f1829e1160d0aca51fc3ff93a77621c488b148ad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1829e1) will **decrease** coverage by `50.92%`.
> The diff coverage is `0.51%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
- Coverage 81.26% 30.34% -50.93%
===========================================
Files 218 228 +10
Lines 11158 11994 +836
Branches 497 529 +32
===========================================
- Hits 9068 3639 -5429
- Misses 2090 8355 +6265
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `64.51% <0.00%> (-21.13%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `90.90% <0.00%> (-3.35%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `41.96% <0.00%> (-26.95%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.59% <100.00%> (+0.02%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `68.00% <100.00%> (-21.80%)` | :arrow_down: |
| ... and [172 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1829e1...954d4f8](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-903069065
It’s ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r637798363
##########
File path: .travis.yml
##########
@@ -27,6 +27,7 @@ env:
global:
- ANSIBLE_CMD="ansible-playbook -i environments/local -e docker_image_prefix=testing"
- GRADLE_PROJS_SKIP=""
+ - OW_SCALA_VERSION=2.13
Review comment:
The scala compilation version was not the culprit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (90464ba) into [master](https://codecov.io/gh/apache/openwhisk/commit/f1829e1160d0aca51fc3ff93a77621c488b148ad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1829e1) will **decrease** coverage by `50.47%`.
> The diff coverage is `0.51%`.
> :exclamation: Current head 90464ba differs from pull request most recent head bc6e18a. Consider uploading reports for the commit bc6e18a to get more accurate results
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
- Coverage 80.85% 30.38% -50.48%
===========================================
Files 218 224 +6
Lines 11158 11749 +591
Branches 497 517 +20
===========================================
- Hits 9022 3570 -5452
- Misses 2136 8179 +6043
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `63.88% <0.00%> (-21.76%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `90.69% <0.00%> (-3.56%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `41.96% <0.00%> (-24.36%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `95.56% <100.00%> (ø)` | |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `66.00% <100.00%> (-20.74%)` | :arrow_down: |
| ... and [179 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1829e1...bc6e18a](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-878037782
> One comment on the persistence nature of the queue. I think queueing with kafka doesn't add anything to openwhisk if openwhisk does not provide at least once guarantee of activation execution. It's unnecessary overhead to queue up through kafka if the activation can get lost elsewhere anyways.
Agree.
I think the immediate next step could be removing Kafka from the critical path completely as currently, container creation messages and activation messages from controller to scheduler are delivered through Kafka.
> This queue is designed in a way that activations are not persistent.
But it could be improved in the future by taking a similar approach with what Kafka does.
I mentioned above that if we really need to support the at-least-once semantic, we can consider this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r657485433
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/common/AverageRingBuffer.scala
##########
@@ -0,0 +1,34 @@
+package org.apache.openwhisk.common
+
+object AverageRingBuffer {
+ def apply(maxSize: Int) = new AverageRingBuffer(maxSize)
+}
+
+/**
+ * This buffer provides the average of the given elements.
+ * The number of elements are limited and the first element is removed if the maximum size is reached.
+ * Since it is based on the Vector, its operation takes effectively constant time.
+ * For more details, please visit https://docs.scala-lang.org/overviews/collections/performance-characteristics.html
+ *
+ * @param maxSize the maximum size of the buffer
+ */
+class AverageRingBuffer(private val maxSize: Int) {
Review comment:
What was the reasoning for picking average as the heuristic. Would median be a better heuristic here? i.e. if all activations take 100 milliseconds and then one activation has a slow call to a db that will heavily skew the average.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834204005
According to the order defined in the wiki, this component requires other components to be merged.
https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design
But there is no big difference in the memory queue logic, so I opened it for reviews.
Since this logic is the core logic and changes on this component are highly likely to introduce huge impacts on the system, I will write down a comprehensive design document.
Once dependent modules are merged into the master branch, I would rebase this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] ningyougang commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
ningyougang commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r629001121
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/common/AverageRingBuffer.scala
##########
@@ -0,0 +1,34 @@
+package org.apache.openwhisk.common
Review comment:
Need to add licence header
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (02740df) into [master](https://codecov.io/gh/apache/openwhisk/commit/aa7e6e2af196ac017ae4b9ea36656bec868a9931?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa7e6e2) will **decrease** coverage by `13.26%`.
> The diff coverage is `0.64%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
- Coverage 78.82% 65.56% -13.27%
===========================================
Files 213 216 +3
Lines 10466 11086 +620
Branches 440 480 +40
===========================================
- Hits 8250 7268 -982
- Misses 2216 3818 +1602
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `69.62% <0.00%> (-16.31%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `93.02% <0.00%> (-1.10%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `47.34% <0.00%> (-22.95%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `93.67% <100.00%> (-1.88%)` | :arrow_down: |
| [.../scala/org/apache/openwhisk/core/entity/Size.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L1NpemUuc2NhbGE=) | `79.10% <100.00%> (ø)` | |
| ... and [51 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [aa7e6e2...02740df](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] ningyougang commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
ningyougang commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-836055038
It is better to add below configuration to scheduler's application.conf, e.g.
```
queue {
idle-grace = "20 seconds"
stop-grace = "20 seconds"
flush-grace = "60 seconds"
graceful-shutdown-timeout = "5 seconds"
max-retention-size = "10000"
max-retention-ms = "60000"
throttling-fraction = "0.9"
duration-buffer-size = "10"
}
```
Or when open another new pr which include ansible/roles/schedulers/tasks/deploy.yml, add there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (954d4f8) into [master](https://codecov.io/gh/apache/openwhisk/commit/f1829e1160d0aca51fc3ff93a77621c488b148ad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1829e1) will **decrease** coverage by `76.09%`.
> The diff coverage is `0.51%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
==========================================
- Coverage 81.26% 5.17% -76.10%
==========================================
Files 218 228 +10
Lines 11158 11994 +836
Branches 497 529 +32
==========================================
- Hits 9068 621 -8447
- Misses 2090 11373 +9283
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `41.01% <0.00%> (-44.63%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `34.09% <0.00%> (-60.17%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `15.54% <0.00%> (-53.37%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `65.40% <100.00%> (-30.17%)` | :arrow_down: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `45.00% <100.00%> (-44.80%)` | :arrow_down: |
| ... and [209 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1829e1...954d4f8](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-906822564
I will merge this in 48 hours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] ningyougang commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
ningyougang commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-855504485
Need rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] ningyougang commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
ningyougang commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r630735283
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/MemoryQueue.scala
##########
@@ -0,0 +1,1096 @@
+package org.apache.openwhisk.core.scheduler.queue
+
+import java.time.{Duration, Instant}
+
+import akka.actor.Status.{Failure => FailureMessage}
+import akka.actor.{ActorRef, ActorSystem, Cancellable, FSM, Props, Stash}
+import akka.util.Timeout
+import org.apache.openwhisk.common._
+import org.apache.openwhisk.core.ack.ActiveAck
+import org.apache.openwhisk.core.connector.ContainerCreationError.{TooManyConcurrentRequests, ZeroNamespaceLimit}
+import org.apache.openwhisk.core.connector._
+import org.apache.openwhisk.core.database.{NoDocumentException, UserContext}
+import org.apache.openwhisk.core.entity._
+import org.apache.openwhisk.core.entity.size._
+import org.apache.openwhisk.core.etcd.EtcdClient
+import org.apache.openwhisk.core.etcd.EtcdKV.ContainerKeys.containerPrefix
+import org.apache.openwhisk.core.etcd.EtcdKV.{ContainerKeys, QueueKeys, ThrottlingKeys}
+import org.apache.openwhisk.core.scheduler.SchedulerEndpoints
+import org.apache.openwhisk.core.scheduler.container.{
+ ContainerCreation,
+ ContainerDeletion,
+ FailedCreationJob,
+ SuccessfulCreationJob
+}
+import org.apache.openwhisk.core.scheduler.grpc.{GetActivation, ActivationResponse => GetActivationResponse}
+import org.apache.openwhisk.core.service._
+import org.apache.openwhisk.core.{ConfigKeys, WhiskConfig}
+import org.apache.openwhisk.http.Messages.{namespaceLimitUnderZero, tooManyConcurrentRequests}
+import pureconfig.generic.auto._
+import pureconfig.loadConfigOrThrow
+import spray.json._
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+import scala.collection.mutable
+import scala.concurrent.duration._
+import scala.concurrent.{duration, ExecutionContextExecutor, Future, Promise}
+import scala.language.postfixOps
+import scala.util.{Failure, Success}
+
+// States
+sealed trait MemoryQueueState
+case object Uninitialized extends MemoryQueueState
+case object Running extends MemoryQueueState
+case object Idle extends MemoryQueueState
+case object Flushing extends MemoryQueueState
+case object Removing extends MemoryQueueState
+case object Removed extends MemoryQueueState
+case object ActionThrottled extends MemoryQueueState
+case object NamespaceThrottled extends MemoryQueueState
+
+// Data
+sealed abstract class MemoryQueueData()
+case class NoData() extends MemoryQueueData()
+case class NoActors() extends MemoryQueueData()
+case class RunningData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class ThrottledData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class FlushingData(schedulerActor: ActorRef,
+ droppingActor: ActorRef,
+ error: ContainerCreationError,
+ reason: String,
+ activeDuringFlush: Boolean = false)
+ extends MemoryQueueData()
+case class RemovingData(schedulerActor: ActorRef, droppingActor: ActorRef, outdated: Boolean) extends MemoryQueueData()
+
+// Events sent by the actor
+case class QueueRemoved(invocationNamespace: String, action: DocInfo, leaderKey: Option[String])
+case class QueueReactivated(invocationNamespace: String, action: FullyQualifiedEntityName, docInfo: DocInfo)
+case class CancelPoll(promise: Promise[Either[MemoryQueueError, ActivationMessage]])
+case object QueueRemovedCompleted
+case object FlushPulse
+
+// Events received by the actor
+case object Start
+case object VersionUpdated
+case object StopSchedulingAsOutdated
+
+sealed trait RequiredAction
+case object Skip extends RequiredAction
+case object AddInitialContainer extends RequiredAction
+case object AddContainer extends RequiredAction
+case class EnableNamespaceThrottling(dropMsg: Boolean) extends RequiredAction
+case object DisableNamespaceThrottling extends RequiredAction
+case object EnableActionThrottling extends RequiredAction
+case object DisableActionThrottling extends RequiredAction
+case object Pausing extends RequiredAction
+case class DecisionResults(required: RequiredAction, num: Int)
+
+case class TimeSeriesActivationEntry(timestamp: Instant, msg: ActivationMessage)
+
+class MemoryQueue(private val etcdClient: EtcdClient,
+ private val durationChecker: DurationChecker,
+ private val action: FullyQualifiedEntityName,
+ messagingProducer: MessageProducer,
+ config: WhiskConfig,
+ invocationNamespace: String,
+ revision: DocRevision,
+ endpoints: SchedulerEndpoints,
+ actionMetaData: WhiskActionMetaData,
+ dataManagementService: ActorRef,
+ watcherService: ActorRef,
+ containerManager: ActorRef,
+ decisionMaker: ActorRef,
+ schedulerId: SchedulerInstanceId,
+ ack: ActiveAck,
+ store: (TransactionId, WhiskActivation, UserContext) => Future[Any],
+ getUserLimit: String => Future[Int],
+ checkToDropStaleActivation: (Queue[TimeSeriesActivationEntry],
+ Long,
+ String,
+ FullyQualifiedEntityName,
+ MemoryQueueState,
+ ActorRef) => Unit,
+ queueConfig: QueueConfig)(implicit logging: Logging)
+ extends FSM[MemoryQueueState, MemoryQueueData]
+ with Stash {
+
+ private implicit val ec: ExecutionContextExecutor = context.dispatcher
+ private implicit val actorSystem: ActorSystem = context.system
+ private implicit val timeout = Timeout(5.seconds)
+ private implicit val order: Ordering[BufferedRequest] = Ordering.by(_.containerId)
+
+ private val unversionedAction = action.copy(version = None)
+ private val checkInterval: FiniteDuration = 100 milliseconds
+ private val StaleThreshold: Double = 100.0
+ private val StaleDuration = Duration.ofMillis(StaleThreshold.toLong)
+ private val dropInterval: FiniteDuration = 10 seconds
+ private val leaderKey = QueueKeys.queue(invocationNamespace, unversionedAction, leader = true)
+ private val inProgressContainerPrefixKey =
+ containerPrefix(ContainerKeys.inProgressPrefix, invocationNamespace, action, Some(revision))
+ private val existingContainerPrefixKey =
+ containerPrefix(ContainerKeys.namespacePrefix, invocationNamespace, action, Some(revision))
+ private val namespaceThrottlingKey = ThrottlingKeys.namespace(EntityName(invocationNamespace))
+ private val actionThrottlingKey = ThrottlingKeys.action(invocationNamespace, unversionedAction)
+ private val pollTimeOut = 1.seconds
+ private var requestBuffer = mutable.PriorityQueue.empty[BufferedRequest]
+ private val memory = actionMetaData.limits.memory.megabytes.MB
+ private val queueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), Some(leaderKey))
+ private val staleQueueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), None)
+
+ private[queue] var containers = Set.empty[String]
+ private[queue] var creationIds = Set.empty[String]
+
+ private[queue] var queue = Queue.empty[TimeSeriesActivationEntry]
+ private[queue] var in: Int = 0
+ private[queue] val namespaceContainerCount = NamespaceContainerCount(invocationNamespace, etcdClient, watcherService)
+ private[queue] var averageDuration: Option[Double] = None
+ private[queue] var averageDurationBuffer = AverageRingBuffer(queueConfig.durationBufferSize)
+ private[queue] var limit: Option[Int] = None
+ private[queue] var initialized = false
+
+ private val logScheduler: Cancellable = context.system.scheduler.schedule(0.seconds, 1.seconds) {
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_QUEUE_WAITING_ACTIVATION(s"$invocationNamespace/$action"),
+ queue.size)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_CONTAINER(invocationNamespace),
+ namespaceContainerCount.existingContainerNumByNamespace)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_INPROGRESS_CONTAINER(invocationNamespace),
+ namespaceContainerCount.inProgressContainerNumByNamespace)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_CONTAINER(invocationNamespace, action.asString),
+ containers.size)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_INPROGRESS_CONTAINER(invocationNamespace, action.asString),
+ creationIds.size)
+ }
+
+ getAverageDuration()
+
+ private val watcherName = s"memory-queue-$action-$revision"
+ // watch existing containers for action and namespace
+ private val watchedKeys = Seq(inProgressContainerPrefixKey, existingContainerPrefixKey)
+
+ watchedKeys.foreach { key =>
+ watcherService ! WatchEndpoint(key, "", isPrefix = true, watcherName, Set(PutEvent, DeleteEvent))
+ }
+
+ startWith(Uninitialized, NoData())
+
+ when(Uninitialized) {
+ case Event(Start, _) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] a new queue is created.")
+ val (schedulerActor, droppingActor) = startMonitoring()
+ initializeThrottling()
+
+ watcherService ! WatchEndpoint(leaderKey, endpoints.serialize, isPrefix = false, watcherName, Set(DeleteEvent))
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // this is the case that the action version is updated, so no data needs to be stored
+ case Event(VersionUpdated, _) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // other messages should not be handled in this state.
+ case _ =>
+ stash()
+ stay
+ }
+
+ when(Running, stateTimeout = queueConfig.idleGrace) {
+ case Event(EnableNamespaceThrottling(dropMsg), data: RunningData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Enable namespace throttling.")
+ enableNamespaceThrottling()
+
+ // if no container could be created, it is same with Flushing state.
+ if (dropMsg) {
+ completeAllActivations(tooManyConcurrentRequests, isWhiskError = false)
+ goto(Flushing) using FlushingData(
+ data.schedulerActor,
+ data.droppingActor,
+ TooManyConcurrentRequests,
+ tooManyConcurrentRequests)
+ } else {
+ // if there are already some containers running, activations can still be processed so goto the NamespaceThrottled state.
+ goto(NamespaceThrottled) using ThrottledData(data.schedulerActor, data.droppingActor)
+ }
+
+ case Event(StateTimeout, data: RunningData) =>
+ if (queue.isEmpty && (containers.size + creationIds.size) <= 0) {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] No activations coming in ${queueConfig.idleGrace}")
+ actorSystem.stop(data.schedulerActor)
+ actorSystem.stop(data.droppingActor)
+
+ goto(Idle) using NoActors()
+ } else {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue is timed out but there are still ${queue.size} activation messages or (running: ${containers.size}, in-progress: ${creationIds.size}) containers")
+ stay
+ }
+
+ case Event(FailedCreationJob(creationId, _, _, _, error, message), RunningData(schedulerActor, droppingActor)) =>
+ creationIds -= creationId.asString
+ // when there is no container, it moves to the Flushing state as no activations can be invoked
+ if (containers.size <= 0) {
+ val isWhiskError = ContainerCreationError.whiskErrors.contains(error)
+ completeAllActivations(message, isWhiskError)
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Failed to create an initial container due to ${if (isWhiskError) "whiskError"
+ else "developerError"}, reason: $message.")
+
+ goto(Flushing) using FlushingData(schedulerActor, droppingActor, error, message)
+ } else
+ // if there are already some containers running, activations can be handled anyway.
+ stay
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(NamespaceThrottled) {
+ case Event(msg: ActivationMessage, _: ThrottledData) =>
+ handleActivationMessage(msg)
+ stay
+
+ case Event(DisableNamespaceThrottling, data: ThrottledData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Disable namespace throttling.")
+ disableNamespaceThrottling()
+ goto(Running) using RunningData(data.schedulerActor, data.schedulerActor)
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(ActionThrottled) {
+ // since there are already too many activation messages, it drops the new messages
+ case Event(msg: ActivationMessage, ThrottledData(_, _)) =>
+ completeErrorActivation(msg, tooManyConcurrentRequests, isWhiskError = false)
+ stay
+ }
+
+ when(Idle, stateTimeout = queueConfig.stopGrace) {
+ case Event(msg: ActivationMessage, _: NoActors) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+ handleActivationMessage(msg)
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ case Event(request: GetActivation, _) if request.action == action =>
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ stay
+
+ case Event(StateTimeout, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] The queue is timed out, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(GracefulShutdown, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(StopSchedulingAsOutdated, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+
+ cleanUpWatcher()
+
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ // since the queue is outdated and there is no activation, delete all old containers.
+ containerManager ! ContainerDeletion(invocationNamespace, action, revision, actionMetaData)
+
+ goto(Removed) using NoData()
+ }
+
+ when(Flushing) {
+ // an initial container is successfully created.
+ case Event(SuccessfulCreationJob(creationId, _, _, _), FlushingData(schedulerActor, droppingActor, _, _, _)) =>
+ creationIds -= creationId.asString
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // log the failed information
+ case Event(FailedCreationJob(creationId, _, _, _, _, message), data: FlushingData) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Failed to create a container due to $message")
+
+ // keep updating the reason
+ stay using data.copy(reason = message)
+
+ // since there is no container, activations cannot be handled.
+ case Event(msg: ActivationMessage, data: FlushingData) =>
+ completeErrorActivation(msg, data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ stay() using data.copy(activeDuringFlush = true)
+
+ // Since SchedulingDecisionMaker keep sending a message to create a container, this state is not automatically timed out.
+ // Instead, StateTimeout message will be sent by a timer.
+ case Event(StateTimeout, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ if (data.activeDuringFlush)
+ stay using data.copy(activeDuringFlush = false)
+ else
+ cleanUpActorsAndGotoRemoved(data)
+
+ case Event(GracefulShutdown, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpActorsAndGotoRemoved(data)
+ }
+
+ // in case there is any activation in the queue, it waits until all of them are handled.
+ when(Removing, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // When there is no message in the queue, SchedulingDecisionMaker would stop sending any message
+ // So the queue can be timed out on every gracefulShutdownTimeout
+ case Event(QueueRemovedCompleted | StateTimeout, data: RemovingData) =>
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(GracefulShutdown, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received GracefulShutdown trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(StopSchedulingAsOutdated, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received StopSchedulingAsOutdated trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data.copy(outdated = true))
+ }
+
+ when(Removed, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // since this Queue will be terminated, rescheduling the msg
+ case Event(msg: ActivationMessage, _: NoData) =>
+ context.parent ! msg
+ stay()
+
+ // this queue is going to stop so let client connect to a new queue
+ case Event(request: GetActivation, _: NoData) if request.action == action =>
+ implicit val tid = request.transactionId
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Get activation request ${request.containerId}, let client connect to a new queue.")
+ forwardAllActivations(context.parent)
+ sender ! GetActivationResponse(Left(NoMemoryQueue()))
+
+ stay
+
+ // actors and data are already wiped
+ case Event(QueueRemovedCompleted, _: NoData) =>
+ stop()
+
+ // This is not supposed to happen. This will ensure the queue does not run forever.
+ // This can happen when QueueManager could not respond with QueueRemovedCompleted for some reason.
+ case Event(StateTimeout, _: NoData) =>
+ context.parent ! queueRemovedMsg
+
+ stop()
+
+ // This queue is going to stop, do nothing
+ case Event(msg @ (StopSchedulingAsOutdated | GracefulShutdown), _: NoData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received $msg but do nothing as it is going to stop.")
+ stay
+ }
+
+ whenUnhandled {
+ // The queue endpoint is removed, trying to restore it.
+ case Event(WatchEndpointRemoved(_, `leaderKey`, value, false), data) =>
+ data match {
+ case RemovingData(_, _, _) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] This queue is shutdown by `/disable` api, do nothing here.")
+ case _ =>
+ dataManagementService ! RegisterInitialData(leaderKey, value, failoverEnabled = false, Some(self)) // the watcher is already setup
+ }
+ stay
+
+ // we don't care the storage results for namespaceThrottlingKey
+ case Event(InitialDataStorageResults(`namespaceThrottlingKey`, _), _) =>
+ stay
+
+ // The queue endpoint is restored
+ case Event(InitialDataStorageResults(`leaderKey`, Right(_)), _) =>
+ stay
+
+ // this can be a case that there is another queue already running.
+ // it can happen if a node is segregated by the temporal network rupture and the queue endpoint is removed.
+ case Event(InitialDataStorageResults(`leaderKey`, Left(_)), data) =>
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] the queue is superseded by a new queue.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! queueRemovedMsg
+
+ // forward all activations to the parent queue manager.
+ // parent queue manager is supposed to removed the reference of this queue and forward messages to a new queue
+ forwardAllActivations(context.parent)
+
+ // only clean up actors because etcd data is already being used by another queue
+ cleanUpActors(data)
+
+ goto(Removed) using NoData()
+
+ case Event(WatchEndpointRemoved(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds -= key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers -= key.split("/").last
+ case _ =>
+ }
+ stay
+
+ case Event(WatchEndpointInserted(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds += key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers += key.split("/").last
+ case _ =>
+ }
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled
+ case Event(SuccessfulCreationJob(creationId, _, _, _), _) =>
+ creationIds -= creationId.asString
+ stay()
+
+ // for other cases
+ case Event(FailedCreationJob(creationId, invocationNamespace, action, revision, _, message), _) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Got failed creation job with revision $revision and error $message.")
+ stay()
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(cancel: CancelPoll, _) =>
+ cancel.promise.trySuccess(Left(NoActivationMessage()))
+
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(msg: ActivationMessage, _) =>
+ handleActivationMessage(msg)
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action == action =>
+ implicit val tid = request.transactionId
+ if (request.alive) {
+ containers += request.containerId
+ handleActivationRequest(request)
+ } else {
+ logging.info(this, s"Remove containerId because ${request.containerId} is not alive")
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ containers -= request.containerId
+ stay
+ }
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action != action =>
+ implicit val tid = request.transactionId
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] version mismatch ${request.action}")
+ sender ! GetActivationResponse(Left(ActionMismatch()))
+
+ stay
+
+ case Event(DropOld, _) =>
+ if (queue.nonEmpty && Duration
+ .between(queue.head.timestamp, Instant.now)
+ .compareTo(Duration.ofMillis(queueConfig.maxRetentionMs)) < 0) {
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Drop some stale activations for $revision, existing container is ${containers.size}, inProgress container is ${creationIds.size}, state data: $stateData, in is $in, current: ${queue.size}.")
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] the head stale message: ${queue.head.msg.activationId}")
+ }
+ queue = MemoryQueue.dropOld(queue, Duration.ofMillis(queueConfig.maxRetentionMs), completeErrorActivation)
+
+ stay
+
+ // common case for all statuses
+ case Event(StatusQuery, _) =>
+ sender ! StatusData(invocationNamespace, action.asString, queue.size, stateName.toString, stateData.toString)
+ stay
+
+ // Common case for all cases
+ case Event(GracefulShutdown, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Gracefully shutdown the memory queue.")
+ // delete relative data, e.g leaderKey, namespaceThrottlingKey, actionThrottlingKey
+ cleanUpData()
+
+ // let queue manager knows this queue is going to stop and let it forward incoming activations to a new queue
+ context.parent ! queueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = false)
+
+ // the version is updated. it's a shared case for all states
+ case Event(StopSchedulingAsOutdated, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = true)
+
+ case Event(t: FailureMessage, _) =>
+ logging.error(this, s"[$invocationNamespace:$action:$stateName] got an unexpected failure message: $t")
+
+ stay
+
+ case Event(msg: DecisionResults, _) =>
+ val DecisionResults(result, num) = msg
+ result match {
+ case AddInitialContainer if num > 0 =>
+ initialized = true
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+
+ case AddContainer if num > 0 =>
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+
+ case enable: EnableNamespaceThrottling =>
+ if (num > 0) {
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+ }
+ self ! enable
+
+ case DisableNamespaceThrottling =>
+ if (num > 0) {
+ val msgs = generateContainerCreationMessages(num)
+ containerManager ! ContainerCreation(msgs, memory, invocationNamespace)
+ }
+ self ! DisableNamespaceThrottling
+
+ case Pausing =>
+ logging.warn(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The limit value is less than 0. No activation can be handled so the queue becomes the Flushing state.")
+ self ! FailedCreationJob(
+ CreationId.void,
+ invocationNamespace,
+ action,
+ revision,
+ ZeroNamespaceLimit,
+ namespaceLimitUnderZero)
+ }
+ stay
+
+ // this should not happen
+ case otherMsg =>
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] received unexpected message: $otherMsg")
+
+ stay
+ }
+
+ onTransition {
+ case Uninitialized -> _ => unstashAll()
+ case _ -> Flushing => setTimer("StopQueue", StateTimeout, queueConfig.flushGrace, repeat = true)
+ case Flushing -> _ => cancelTimer("StopQueue")
+ }
+
+ onTermination {
+ case _ =>
+ // logscheduler must be canceled when FSM is terminated
+ logScheduler.cancel()
+
+ // the lifecycle of DecisionMaker conforms to the one of MemoryQueue
+ actorSystem.stop(decisionMaker)
+ }
+
+ initialize()
+
+ private def cleanUpDataAndGotoRemoved() = {
+ cleanUpWatcher()
+ cleanUpData()
+ context.parent ! queueRemovedMsg
+
+ goto(Removed) using NoData()
+ }
+
+ private def cleanUpActorsAndGotoRemoved(data: FlushingData) = {
+ cleanUpActors(data)
+ cleanUpData()
+
+ context.parent ! queueRemovedMsg
+
+ goto(Removed) using NoData()
+ }
+
+ private def cleanUpActorsAndGotoRemovedIfPossible(data: RemovingData) = {
+ if (queue.isEmpty && requestBuffer.isEmpty) {
Review comment:
should add `requestBuffer = requestBuffer.filter(!_.promise.isCompleted)` before this line: `if (queue.isEmpty && requestBuffer.isEmpty) {`
This is avoid `when updates queue, in spite of all requests in requestBuffer will be finished finally, but the completed request may not be removed from requestBuffer`,so need to add `requestBuffer.filter(!_.promise.isCompleted)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r667633847
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/MemoryQueue.scala
##########
@@ -0,0 +1,1097 @@
+package org.apache.openwhisk.core.scheduler.queue
+
+import java.time.{Duration, Instant}
+
+import akka.actor.Status.{Failure => FailureMessage}
+import akka.actor.{ActorRef, ActorSystem, Cancellable, FSM, Props, Stash}
+import akka.util.Timeout
+import org.apache.openwhisk.common._
+import org.apache.openwhisk.core.ack.ActiveAck
+import org.apache.openwhisk.core.connector.ContainerCreationError.{TooManyConcurrentRequests, ZeroNamespaceLimit}
+import org.apache.openwhisk.core.connector._
+import org.apache.openwhisk.core.database.{NoDocumentException, UserContext}
+import org.apache.openwhisk.core.entity._
+import org.apache.openwhisk.core.entity.size._
+import org.apache.openwhisk.core.etcd.EtcdClient
+import org.apache.openwhisk.core.etcd.EtcdKV.ContainerKeys.containerPrefix
+import org.apache.openwhisk.core.etcd.EtcdKV.{ContainerKeys, QueueKeys, ThrottlingKeys}
+import org.apache.openwhisk.core.scheduler.SchedulerEndpoints
+import org.apache.openwhisk.core.scheduler.grpc.{GetActivation, ActivationResponse => GetActivationResponse}
+import org.apache.openwhisk.core.scheduler.message.{
+ ContainerCreation,
+ ContainerDeletion,
+ FailedCreationJob,
+ SuccessfulCreationJob
+}
+import org.apache.openwhisk.core.service._
+import org.apache.openwhisk.core.{ConfigKeys, WhiskConfig}
+import org.apache.openwhisk.http.Messages.{namespaceLimitUnderZero, tooManyConcurrentRequests}
+import pureconfig.generic.auto._
+import pureconfig.loadConfigOrThrow
+import spray.json._
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+import scala.collection.mutable
+import scala.concurrent.duration._
+import scala.concurrent.{duration, ExecutionContextExecutor, Future, Promise}
+import scala.language.postfixOps
+import scala.util.{Failure, Success}
+
+// States
+sealed trait MemoryQueueState
+case object Uninitialized extends MemoryQueueState
+case object Running extends MemoryQueueState
+case object Idle extends MemoryQueueState
+case object Flushing extends MemoryQueueState
+case object Removing extends MemoryQueueState
+case object Removed extends MemoryQueueState
+case object ActionThrottled extends MemoryQueueState
+case object NamespaceThrottled extends MemoryQueueState
+
+// Data
+sealed abstract class MemoryQueueData()
+case class NoData() extends MemoryQueueData()
+case class NoActors() extends MemoryQueueData()
+case class RunningData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class ThrottledData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class FlushingData(schedulerActor: ActorRef,
+ droppingActor: ActorRef,
+ error: ContainerCreationError,
+ reason: String,
+ activeDuringFlush: Boolean = false)
+ extends MemoryQueueData()
+case class RemovingData(schedulerActor: ActorRef, droppingActor: ActorRef, outdated: Boolean) extends MemoryQueueData()
+
+// Events sent by the actor
+case class QueueRemoved(invocationNamespace: String, action: DocInfo, leaderKey: Option[String])
+case class QueueReactivated(invocationNamespace: String, action: FullyQualifiedEntityName, docInfo: DocInfo)
+case class CancelPoll(promise: Promise[Either[MemoryQueueError, ActivationMessage]])
+case object QueueRemovedCompleted
+case object FlushPulse
+
+// Events received by the actor
+case object Start
+case object VersionUpdated
+case object StopSchedulingAsOutdated
+
+sealed trait RequiredAction
+case object Skip extends RequiredAction
+case object AddInitialContainer extends RequiredAction
+case object AddContainer extends RequiredAction
+case class EnableNamespaceThrottling(dropMsg: Boolean) extends RequiredAction
+case object DisableNamespaceThrottling extends RequiredAction
+case object EnableActionThrottling extends RequiredAction
+case object DisableActionThrottling extends RequiredAction
+case object Pausing extends RequiredAction
+case class DecisionResults(required: RequiredAction, num: Int)
+
+case class TimeSeriesActivationEntry(timestamp: Instant, msg: ActivationMessage)
+
+class MemoryQueue(private val etcdClient: EtcdClient,
+ private val durationChecker: DurationChecker,
+ private val action: FullyQualifiedEntityName,
+ messagingProducer: MessageProducer,
+ config: WhiskConfig,
+ invocationNamespace: String,
+ revision: DocRevision,
+ endpoints: SchedulerEndpoints,
+ actionMetaData: WhiskActionMetaData,
+ dataManagementService: ActorRef,
+ watcherService: ActorRef,
+ containerManager: ActorRef,
+ decisionMaker: ActorRef,
+ schedulerId: SchedulerInstanceId,
+ ack: ActiveAck,
+ store: (TransactionId, WhiskActivation, UserContext) => Future[Any],
+ getUserLimit: String => Future[Int],
+ checkToDropStaleActivation: (Queue[TimeSeriesActivationEntry],
+ Long,
+ String,
+ FullyQualifiedEntityName,
+ MemoryQueueState,
+ ActorRef) => Unit,
+ queueConfig: QueueConfig)(implicit logging: Logging)
+ extends FSM[MemoryQueueState, MemoryQueueData]
+ with Stash {
+
+ private implicit val ec: ExecutionContextExecutor = context.dispatcher
+ private implicit val actorSystem: ActorSystem = context.system
+ private implicit val timeout = Timeout(5.seconds)
+ private implicit val order: Ordering[BufferedRequest] = Ordering.by(_.containerId)
+
+ private val unversionedAction = action.copy(version = None)
+ private val checkInterval: FiniteDuration = 100 milliseconds
+ private val StaleThreshold: Double = 100.0
+ private val StaleDuration = Duration.ofMillis(StaleThreshold.toLong)
+ private val dropInterval: FiniteDuration = 10 seconds
+ private val leaderKey = QueueKeys.queue(invocationNamespace, unversionedAction, leader = true)
+ private val inProgressContainerPrefixKey =
+ containerPrefix(ContainerKeys.inProgressPrefix, invocationNamespace, action, Some(revision))
+ private val existingContainerPrefixKey =
+ containerPrefix(ContainerKeys.namespacePrefix, invocationNamespace, action, Some(revision))
+ private val namespaceThrottlingKey = ThrottlingKeys.namespace(EntityName(invocationNamespace))
+ private val actionThrottlingKey = ThrottlingKeys.action(invocationNamespace, unversionedAction)
+ private val pollTimeOut = 1.seconds
+ private var requestBuffer = mutable.PriorityQueue.empty[BufferedRequest]
+ private val memory = actionMetaData.limits.memory.megabytes.MB
+ private val queueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), Some(leaderKey))
+ private val staleQueueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), None)
+
+ private[queue] var containers = Set.empty[String]
+ private[queue] var creationIds = Set.empty[String]
+
+ private[queue] var queue = Queue.empty[TimeSeriesActivationEntry]
+ private[queue] var in: Int = 0
+ private[queue] val namespaceContainerCount = NamespaceContainerCount(invocationNamespace, etcdClient, watcherService)
+ private[queue] var averageDuration: Option[Double] = None
+ private[queue] var averageDurationBuffer = AverageRingBuffer(queueConfig.durationBufferSize)
+ private[queue] var limit: Option[Int] = None
+ private[queue] var initialized = false
+
+ private val logScheduler: Cancellable = context.system.scheduler.scheduleWithFixedDelay(0.seconds, 1.seconds) { () =>
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_QUEUE_WAITING_ACTIVATION(s"$invocationNamespace/$action"),
+ queue.size)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_CONTAINER(invocationNamespace),
+ namespaceContainerCount.existingContainerNumByNamespace)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_INPROGRESS_CONTAINER(invocationNamespace),
+ namespaceContainerCount.inProgressContainerNumByNamespace)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_CONTAINER(invocationNamespace, action.asString),
+ containers.size)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_INPROGRESS_CONTAINER(invocationNamespace, action.asString),
+ creationIds.size)
+ }
+
+ getAverageDuration()
+
+ private val watcherName = s"memory-queue-$action-$revision"
+ // watch existing containers for action and namespace
+ private val watchedKeys = Seq(inProgressContainerPrefixKey, existingContainerPrefixKey)
+
+ watchedKeys.foreach { key =>
+ watcherService ! WatchEndpoint(key, "", isPrefix = true, watcherName, Set(PutEvent, DeleteEvent))
+ }
+
+ startWith(Uninitialized, NoData())
+
+ when(Uninitialized) {
+ case Event(Start, _) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] a new queue is created.")
+ val (schedulerActor, droppingActor) = startMonitoring()
+ initializeThrottling()
+
+ watcherService ! WatchEndpoint(leaderKey, endpoints.serialize, isPrefix = false, watcherName, Set(DeleteEvent))
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // this is the case that the action version is updated, so no data needs to be stored
+ case Event(VersionUpdated, _) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // other messages should not be handled in this state.
+ case _ =>
+ stash()
+ stay
+ }
+
+ when(Running, stateTimeout = queueConfig.idleGrace) {
+ case Event(EnableNamespaceThrottling(dropMsg), data: RunningData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Enable namespace throttling.")
+ enableNamespaceThrottling()
+
+ // if no container could be created, it is same with Flushing state.
+ if (dropMsg) {
+ completeAllActivations(tooManyConcurrentRequests, isWhiskError = false)
+ goto(Flushing) using FlushingData(
+ data.schedulerActor,
+ data.droppingActor,
+ TooManyConcurrentRequests,
+ tooManyConcurrentRequests)
+ } else {
+ // if there are already some containers running, activations can still be processed so goto the NamespaceThrottled state.
+ goto(NamespaceThrottled) using ThrottledData(data.schedulerActor, data.droppingActor)
+ }
+
+ case Event(StateTimeout, data: RunningData) =>
+ if (queue.isEmpty && (containers.size + creationIds.size) <= 0) {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] No activations coming in ${queueConfig.idleGrace}")
+ actorSystem.stop(data.schedulerActor)
+ actorSystem.stop(data.droppingActor)
+
+ goto(Idle) using NoActors()
+ } else {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue is timed out but there are still ${queue.size} activation messages or (running: ${containers.size}, in-progress: ${creationIds.size}) containers")
+ stay
+ }
+
+ case Event(FailedCreationJob(creationId, _, _, _, error, message), RunningData(schedulerActor, droppingActor)) =>
+ creationIds -= creationId.asString
+ // when there is no container, it moves to the Flushing state as no activations can be invoked
+ if (containers.size <= 0) {
+ val isWhiskError = ContainerCreationError.whiskErrors.contains(error)
+ completeAllActivations(message, isWhiskError)
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Failed to create an initial container due to ${if (isWhiskError) "whiskError"
+ else "developerError"}, reason: $message.")
+
+ goto(Flushing) using FlushingData(schedulerActor, droppingActor, error, message)
+ } else
+ // if there are already some containers running, activations can be handled anyway.
+ stay
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(NamespaceThrottled) {
+ case Event(msg: ActivationMessage, _: ThrottledData) =>
+ handleActivationMessage(msg)
+ stay
+
+ case Event(DisableNamespaceThrottling, data: ThrottledData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Disable namespace throttling.")
+ disableNamespaceThrottling()
+ goto(Running) using RunningData(data.schedulerActor, data.schedulerActor)
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(ActionThrottled) {
+ // since there are already too many activation messages, it drops the new messages
+ case Event(msg: ActivationMessage, ThrottledData(_, _)) =>
+ completeErrorActivation(msg, tooManyConcurrentRequests, isWhiskError = false)
+ stay
+ }
+
+ when(Idle, stateTimeout = queueConfig.stopGrace) {
+ case Event(msg: ActivationMessage, _: NoActors) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+ handleActivationMessage(msg)
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ case Event(request: GetActivation, _) if request.action == action =>
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ stay
+
+ case Event(StateTimeout, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] The queue is timed out, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(GracefulShutdown, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(StopSchedulingAsOutdated, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+
+ cleanUpWatcher()
+
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ // since the queue is outdated and there is no activation, delete all old containers.
+ containerManager ! ContainerDeletion(invocationNamespace, action, revision, actionMetaData)
+
+ goto(Removed) using NoData()
+ }
+
+ when(Flushing) {
+ // an initial container is successfully created.
+ case Event(SuccessfulCreationJob(creationId, _, _, _), FlushingData(schedulerActor, droppingActor, _, _, _)) =>
+ creationIds -= creationId.asString
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // log the failed information
+ case Event(FailedCreationJob(creationId, _, _, _, _, message), data: FlushingData) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Failed to create a container due to $message")
+
+ // keep updating the reason
+ stay using data.copy(reason = message)
+
+ // since there is no container, activations cannot be handled.
+ case Event(msg: ActivationMessage, data: FlushingData) =>
+ completeErrorActivation(msg, data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ stay() using data.copy(activeDuringFlush = true)
+
+ // Since SchedulingDecisionMaker keep sending a message to create a container, this state is not automatically timed out.
+ // Instead, StateTimeout message will be sent by a timer.
+ case Event(StateTimeout, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ if (data.activeDuringFlush)
+ stay using data.copy(activeDuringFlush = false)
+ else
+ cleanUpActorsAndGotoRemoved(data)
+
+ case Event(GracefulShutdown, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpActorsAndGotoRemoved(data)
+ }
+
+ // in case there is any activation in the queue, it waits until all of them are handled.
+ when(Removing, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // When there is no message in the queue, SchedulingDecisionMaker would stop sending any message
+ // So the queue can be timed out on every gracefulShutdownTimeout
+ case Event(QueueRemovedCompleted | StateTimeout, data: RemovingData) =>
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(GracefulShutdown, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received GracefulShutdown trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(StopSchedulingAsOutdated, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received StopSchedulingAsOutdated trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data.copy(outdated = true))
+ }
+
+ when(Removed, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // since this Queue will be terminated, rescheduling the msg
+ case Event(msg: ActivationMessage, _: NoData) =>
+ context.parent ! msg
+ stay()
+
+ // this queue is going to stop so let client connect to a new queue
+ case Event(request: GetActivation, _: NoData) if request.action == action =>
+ implicit val tid = request.transactionId
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Get activation request ${request.containerId}, let client connect to a new queue.")
+ forwardAllActivations(context.parent)
+ sender ! GetActivationResponse(Left(NoMemoryQueue()))
+
+ stay
+
+ // actors and data are already wiped
+ case Event(QueueRemovedCompleted, _: NoData) =>
+ stop()
+
+ // This is not supposed to happen. This will ensure the queue does not run forever.
+ // This can happen when QueueManager could not respond with QueueRemovedCompleted for some reason.
+ case Event(StateTimeout, _: NoData) =>
+ context.parent ! queueRemovedMsg
+
+ stop()
+
+ // This queue is going to stop, do nothing
+ case Event(msg @ (StopSchedulingAsOutdated | GracefulShutdown), _: NoData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received $msg but do nothing as it is going to stop.")
+ stay
+ }
+
+ whenUnhandled {
+ // The queue endpoint is removed, trying to restore it.
+ case Event(WatchEndpointRemoved(_, `leaderKey`, value, false), data) =>
+ data match {
+ case RemovingData(_, _, _) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] This queue is shutdown by `/disable` api, do nothing here.")
+ case _ =>
+ dataManagementService ! RegisterInitialData(leaderKey, value, failoverEnabled = false, Some(self)) // the watcher is already setup
+ }
+ stay
+
+ // we don't care the storage results for namespaceThrottlingKey
+ case Event(InitialDataStorageResults(`namespaceThrottlingKey`, _), _) =>
+ stay
+
+ // The queue endpoint is restored
+ case Event(InitialDataStorageResults(`leaderKey`, Right(_)), _) =>
+ stay
+
+ // this can be a case that there is another queue already running.
+ // it can happen if a node is segregated by the temporal network rupture and the queue endpoint is removed.
+ case Event(InitialDataStorageResults(`leaderKey`, Left(_)), data) =>
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] the queue is superseded by a new queue.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! queueRemovedMsg
+
+ // forward all activations to the parent queue manager.
+ // parent queue manager is supposed to removed the reference of this queue and forward messages to a new queue
+ forwardAllActivations(context.parent)
+
+ // only clean up actors because etcd data is already being used by another queue
+ cleanUpActors(data)
+
+ goto(Removed) using NoData()
+
+ case Event(WatchEndpointRemoved(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds -= key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers -= key.split("/").last
+ case _ =>
+ }
+ stay
+
+ case Event(WatchEndpointInserted(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds += key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers += key.split("/").last
+ case _ =>
+ }
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled
+ case Event(SuccessfulCreationJob(creationId, _, _, _), _) =>
+ creationIds -= creationId.asString
+ stay()
+
+ // for other cases
+ case Event(FailedCreationJob(creationId, invocationNamespace, action, revision, _, message), _) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Got failed creation job with revision $revision and error $message.")
+ stay()
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(cancel: CancelPoll, _) =>
+ cancel.promise.trySuccess(Left(NoActivationMessage()))
+
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(msg: ActivationMessage, _) =>
+ handleActivationMessage(msg)
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action == action =>
+ implicit val tid = request.transactionId
+ if (request.alive) {
+ containers += request.containerId
+ handleActivationRequest(request)
+ } else {
+ logging.info(this, s"Remove containerId because ${request.containerId} is not alive")
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ containers -= request.containerId
+ stay
+ }
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action != action =>
+ implicit val tid = request.transactionId
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] version mismatch ${request.action}")
+ sender ! GetActivationResponse(Left(ActionMismatch()))
+
+ stay
+
+ case Event(DropOld, _) =>
+ if (queue.nonEmpty && Duration
+ .between(queue.head.timestamp, Instant.now)
+ .compareTo(Duration.ofMillis(queueConfig.maxRetentionMs)) < 0) {
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Drop some stale activations for $revision, existing container is ${containers.size}, inProgress container is ${creationIds.size}, state data: $stateData, in is $in, current: ${queue.size}.")
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] the head stale message: ${queue.head.msg.activationId}")
+ }
+ queue = MemoryQueue.dropOld(queue, Duration.ofMillis(queueConfig.maxRetentionMs), completeErrorActivation)
+
+ stay
+
+ // common case for all statuses
+ case Event(StatusQuery, _) =>
+ sender ! StatusData(invocationNamespace, action.asString, queue.size, stateName.toString, stateData.toString)
+ stay
+
+ // Common case for all cases
+ case Event(GracefulShutdown, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Gracefully shutdown the memory queue.")
+ // delete relative data, e.g leaderKey, namespaceThrottlingKey, actionThrottlingKey
+ cleanUpData()
+
+ // let queue manager knows this queue is going to stop and let it forward incoming activations to a new queue
+ context.parent ! queueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = false)
+
+ // the version is updated. it's a shared case for all states
+ case Event(StopSchedulingAsOutdated, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = true)
+
+ case Event(t: FailureMessage, _) =>
+ logging.error(this, s"[$invocationNamespace:$action:$stateName] got an unexpected failure message: $t")
+
+ stay
+
+ case Event(msg: DecisionResults, _) =>
+ val DecisionResults(result, num) = msg
+ result match {
+ case AddInitialContainer if num > 0 =>
Review comment:
AFAIK, there is no case where SchedulingDecisionMaker sends `AddInitialContainer` with `num == 0`?
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/common/AverageRingBuffer.scala
##########
@@ -0,0 +1,34 @@
+package org.apache.openwhisk.common
+
+object AverageRingBuffer {
+ def apply(maxSize: Int) = new AverageRingBuffer(maxSize)
+}
+
+/**
+ * This buffer provides the average of the given elements.
+ * The number of elements are limited and the first element is removed if the maximum size is reached.
+ * Since it is based on the Vector, its operation takes effectively constant time.
+ * For more details, please visit https://docs.scala-lang.org/overviews/collections/performance-characteristics.html
+ *
+ * @param maxSize the maximum size of the buffer
+ */
+class AverageRingBuffer(private val maxSize: Int) {
Review comment:
Yes, that makes sense.
The ring size in our case was relatively small such as 10, so we calculated the average of recent 10 activations. Even if there was a skew at some point, it quickly gets back to the "normal" average if the slow activation is a transient issue.
Also, it only affects the timing to add more containers, and activations are still being processed by existing containers. So there was no critical impact.
Basically, the median would be better, but the average is a much simpler and cheaper solution so I chose it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] ningyougang commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
ningyougang commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r637786020
##########
File path: .travis.yml
##########
@@ -27,6 +27,7 @@ env:
global:
- ANSIBLE_CMD="ansible-playbook -i environments/local -e docker_image_prefix=testing"
- GRADLE_PROJS_SKIP=""
+ - OW_SCALA_VERSION=2.13
Review comment:
Great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (364cc8e) into [master](https://codecov.io/gh/apache/openwhisk/commit/ecb2a980659f28d0adbd9ef837afaf4cb2b695bf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ecb2a98) will **decrease** coverage by `0.25%`.
> The diff coverage is `4.20%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
==========================================
- Coverage 28.91% 28.66% -0.26%
==========================================
Files 219 231 +12
Lines 11775 12815 +1040
Branches 506 555 +49
==========================================
+ Hits 3405 3673 +268
- Misses 8370 9142 +772
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `64.97% <0.00%> (-4.83%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `91.01% <0.00%> (-1.04%)` | :arrow_down: |
| [...rpool/logging/DockerToActivationFileLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL0RvY2tlclRvQWN0aXZhdGlvbkZpbGVMb2dTdG9yZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [.../containerpool/logging/ElasticSearchLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL0VsYXN0aWNTZWFyY2hMb2dTdG9yZS5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...sk/core/containerpool/logging/SplunkLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL1NwbHVua0xvZ1N0b3JlLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...penwhisk/core/database/ActivationFileStorage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQWN0aXZhdGlvbkZpbGVTdG9yYWdlLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...ache/openwhisk/core/database/ActivationStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQWN0aXZhdGlvblN0b3JlLnNjYWxh) | `92.30% <ø> (ø)` | |
| [...nwhisk/core/database/ArtifactActivationStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQXJ0aWZhY3RBY3RpdmF0aW9uU3RvcmUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...penwhisk/core/database/ArtifactStoreProvider.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQXJ0aWZhY3RTdG9yZVByb3ZpZGVyLnNjYWxh) | `66.66% <ø> (ø)` | |
| ... and [74 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [99fb2c5...364cc8e](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (99c95cd) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **decrease** coverage by `20.32%`.
> The diff coverage is `0.51%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
- Coverage 43.88% 23.55% -20.33%
===========================================
Files 231 234 +3
Lines 12807 13381 +574
Branches 528 513 -15
===========================================
- Hits 5620 3152 -2468
- Misses 7187 10229 +3042
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `62.10% <0.00%> (-10.37%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `88.19% <100.00%> (-7.46%)` | :arrow_down: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `53.00% <100.00%> (-14.35%)` | :arrow_down: |
| [...pache/openwhisk/http/LenientSprayJsonSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvTGVuaWVudFNwcmF5SnNvblN1cHBvcnQuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...hisk/core/scheduler/message/ContainerMessage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvbWVzc2FnZS9Db250YWluZXJNZXNzYWdlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../openwhisk/core/scheduler/FPCSchedulerServer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvRlBDU2NoZWR1bGVyU2VydmVyLnNjYWxh) | `0.00% <0.00%> (-95.84%)` | :arrow_down: |
| ... and [84 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...99c95cd](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (90464ba) into [master](https://codecov.io/gh/apache/openwhisk/commit/9c445f372f4e504acdd97fa20d046692fe61294d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9c445f3) will **decrease** coverage by `75.87%`.
> The diff coverage is `0.51%`.
> :exclamation: Current head 90464ba differs from pull request most recent head bc6e18a. Consider uploading reports for the commit bc6e18a to get more accurate results
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
==========================================
- Coverage 81.11% 5.23% -75.88%
==========================================
Files 214 224 +10
Lines 10913 11749 +836
Branches 469 517 +48
==========================================
- Hits 8852 615 -8237
- Misses 2061 11134 +9073
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `40.74% <0.00%> (-44.84%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `32.55% <0.00%> (-61.56%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/connector/Message.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29ubmVjdG9yL01lc3NhZ2Uuc2NhbGE=) | `15.54% <0.00%> (-47.67%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...sk/core/scheduler/grpc/ActivationServiceImpl.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvZ3JwYy9BY3RpdmF0aW9uU2VydmljZUltcGwuc2NhbGE=) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `65.18% <100.00%> (-30.36%)` | :arrow_down: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `45.00% <100.00%> (-44.80%)` | :arrow_down: |
| ... and [205 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1829e1...bc6e18a](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-881155383
After rebasing this PR, it started failing to run compile `compileScoverageScala`.
It seems it works well with scala 2.13 but fails with 2.12.
Not quite sure the reason yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6661b33) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **decrease** coverage by `39.18%`.
> The diff coverage is `0.51%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
==========================================
- Coverage 43.88% 4.69% -39.19%
==========================================
Files 231 234 +3
Lines 12807 13381 +574
Branches 528 513 -15
==========================================
- Hits 5620 628 -4992
- Misses 7187 12753 +5566
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `41.09% <0.00%> (-31.37%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `65.83% <100.00%> (-29.82%)` | :arrow_down: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `45.00% <100.00%> (-22.35%)` | :arrow_down: |
| [...ain/scala/org/apache/openwhisk/spi/SpiLoader.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL3NwaS9TcGlMb2FkZXIuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...scala/org/apache/openwhisk/core/FeatureFlags.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvRmVhdHVyZUZsYWdzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...la/org/apache/openwhisk/http/BasicRasService.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvQmFzaWNSYXNTZXJ2aWNlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [123 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...6661b33](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d2668a2) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **increase** coverage by `19.89%`.
> The diff coverage is `0.86%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
===========================================
+ Coverage 43.73% 63.63% +19.89%
===========================================
Files 231 227 -4
Lines 12807 13121 +314
Branches 528 544 +16
===========================================
+ Hits 5601 8349 +2748
+ Misses 7206 4772 -2434
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `67.87% <0.00%> (-4.60%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `93.82% <100.00%> (-1.83%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `60.00% <100.00%> (+37.77%)` | :arrow_up: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `87.00% <100.00%> (+19.65%)` | :arrow_up: |
| [...che/openwhisk/core/invoker/LogStoreCollector.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9Mb2dTdG9yZUNvbGxlY3Rvci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...hisk/core/scheduler/message/ContainerMessage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvbWVzc2FnZS9Db250YWluZXJNZXNzYWdlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../openwhisk/core/scheduler/FPCSchedulerServer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvRlBDU2NoZWR1bGVyU2VydmVyLnNjYWxh) | `0.00% <0.00%> (-95.84%)` | :arrow_down: |
| ... and [168 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...d2668a2](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r657493878
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/MemoryQueue.scala
##########
@@ -0,0 +1,1097 @@
+package org.apache.openwhisk.core.scheduler.queue
+
+import java.time.{Duration, Instant}
+
+import akka.actor.Status.{Failure => FailureMessage}
+import akka.actor.{ActorRef, ActorSystem, Cancellable, FSM, Props, Stash}
+import akka.util.Timeout
+import org.apache.openwhisk.common._
+import org.apache.openwhisk.core.ack.ActiveAck
+import org.apache.openwhisk.core.connector.ContainerCreationError.{TooManyConcurrentRequests, ZeroNamespaceLimit}
+import org.apache.openwhisk.core.connector._
+import org.apache.openwhisk.core.database.{NoDocumentException, UserContext}
+import org.apache.openwhisk.core.entity._
+import org.apache.openwhisk.core.entity.size._
+import org.apache.openwhisk.core.etcd.EtcdClient
+import org.apache.openwhisk.core.etcd.EtcdKV.ContainerKeys.containerPrefix
+import org.apache.openwhisk.core.etcd.EtcdKV.{ContainerKeys, QueueKeys, ThrottlingKeys}
+import org.apache.openwhisk.core.scheduler.SchedulerEndpoints
+import org.apache.openwhisk.core.scheduler.grpc.{GetActivation, ActivationResponse => GetActivationResponse}
+import org.apache.openwhisk.core.scheduler.message.{
+ ContainerCreation,
+ ContainerDeletion,
+ FailedCreationJob,
+ SuccessfulCreationJob
+}
+import org.apache.openwhisk.core.service._
+import org.apache.openwhisk.core.{ConfigKeys, WhiskConfig}
+import org.apache.openwhisk.http.Messages.{namespaceLimitUnderZero, tooManyConcurrentRequests}
+import pureconfig.generic.auto._
+import pureconfig.loadConfigOrThrow
+import spray.json._
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+import scala.collection.mutable
+import scala.concurrent.duration._
+import scala.concurrent.{duration, ExecutionContextExecutor, Future, Promise}
+import scala.language.postfixOps
+import scala.util.{Failure, Success}
+
+// States
+sealed trait MemoryQueueState
+case object Uninitialized extends MemoryQueueState
+case object Running extends MemoryQueueState
+case object Idle extends MemoryQueueState
+case object Flushing extends MemoryQueueState
+case object Removing extends MemoryQueueState
+case object Removed extends MemoryQueueState
+case object ActionThrottled extends MemoryQueueState
+case object NamespaceThrottled extends MemoryQueueState
+
+// Data
+sealed abstract class MemoryQueueData()
+case class NoData() extends MemoryQueueData()
+case class NoActors() extends MemoryQueueData()
+case class RunningData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class ThrottledData(schedulerActor: ActorRef, droppingActor: ActorRef) extends MemoryQueueData()
+case class FlushingData(schedulerActor: ActorRef,
+ droppingActor: ActorRef,
+ error: ContainerCreationError,
+ reason: String,
+ activeDuringFlush: Boolean = false)
+ extends MemoryQueueData()
+case class RemovingData(schedulerActor: ActorRef, droppingActor: ActorRef, outdated: Boolean) extends MemoryQueueData()
+
+// Events sent by the actor
+case class QueueRemoved(invocationNamespace: String, action: DocInfo, leaderKey: Option[String])
+case class QueueReactivated(invocationNamespace: String, action: FullyQualifiedEntityName, docInfo: DocInfo)
+case class CancelPoll(promise: Promise[Either[MemoryQueueError, ActivationMessage]])
+case object QueueRemovedCompleted
+case object FlushPulse
+
+// Events received by the actor
+case object Start
+case object VersionUpdated
+case object StopSchedulingAsOutdated
+
+sealed trait RequiredAction
+case object Skip extends RequiredAction
+case object AddInitialContainer extends RequiredAction
+case object AddContainer extends RequiredAction
+case class EnableNamespaceThrottling(dropMsg: Boolean) extends RequiredAction
+case object DisableNamespaceThrottling extends RequiredAction
+case object EnableActionThrottling extends RequiredAction
+case object DisableActionThrottling extends RequiredAction
+case object Pausing extends RequiredAction
+case class DecisionResults(required: RequiredAction, num: Int)
+
+case class TimeSeriesActivationEntry(timestamp: Instant, msg: ActivationMessage)
+
+class MemoryQueue(private val etcdClient: EtcdClient,
+ private val durationChecker: DurationChecker,
+ private val action: FullyQualifiedEntityName,
+ messagingProducer: MessageProducer,
+ config: WhiskConfig,
+ invocationNamespace: String,
+ revision: DocRevision,
+ endpoints: SchedulerEndpoints,
+ actionMetaData: WhiskActionMetaData,
+ dataManagementService: ActorRef,
+ watcherService: ActorRef,
+ containerManager: ActorRef,
+ decisionMaker: ActorRef,
+ schedulerId: SchedulerInstanceId,
+ ack: ActiveAck,
+ store: (TransactionId, WhiskActivation, UserContext) => Future[Any],
+ getUserLimit: String => Future[Int],
+ checkToDropStaleActivation: (Queue[TimeSeriesActivationEntry],
+ Long,
+ String,
+ FullyQualifiedEntityName,
+ MemoryQueueState,
+ ActorRef) => Unit,
+ queueConfig: QueueConfig)(implicit logging: Logging)
+ extends FSM[MemoryQueueState, MemoryQueueData]
+ with Stash {
+
+ private implicit val ec: ExecutionContextExecutor = context.dispatcher
+ private implicit val actorSystem: ActorSystem = context.system
+ private implicit val timeout = Timeout(5.seconds)
+ private implicit val order: Ordering[BufferedRequest] = Ordering.by(_.containerId)
+
+ private val unversionedAction = action.copy(version = None)
+ private val checkInterval: FiniteDuration = 100 milliseconds
+ private val StaleThreshold: Double = 100.0
+ private val StaleDuration = Duration.ofMillis(StaleThreshold.toLong)
+ private val dropInterval: FiniteDuration = 10 seconds
+ private val leaderKey = QueueKeys.queue(invocationNamespace, unversionedAction, leader = true)
+ private val inProgressContainerPrefixKey =
+ containerPrefix(ContainerKeys.inProgressPrefix, invocationNamespace, action, Some(revision))
+ private val existingContainerPrefixKey =
+ containerPrefix(ContainerKeys.namespacePrefix, invocationNamespace, action, Some(revision))
+ private val namespaceThrottlingKey = ThrottlingKeys.namespace(EntityName(invocationNamespace))
+ private val actionThrottlingKey = ThrottlingKeys.action(invocationNamespace, unversionedAction)
+ private val pollTimeOut = 1.seconds
+ private var requestBuffer = mutable.PriorityQueue.empty[BufferedRequest]
+ private val memory = actionMetaData.limits.memory.megabytes.MB
+ private val queueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), Some(leaderKey))
+ private val staleQueueRemovedMsg = QueueRemoved(invocationNamespace, action.toDocId.asDocInfo(revision), None)
+
+ private[queue] var containers = Set.empty[String]
+ private[queue] var creationIds = Set.empty[String]
+
+ private[queue] var queue = Queue.empty[TimeSeriesActivationEntry]
+ private[queue] var in: Int = 0
+ private[queue] val namespaceContainerCount = NamespaceContainerCount(invocationNamespace, etcdClient, watcherService)
+ private[queue] var averageDuration: Option[Double] = None
+ private[queue] var averageDurationBuffer = AverageRingBuffer(queueConfig.durationBufferSize)
+ private[queue] var limit: Option[Int] = None
+ private[queue] var initialized = false
+
+ private val logScheduler: Cancellable = context.system.scheduler.scheduleWithFixedDelay(0.seconds, 1.seconds) { () =>
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_QUEUE_WAITING_ACTIVATION(s"$invocationNamespace/$action"),
+ queue.size)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_CONTAINER(invocationNamespace),
+ namespaceContainerCount.existingContainerNumByNamespace)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_NAMESPACE_INPROGRESS_CONTAINER(invocationNamespace),
+ namespaceContainerCount.inProgressContainerNumByNamespace)
+
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_CONTAINER(invocationNamespace, action.asString),
+ containers.size)
+ MetricEmitter.emitGaugeMetric(
+ LoggingMarkers.SCHEDULER_ACTION_INPROGRESS_CONTAINER(invocationNamespace, action.asString),
+ creationIds.size)
+ }
+
+ getAverageDuration()
+
+ private val watcherName = s"memory-queue-$action-$revision"
+ // watch existing containers for action and namespace
+ private val watchedKeys = Seq(inProgressContainerPrefixKey, existingContainerPrefixKey)
+
+ watchedKeys.foreach { key =>
+ watcherService ! WatchEndpoint(key, "", isPrefix = true, watcherName, Set(PutEvent, DeleteEvent))
+ }
+
+ startWith(Uninitialized, NoData())
+
+ when(Uninitialized) {
+ case Event(Start, _) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] a new queue is created.")
+ val (schedulerActor, droppingActor) = startMonitoring()
+ initializeThrottling()
+
+ watcherService ! WatchEndpoint(leaderKey, endpoints.serialize, isPrefix = false, watcherName, Set(DeleteEvent))
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // this is the case that the action version is updated, so no data needs to be stored
+ case Event(VersionUpdated, _) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // other messages should not be handled in this state.
+ case _ =>
+ stash()
+ stay
+ }
+
+ when(Running, stateTimeout = queueConfig.idleGrace) {
+ case Event(EnableNamespaceThrottling(dropMsg), data: RunningData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Enable namespace throttling.")
+ enableNamespaceThrottling()
+
+ // if no container could be created, it is same with Flushing state.
+ if (dropMsg) {
+ completeAllActivations(tooManyConcurrentRequests, isWhiskError = false)
+ goto(Flushing) using FlushingData(
+ data.schedulerActor,
+ data.droppingActor,
+ TooManyConcurrentRequests,
+ tooManyConcurrentRequests)
+ } else {
+ // if there are already some containers running, activations can still be processed so goto the NamespaceThrottled state.
+ goto(NamespaceThrottled) using ThrottledData(data.schedulerActor, data.droppingActor)
+ }
+
+ case Event(StateTimeout, data: RunningData) =>
+ if (queue.isEmpty && (containers.size + creationIds.size) <= 0) {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] No activations coming in ${queueConfig.idleGrace}")
+ actorSystem.stop(data.schedulerActor)
+ actorSystem.stop(data.droppingActor)
+
+ goto(Idle) using NoActors()
+ } else {
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue is timed out but there are still ${queue.size} activation messages or (running: ${containers.size}, in-progress: ${creationIds.size}) containers")
+ stay
+ }
+
+ case Event(FailedCreationJob(creationId, _, _, _, error, message), RunningData(schedulerActor, droppingActor)) =>
+ creationIds -= creationId.asString
+ // when there is no container, it moves to the Flushing state as no activations can be invoked
+ if (containers.size <= 0) {
+ val isWhiskError = ContainerCreationError.whiskErrors.contains(error)
+ completeAllActivations(message, isWhiskError)
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Failed to create an initial container due to ${if (isWhiskError) "whiskError"
+ else "developerError"}, reason: $message.")
+
+ goto(Flushing) using FlushingData(schedulerActor, droppingActor, error, message)
+ } else
+ // if there are already some containers running, activations can be handled anyway.
+ stay
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(NamespaceThrottled) {
+ case Event(msg: ActivationMessage, _: ThrottledData) =>
+ handleActivationMessage(msg)
+ stay
+
+ case Event(DisableNamespaceThrottling, data: ThrottledData) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Disable namespace throttling.")
+ disableNamespaceThrottling()
+ goto(Running) using RunningData(data.schedulerActor, data.schedulerActor)
+ }
+
+ // there is no timeout for this state as when there is no further message, it would move to the Running state again.
+ when(ActionThrottled) {
+ // since there are already too many activation messages, it drops the new messages
+ case Event(msg: ActivationMessage, ThrottledData(_, _)) =>
+ completeErrorActivation(msg, tooManyConcurrentRequests, isWhiskError = false)
+ stay
+ }
+
+ when(Idle, stateTimeout = queueConfig.stopGrace) {
+ case Event(msg: ActivationMessage, _: NoActors) =>
+ val (schedulerActor, droppingActor) = startMonitoring()
+ handleActivationMessage(msg)
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ case Event(request: GetActivation, _) if request.action == action =>
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ stay
+
+ case Event(StateTimeout, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] The queue is timed out, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(GracefulShutdown, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpDataAndGotoRemoved()
+
+ case Event(StopSchedulingAsOutdated, _: NoActors) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+
+ cleanUpWatcher()
+
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ // since the queue is outdated and there is no activation, delete all old containers.
+ containerManager ! ContainerDeletion(invocationNamespace, action, revision, actionMetaData)
+
+ goto(Removed) using NoData()
+ }
+
+ when(Flushing) {
+ // an initial container is successfully created.
+ case Event(SuccessfulCreationJob(creationId, _, _, _), FlushingData(schedulerActor, droppingActor, _, _, _)) =>
+ creationIds -= creationId.asString
+
+ goto(Running) using RunningData(schedulerActor, droppingActor)
+
+ // log the failed information
+ case Event(FailedCreationJob(creationId, _, _, _, _, message), data: FlushingData) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Failed to create a container due to $message")
+
+ // keep updating the reason
+ stay using data.copy(reason = message)
+
+ // since there is no container, activations cannot be handled.
+ case Event(msg: ActivationMessage, data: FlushingData) =>
+ completeErrorActivation(msg, data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ stay() using data.copy(activeDuringFlush = true)
+
+ // Since SchedulingDecisionMaker keep sending a message to create a container, this state is not automatically timed out.
+ // Instead, StateTimeout message will be sent by a timer.
+ case Event(StateTimeout, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ if (data.activeDuringFlush)
+ stay using data.copy(activeDuringFlush = false)
+ else
+ cleanUpActorsAndGotoRemoved(data)
+
+ case Event(GracefulShutdown, data: FlushingData) =>
+ completeAllActivations(data.reason, ContainerCreationError.whiskErrors.contains(data.error))
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Received GracefulShutdown, stop the queue.")
+ cleanUpActorsAndGotoRemoved(data)
+ }
+
+ // in case there is any activation in the queue, it waits until all of them are handled.
+ when(Removing, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // When there is no message in the queue, SchedulingDecisionMaker would stop sending any message
+ // So the queue can be timed out on every gracefulShutdownTimeout
+ case Event(QueueRemovedCompleted | StateTimeout, data: RemovingData) =>
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(GracefulShutdown, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received GracefulShutdown trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data)
+
+ case Event(StopSchedulingAsOutdated, data: RemovingData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received StopSchedulingAsOutdated trying to stop the queue.")
+ cleanUpActorsAndGotoRemovedIfPossible(data.copy(outdated = true))
+ }
+
+ when(Removed, stateTimeout = queueConfig.gracefulShutdownTimeout) {
+ // since this Queue will be terminated, rescheduling the msg
+ case Event(msg: ActivationMessage, _: NoData) =>
+ context.parent ! msg
+ stay()
+
+ // this queue is going to stop so let client connect to a new queue
+ case Event(request: GetActivation, _: NoData) if request.action == action =>
+ implicit val tid = request.transactionId
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Get activation request ${request.containerId}, let client connect to a new queue.")
+ forwardAllActivations(context.parent)
+ sender ! GetActivationResponse(Left(NoMemoryQueue()))
+
+ stay
+
+ // actors and data are already wiped
+ case Event(QueueRemovedCompleted, _: NoData) =>
+ stop()
+
+ // This is not supposed to happen. This will ensure the queue does not run forever.
+ // This can happen when QueueManager could not respond with QueueRemovedCompleted for some reason.
+ case Event(StateTimeout, _: NoData) =>
+ context.parent ! queueRemovedMsg
+
+ stop()
+
+ // This queue is going to stop, do nothing
+ case Event(msg @ (StopSchedulingAsOutdated | GracefulShutdown), _: NoData) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] The queue received $msg but do nothing as it is going to stop.")
+ stay
+ }
+
+ whenUnhandled {
+ // The queue endpoint is removed, trying to restore it.
+ case Event(WatchEndpointRemoved(_, `leaderKey`, value, false), data) =>
+ data match {
+ case RemovingData(_, _, _) =>
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName] This queue is shutdown by `/disable` api, do nothing here.")
+ case _ =>
+ dataManagementService ! RegisterInitialData(leaderKey, value, failoverEnabled = false, Some(self)) // the watcher is already setup
+ }
+ stay
+
+ // we don't care the storage results for namespaceThrottlingKey
+ case Event(InitialDataStorageResults(`namespaceThrottlingKey`, _), _) =>
+ stay
+
+ // The queue endpoint is restored
+ case Event(InitialDataStorageResults(`leaderKey`, Right(_)), _) =>
+ stay
+
+ // this can be a case that there is another queue already running.
+ // it can happen if a node is segregated by the temporal network rupture and the queue endpoint is removed.
+ case Event(InitialDataStorageResults(`leaderKey`, Left(_)), data) =>
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] the queue is superseded by a new queue.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! queueRemovedMsg
+
+ // forward all activations to the parent queue manager.
+ // parent queue manager is supposed to removed the reference of this queue and forward messages to a new queue
+ forwardAllActivations(context.parent)
+
+ // only clean up actors because etcd data is already being used by another queue
+ cleanUpActors(data)
+
+ goto(Removed) using NoData()
+
+ case Event(WatchEndpointRemoved(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds -= key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers -= key.split("/").last
+ case _ =>
+ }
+ stay
+
+ case Event(WatchEndpointInserted(watchKey, key, _, true), _) =>
+ watchKey match {
+ case `inProgressContainerPrefixKey` =>
+ creationIds += key.split("/").last
+ case `existingContainerPrefixKey` =>
+ containers += key.split("/").last
+ case _ =>
+ }
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled
+ case Event(SuccessfulCreationJob(creationId, _, _, _), _) =>
+ creationIds -= creationId.asString
+ stay()
+
+ // for other cases
+ case Event(FailedCreationJob(creationId, invocationNamespace, action, revision, _, message), _) =>
+ creationIds -= creationId.asString
+ logging.info(
+ this,
+ s"[$invocationNamespace:$action:$stateName][$creationId] Got failed creation job with revision $revision and error $message.")
+ stay()
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(cancel: CancelPoll, _) =>
+ cancel.promise.trySuccess(Left(NoActivationMessage()))
+
+ stay
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(msg: ActivationMessage, _) =>
+ handleActivationMessage(msg)
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action == action =>
+ implicit val tid = request.transactionId
+ if (request.alive) {
+ containers += request.containerId
+ handleActivationRequest(request)
+ } else {
+ logging.info(this, s"Remove containerId because ${request.containerId} is not alive")
+ sender ! GetActivationResponse(Left(NoActivationMessage()))
+ containers -= request.containerId
+ stay
+ }
+
+ // common case for Running, NamespaceThrottled, ActionThrottled, Removing
+ case Event(request: GetActivation, _) if request.action != action =>
+ implicit val tid = request.transactionId
+ logging.warn(this, s"[$invocationNamespace:$action:$stateName] version mismatch ${request.action}")
+ sender ! GetActivationResponse(Left(ActionMismatch()))
+
+ stay
+
+ case Event(DropOld, _) =>
+ if (queue.nonEmpty && Duration
+ .between(queue.head.timestamp, Instant.now)
+ .compareTo(Duration.ofMillis(queueConfig.maxRetentionMs)) < 0) {
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] Drop some stale activations for $revision, existing container is ${containers.size}, inProgress container is ${creationIds.size}, state data: $stateData, in is $in, current: ${queue.size}.")
+ logging.error(
+ this,
+ s"[$invocationNamespace:$action:$stateName] the head stale message: ${queue.head.msg.activationId}")
+ }
+ queue = MemoryQueue.dropOld(queue, Duration.ofMillis(queueConfig.maxRetentionMs), completeErrorActivation)
+
+ stay
+
+ // common case for all statuses
+ case Event(StatusQuery, _) =>
+ sender ! StatusData(invocationNamespace, action.asString, queue.size, stateName.toString, stateData.toString)
+ stay
+
+ // Common case for all cases
+ case Event(GracefulShutdown, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] Gracefully shutdown the memory queue.")
+ // delete relative data, e.g leaderKey, namespaceThrottlingKey, actionThrottlingKey
+ cleanUpData()
+
+ // let queue manager knows this queue is going to stop and let it forward incoming activations to a new queue
+ context.parent ! queueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = false)
+
+ // the version is updated. it's a shared case for all states
+ case Event(StopSchedulingAsOutdated, data) =>
+ logging.info(this, s"[$invocationNamespace:$action:$stateName] stop further scheduling.")
+ // let QueueManager know this queue is no longer in charge.
+ context.parent ! staleQueueRemovedMsg
+
+ goto(Removing) using getRemovingData(data, outdated = true)
+
+ case Event(t: FailureMessage, _) =>
+ logging.error(this, s"[$invocationNamespace:$action:$stateName] got an unexpected failure message: $t")
+
+ stay
+
+ case Event(msg: DecisionResults, _) =>
+ val DecisionResults(result, num) = msg
+ result match {
+ case AddInitialContainer if num > 0 =>
Review comment:
Should this be `if num = 0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r657497314
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/SchedulingDecisionMaker.scala
##########
@@ -0,0 +1,244 @@
+package org.apache.openwhisk.core.scheduler.queue
+
+import akka.actor.{Actor, ActorSystem, Props}
+import org.apache.openwhisk.common.Logging
+import org.apache.openwhisk.core.entity.FullyQualifiedEntityName
+
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Failure, Success}
+
+class SchedulingDecisionMaker(
+ invocationNamespace: String,
+ action: FullyQualifiedEntityName,
+ StaleThreshold: Double = 100.0)(implicit val actorSystem: ActorSystem, ec: ExecutionContext, logging: Logging)
+ extends Actor {
+
+ override def receive: Receive = {
+ case msg: QueueSnapshot =>
+ decide(msg)
+ .andThen {
+ case Success(DecisionResults(Skip, _)) =>
+ // do nothing
+ case Success(result: DecisionResults) =>
+ msg.recipient ! result
+ case Failure(e) =>
+ logging.error(this, s"failed to make a scheduling decision due to $e");
+ }
+ }
+
+ private[queue] def decide(snapshot: QueueSnapshot) = {
+ val QueueSnapshot(
+ initialized,
+ incoming,
+ currentMsg,
+ existing,
+ inProgress,
+ staleActivationNum,
+ existingContainerCountInNs,
+ inProgressContainerCountInNs,
+ averageDuration,
+ limit,
+ stateName,
+ _) = snapshot
+ val totalContainers = existing + inProgress
+ val availableMsg = currentMsg + incoming
+
+ if (limit <= 0) {
+ // this is an error case, the limit should be bigger than 0
+ stateName match {
+ case Flushing => Future.successful(DecisionResults(Skip, 0))
+ case _ => Future.successful(DecisionResults(Pausing, 0))
+ }
+ } else {
+ val capacity = limit - existingContainerCountInNs - inProgressContainerCountInNs
+ if (capacity <= 0) {
+ stateName match {
+
+ /**
+ * If the container is created later (for any reason), all activations fail(too many requests).
+ *
+ * However, if the container exists(totalContainers != 0), the activation is not treated as a failure and the activation is delivered to the container.
+ */
+ case Running =>
+ logging.info(
+ this,
+ s"there is no capacity activations will be dropped or throttled, (availableMsg: $availableMsg totalContainers: $totalContainers, limit: $limit, namespaceContainers: ${existingContainerCountInNs}, namespaceInProgressContainer: ${inProgressContainerCountInNs}) [$invocationNamespace:$action]")
+ Future.successful(DecisionResults(EnableNamespaceThrottling(dropMsg = totalContainers == 0), 0))
+
+ // do nothing
+ case _ =>
+ // no need to print any messages if the state is already NamespaceThrottled
+ Future.successful(DecisionResults(Skip, 0))
+ }
+ } else {
+ (stateName, averageDuration) match {
+ // there is no container
+ case (Running, None) if totalContainers == 0 && !initialized =>
+ logging.info(
+ this,
+ s"add one initial container if totalContainers($totalContainers) == 0 [$invocationNamespace:$action]")
+ Future.successful(DecisionResults(AddInitialContainer, 1))
+
+ // Todo: when disabling throttling we may create some containers.
+ case (NamespaceThrottled, _) =>
+ Future.successful(DecisionResults(DisableNamespaceThrottling, 0))
+
+ // this is an exceptional case, create a container immediately
+ case (Running, _) if totalContainers == 0 && availableMsg > 0 =>
+ logging.info(
+ this,
+ s"add one container if totalContainers($totalContainers) == 0 && availableMsg($availableMsg) > 0 [$invocationNamespace:$action]")
+ Future.successful(DecisionResults(AddContainer, 1))
+
+ case (Flushing, _) if totalContainers == 0 =>
+ logging.info(
+ this,
+ s"add one container case Paused if totalContainers($totalContainers) == 0 [$invocationNamespace:$action]")
+ // it is highly likely the queue could not create an initial container if the limit is 0
+ Future.successful(DecisionResults(AddInitialContainer, 1))
+
+ // there is no activation result yet, but some activations became stale
+ // it may cause some over-provisioning if it takes much time to create a container and execution time is short.
+ // but it is a kind of trade-off and we place latency on top of over-provisioning
+ case (Running, None) if staleActivationNum > 0 =>
+ // we can safely get the value as we already checked the existence
+ val num = staleActivationNum - inProgress
+ // if it tries to create more containers than existing messages, we just create shortage
+ val actualNum = if (num > availableMsg) availableMsg else num
+ addServersIfPossible(
+ existing,
+ inProgress,
+ 0,
+ availableMsg,
+ capacity,
+ actualNum,
+ staleActivationNum,
+ 0.0,
+ Running)
+
+ case (Running, Some(duration)) if staleActivationNum > 0 =>
+ // we can safely get the value as we already checked the existence
+ val containerThroughput = StaleThreshold / duration
+ val num = ceiling(availableMsg.toDouble / containerThroughput)
+ // if it tries to create more containers than existing messages, we just create shortage
+ val actualNum = (if (num > availableMsg) availableMsg else num) - inProgress
+ addServersIfPossible(
+ existing,
+ inProgress,
+ containerThroughput,
+ availableMsg,
+ capacity,
+ actualNum,
+ staleActivationNum,
+ duration,
+ Running)
+
+ // need more containers and a message is already processed
+ case (Running, Some(duration)) =>
+ // we can safely get the value as we already checked the existence
+ val containerThroughput = StaleThreshold / duration
+ val expectedTps = containerThroughput * (existing + inProgress)
+
+ if (availableMsg >= expectedTps && existing + inProgress < availableMsg) {
+ val num = ceiling((availableMsg / containerThroughput) - existing - inProgress)
+ // if it tries to create more containers than existing messages, we just create shortage
+ val actualNum = if (num + totalContainers > availableMsg) availableMsg - totalContainers else num
+ addServersIfPossible(
+ existing,
+ inProgress,
+ containerThroughput,
+ availableMsg,
+ capacity,
+ actualNum,
+ staleActivationNum,
+ duration,
+ Running)
+ } else {
+ Future.successful(DecisionResults(Skip, 0))
+ }
+
+ // generally we assume there are enough containers for actions when shutting down the scheduler
+ // but if there were already too many activation in the queue with not enough containers,
+ // we should add more containers to quickly consume those messages.
+ // this case is for that as a last resort.
+ case (Removing, Some(duration)) if staleActivationNum > 0 =>
+ // we can safely get the value as we already checked the existence
+ val containerThroughput = StaleThreshold / duration
+ val num = ceiling(availableMsg.toDouble / containerThroughput)
+ // if it tries to create more containers than existing messages, we just create shortage
+ val actualNum = (if (num > availableMsg) availableMsg else num) - inProgress
+ addServersIfPossible(
+ existing,
+ inProgress,
+ containerThroughput,
+ availableMsg,
+ capacity,
+ actualNum,
+ staleActivationNum,
+ duration,
+ Running)
+
+ // same with the above case but no duration exist.
+ case (Removing, None) if staleActivationNum > 0 =>
+ // we can safely get the value as we already checked the existence
+ val num = staleActivationNum - inProgress
+ // if it tries to create more containers than existing messages, we just create shortage
+ val actualNum = if (num > availableMsg) availableMsg else num
+ addServersIfPossible(
+ existing,
+ inProgress,
+ 0,
+ availableMsg,
+ capacity,
+ actualNum,
+ staleActivationNum,
+ 0.0,
+ Running)
+
+ // do nothing
+ case _ =>
+ Future.successful(DecisionResults(Skip, 0))
+ }
+ }
+ }
+ }
+
+ private def addServersIfPossible(existing: Int,
Review comment:
Should this be called `addContainersIfPossible`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (99c95cd) into [master](https://codecov.io/gh/apache/openwhisk/commit/bf62f740057f5210ff05582d119fd692fb6c6341?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf62f74) will **decrease** coverage by `39.18%`.
> The diff coverage is `0.51%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
==========================================
- Coverage 43.88% 4.69% -39.19%
==========================================
Files 231 234 +3
Lines 12807 13381 +574
Branches 528 513 -15
==========================================
- Hits 5620 628 -4992
- Misses 7187 12753 +5566
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `41.09% <0.00%> (-31.37%)` | :arrow_down: |
| [.../org/apache/openwhisk/core/entity/CreationId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0NyZWF0aW9uSWQuc2NhbGE=) | `0.00% <0.00%> (-22.23%)` | :arrow_down: |
| [...e/openwhisk/core/scheduler/queue/MemoryQueue.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvTWVtb3J5UXVldWUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...core/scheduler/queue/SchedulingDecisionMaker.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvU2NoZWR1bGluZ0RlY2lzaW9uTWFrZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [.../scala/org/apache/openwhisk/core/WhiskConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2hpc2tDb25maWcuc2NhbGE=) | `65.83% <100.00%> (-29.82%)` | :arrow_down: |
| [...cala/org/apache/openwhisk/http/ErrorResponse.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvRXJyb3JSZXNwb25zZS5zY2FsYQ==) | `45.00% <100.00%> (-22.35%)` | :arrow_down: |
| [...ain/scala/org/apache/openwhisk/spi/SpiLoader.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL3NwaS9TcGlMb2FkZXIuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...scala/org/apache/openwhisk/core/FeatureFlags.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvRmVhdHVyZUZsYWdzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...la/org/apache/openwhisk/http/BasicRasService.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvQmFzaWNSYXNTZXJ2aWNlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [123 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bf62f74...99c95cd](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on a change in pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on a change in pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#discussion_r657485433
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/common/AverageRingBuffer.scala
##########
@@ -0,0 +1,34 @@
+package org.apache.openwhisk.common
+
+object AverageRingBuffer {
+ def apply(maxSize: Int) = new AverageRingBuffer(maxSize)
+}
+
+/**
+ * This buffer provides the average of the given elements.
+ * The number of elements are limited and the first element is removed if the maximum size is reached.
+ * Since it is based on the Vector, its operation takes effectively constant time.
+ * For more details, please visit https://docs.scala-lang.org/overviews/collections/performance-characteristics.html
+ *
+ * @param maxSize the maximum size of the buffer
+ */
+class AverageRingBuffer(private val maxSize: Int) {
Review comment:
What was the reasoning for picking average as the heuristic. Would median be a better heuristic here? i.e. if all activations take 100 milliseconds and then one activation has a slow call to a db that takes 10 seconds it will heavily skew the average.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter edited a comment on pull request #5110: [New Scheduler] Add memory queue for the new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5110:
URL: https://github.com/apache/openwhisk/pull/5110#issuecomment-834316259
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5110](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (364cc8e) into [master](https://codecov.io/gh/apache/openwhisk/commit/ecb2a980659f28d0adbd9ef837afaf4cb2b695bf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ecb2a98) will **decrease** coverage by `4.33%`.
> The diff coverage is `3.88%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #5110 +/- ##
==========================================
- Coverage 28.91% 24.58% -4.34%
==========================================
Files 219 231 +12
Lines 11775 12815 +1040
Branches 506 555 +49
==========================================
- Hits 3405 3150 -255
- Misses 8370 9665 +1295
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/openwhisk/common/AverageRingBuffer.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9BdmVyYWdlUmluZ0J1ZmZlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/openwhisk/common/Logging.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Mb2dnaW5nLnNjYWxh) | `62.21% <0.00%> (-7.60%)` | :arrow_down: |
| [...la/org/apache/openwhisk/common/TransactionId.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9UcmFuc2FjdGlvbklkLnNjYWxh) | `82.02% <0.00%> (-10.03%)` | :arrow_down: |
| [...rpool/logging/DockerToActivationFileLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL0RvY2tlclRvQWN0aXZhdGlvbkZpbGVMb2dTdG9yZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [.../containerpool/logging/ElasticSearchLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL0VsYXN0aWNTZWFyY2hMb2dTdG9yZS5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...sk/core/containerpool/logging/SplunkLogStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9sb2dnaW5nL1NwbHVua0xvZ1N0b3JlLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...penwhisk/core/database/ActivationFileStorage.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQWN0aXZhdGlvbkZpbGVTdG9yYWdlLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...ache/openwhisk/core/database/ActivationStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQWN0aXZhdGlvblN0b3JlLnNjYWxh) | `92.30% <ø> (ø)` | |
| [...nwhisk/core/database/ArtifactActivationStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQXJ0aWZhY3RBY3RpdmF0aW9uU3RvcmUuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
| [...penwhisk/core/database/ArtifactStoreProvider.scala](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQXJ0aWZhY3RTdG9yZVByb3ZpZGVyLnNjYWxh) | `66.66% <ø> (ø)` | |
| ... and [127 more](https://codecov.io/gh/apache/openwhisk/pull/5110/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [99fb2c5...364cc8e](https://codecov.io/gh/apache/openwhisk/pull/5110?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org