You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2021/03/01 06:04:52 UTC
[GitHub] [openwhisk] jiangpengcheng opened a new pull request #5072: [New Scheduler] Add container counter
jiangpengcheng opened a new pull request #5072:
URL: https://github.com/apache/openwhisk/pull/5072
Get container count from ETCD when related data get updated in ETCD
<!--- Provide a concise summary of your changes in the Title -->
## Description
<!--- Provide a detailed description of your changes. -->
<!--- Include details of what problem you are solving and how your changes are tested. -->
It watchs ETCD data changes using `WatcherService` implemented in #5069, when any new container data under specified namespace is inserted/removed to/from ETCD, it will get count of containers for the namespace by querying ETCD
## Related issue and scope
<!--- Please include a link to a related issue if there is one. -->
- [ ] I opened an issue to propose and discuss this change (#????)
## My changes affect the following components
<!--- Select below all system components are affected by your change. -->
<!--- Enter an `x` in all applicable boxes. -->
- [ ] API
- [ ] Controller
- [ ] Message Bus (e.g., Kafka)
- [ ] Loadbalancer
- [x] Scheduler
- [ ] Invoker
- [ ] Intrinsic actions (e.g., sequences, conductors)
- [ ] Data stores (e.g., CouchDB)
- [x] Tests
- [ ] Deployment
- [ ] CLI
- [ ] General tooling
- [ ] Documentation
## Types of changes
<!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
- [ ] Bug fix (generally a non-breaking change which closes an issue).
- [x] Enhancement or new feature (adds new functionality).
- [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
## Checklist:
<!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
- [x] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
- [x] I reviewed the [style guides](https://github.com/apache/openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :).
- [x] I added tests to cover my changes.
- [ ] My changes require further changes to the documentation.
- [ ] I updated the documentation where necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] jiangpengcheng commented on a change in pull request #5072: [New Scheduler] Add container counter
Posted by GitBox <gi...@apache.org>.
jiangpengcheng commented on a change in pull request #5072:
URL: https://github.com/apache/openwhisk/pull/5072#discussion_r584465898
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/ContainerCounter.scala
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.openwhisk.core.scheduler.queue
+
+import java.util.concurrent.atomic.AtomicInteger
+
+import akka.actor.{Actor, ActorRef, ActorSystem, Props}
+import org.apache.openwhisk.common.Logging
+import org.apache.openwhisk.core.etcd.EtcdClient
+import org.apache.openwhisk.core.etcd.EtcdKV.ContainerKeys
+import org.apache.openwhisk.core.service.{DeleteEvent, PutEvent, UnwatchEndpoint, WatchEndpoint, WatchEndpointOperation}
+
+import scala.collection.concurrent.TrieMap
+import scala.concurrent.{ExecutionContext, Future}
+
+class ContainerCounter(invocationNamespace: String, etcdClient: EtcdClient, watcherService: ActorRef)(
+ implicit val actorSystem: ActorSystem,
+ ec: ExecutionContext,
+ logging: Logging) {
+ private[queue] var existingContainerNumByNamespace: Int = 0
+ private[queue] var inProgressContainerNumByNamespace: Int = 0
+ private[queue] val references = new AtomicInteger(0)
+ private val watcherName = s"container-counter-$invocationNamespace"
+
+ private val inProgressContainerPrefixKeyByNamespace =
+ ContainerKeys.inProgressContainerPrefixByNamespace(invocationNamespace)
+ private val existingContainerPrefixKeyByNamespace =
+ ContainerKeys.existingContainersPrefixByNamespace(invocationNamespace)
+
+ private val watchedKeys = Seq(inProgressContainerPrefixKeyByNamespace, existingContainerPrefixKeyByNamespace)
+
+ private val watcher =
+ actorSystem.actorOf(Props(new Actor {
+ private var countingKeys = Set.empty[String]
+ private var waitingForCountKeys = Set.empty[String]
+
+ override def receive: Receive = {
+ case operation: WatchEndpointOperation if operation.isPrefix =>
+ if (countingKeys
+ .contains(operation.watchKey))
+ waitingForCountKeys += operation.watchKey
+ else {
+ countingKeys += operation.watchKey
+ refreshContainerCount(operation.watchKey)
+ }
+
+ case ReadyToGetCount(key) =>
+ if (waitingForCountKeys.contains(key)) {
+ waitingForCountKeys -= key
+ refreshContainerCount(key)
+ } else
+ countingKeys -= key
+ }
+ }))
+
+ private def refreshContainerCount(key: String): Future[Unit] = {
+ etcdClient
+ .getCount(key)
+ .map { count =>
+ key match {
+ case `inProgressContainerPrefixKeyByNamespace` => inProgressContainerNumByNamespace = count.toInt
+ case `existingContainerPrefixKeyByNamespace` => existingContainerNumByNamespace = count.toInt
+ }
+ watcher ! ReadyToGetCount(key)
+ }
+ .recover {
+ case t: Throwable =>
+ logging.error(
+ this,
+ s"failed to get the number of existing containers for ${invocationNamespace} due to ${t}.")
+ watcher ! ReadyToGetCount(key)
+ }
+ }
+
+ def increaseReference(): ContainerCounter = {
+ if (references.incrementAndGet() == 1) {
+ watchedKeys.foreach { key =>
+ watcherService.tell(WatchEndpoint(key, "", true, watcherName, Set(PutEvent, DeleteEvent)), watcher)
+ }
+
+ }
+ this
+ }
+
+ def close(): Unit = {
+ if (references.decrementAndGet() == 0) {
+ watchedKeys.foreach { key =>
+ watcherService ! UnwatchEndpoint(key, true, watcherName)
+ }
+ NamespaceContainerCount.instances.remove(invocationNamespace)
+ }
+ }
+}
+
+object NamespaceContainerCount {
+ private[queue] val instances = TrieMap[String, ContainerCounter]()
+ def apply(namespace: String, etcdClient: EtcdClient, watcherService: ActorRef)(implicit actorSystem: ActorSystem,
+ ec: ExecutionContext,
+ logging: Logging): ContainerCounter = {
+ instances
+ .getOrElseUpdate(namespace, new ContainerCounter(namespace, etcdClient, watcherService))
Review comment:
the **Counter** are shared among queues which under same namespace, to reduce the query burden against etcd
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-io edited a comment on pull request #5072: [New Scheduler] Add container counter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #5072:
URL: https://github.com/apache/openwhisk/pull/5072#issuecomment-787742266
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=h1) Report
> Merging [#5072](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=desc) (c99d1a7) into [master](https://codecov.io/gh/apache/openwhisk/commit/d8cf17247bbcd8c1250873254d0c213fa28116ce?el=desc) (d8cf172) will **decrease** coverage by `6.77%`.
> The diff coverage is `84.21%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #5072 +/- ##
==========================================
- Coverage 81.81% 75.04% -6.78%
==========================================
Files 204 205 +1
Lines 9950 9988 +38
Branches 447 453 +6
==========================================
- Hits 8141 7495 -646
- Misses 1809 2493 +684
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...nwhisk/core/scheduler/queue/ContainerCounter.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvQ29udGFpbmVyQ291bnRlci5zY2FsYQ==) | `84.21% <84.21%> (ø)` | |
| [...core/database/cosmosdb/RxObservableImplicits.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvUnhPYnNlcnZhYmxlSW1wbGljaXRzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ore/database/cosmosdb/cache/CacheInvalidator.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3Iuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...e/database/cosmosdb/cache/ChangeFeedConsumer.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NoYW5nZUZlZWRDb25zdW1lci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...core/database/cosmosdb/CosmosDBArtifactStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJBcnRpZmFjdFN0b3JlLnNjYWxh) | `0.00% <0.00%> (-95.48%)` | :arrow_down: |
| [...sk/core/database/cosmosdb/CosmosDBViewMapper.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJWaWV3TWFwcGVyLnNjYWxh) | `0.00% <0.00%> (-93.90%)` | :arrow_down: |
| [...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3JDb25maWcuc2NhbGE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
| [...enwhisk/connector/kafka/KamonMetricsReporter.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYW1vbk1ldHJpY3NSZXBvcnRlci5zY2FsYQ==) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
| [...e/database/cosmosdb/cache/KafkaEventProducer.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0thZmthRXZlbnRQcm9kdWNlci5zY2FsYQ==) | `0.00% <0.00%> (-78.58%)` | :arrow_down: |
| [...whisk/core/database/cosmosdb/CosmosDBSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJTdXBwb3J0LnNjYWxh) | `0.00% <0.00%> (-74.08%)` | :arrow_down: |
| ... and [17 more](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=footer). Last update [d8cf172...c99d1a7](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 merged pull request #5072: [New Scheduler] Add container counter
Posted by GitBox <gi...@apache.org>.
style95 merged pull request #5072:
URL: https://github.com/apache/openwhisk/pull/5072
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-io commented on pull request #5072: [New Scheduler] Add container counter
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #5072:
URL: https://github.com/apache/openwhisk/pull/5072#issuecomment-787742266
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=h1) Report
> Merging [#5072](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=desc) (692aa7a) into [master](https://codecov.io/gh/apache/openwhisk/commit/d8cf17247bbcd8c1250873254d0c213fa28116ce?el=desc) (d8cf172) will **decrease** coverage by `6.82%`.
> The diff coverage is `84.21%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #5072 +/- ##
==========================================
- Coverage 81.81% 74.98% -6.83%
==========================================
Files 204 205 +1
Lines 9950 9988 +38
Branches 447 453 +6
==========================================
- Hits 8141 7490 -651
- Misses 1809 2498 +689
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...nwhisk/core/scheduler/queue/ContainerCounter.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvQ29udGFpbmVyQ291bnRlci5zY2FsYQ==) | `84.21% <84.21%> (ø)` | |
| [...core/database/cosmosdb/RxObservableImplicits.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvUnhPYnNlcnZhYmxlSW1wbGljaXRzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ore/database/cosmosdb/cache/CacheInvalidator.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3Iuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...e/database/cosmosdb/cache/ChangeFeedConsumer.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NoYW5nZUZlZWRDb25zdW1lci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...core/database/cosmosdb/CosmosDBArtifactStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJBcnRpZmFjdFN0b3JlLnNjYWxh) | `0.00% <0.00%> (-95.48%)` | :arrow_down: |
| [...sk/core/database/cosmosdb/CosmosDBViewMapper.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJWaWV3TWFwcGVyLnNjYWxh) | `0.00% <0.00%> (-93.90%)` | :arrow_down: |
| [...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3JDb25maWcuc2NhbGE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
| [...enwhisk/connector/kafka/KamonMetricsReporter.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYW1vbk1ldHJpY3NSZXBvcnRlci5zY2FsYQ==) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
| [...e/database/cosmosdb/cache/KafkaEventProducer.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0thZmthRXZlbnRQcm9kdWNlci5zY2FsYQ==) | `0.00% <0.00%> (-78.58%)` | :arrow_down: |
| [...whisk/core/database/cosmosdb/CosmosDBSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJTdXBwb3J0LnNjYWxh) | `0.00% <0.00%> (-74.08%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=footer). Last update [d8cf172...692aa7a](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-io edited a comment on pull request #5072: [New Scheduler] Add container counter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #5072:
URL: https://github.com/apache/openwhisk/pull/5072#issuecomment-787742266
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=h1) Report
> Merging [#5072](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=desc) (692aa7a) into [master](https://codecov.io/gh/apache/openwhisk/commit/d8cf17247bbcd8c1250873254d0c213fa28116ce?el=desc) (d8cf172) will **decrease** coverage by `6.77%`.
> The diff coverage is `84.21%`.
[](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #5072 +/- ##
==========================================
- Coverage 81.81% 75.04% -6.78%
==========================================
Files 204 205 +1
Lines 9950 9988 +38
Branches 447 453 +6
==========================================
- Hits 8141 7495 -646
- Misses 1809 2493 +684
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...nwhisk/core/scheduler/queue/ContainerCounter.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvQ29udGFpbmVyQ291bnRlci5zY2FsYQ==) | `84.21% <84.21%> (ø)` | |
| [...core/database/cosmosdb/RxObservableImplicits.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvUnhPYnNlcnZhYmxlSW1wbGljaXRzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ore/database/cosmosdb/cache/CacheInvalidator.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3Iuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...e/database/cosmosdb/cache/ChangeFeedConsumer.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NoYW5nZUZlZWRDb25zdW1lci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...core/database/cosmosdb/CosmosDBArtifactStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJBcnRpZmFjdFN0b3JlLnNjYWxh) | `0.00% <0.00%> (-95.48%)` | :arrow_down: |
| [...sk/core/database/cosmosdb/CosmosDBViewMapper.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJWaWV3TWFwcGVyLnNjYWxh) | `0.00% <0.00%> (-93.90%)` | :arrow_down: |
| [...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3JDb25maWcuc2NhbGE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
| [...enwhisk/connector/kafka/KamonMetricsReporter.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYW1vbk1ldHJpY3NSZXBvcnRlci5zY2FsYQ==) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
| [...e/database/cosmosdb/cache/KafkaEventProducer.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0thZmthRXZlbnRQcm9kdWNlci5zY2FsYQ==) | `0.00% <0.00%> (-78.58%)` | :arrow_down: |
| [...whisk/core/database/cosmosdb/CosmosDBSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJTdXBwb3J0LnNjYWxh) | `0.00% <0.00%> (-74.08%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/apache/openwhisk/pull/5072/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=footer). Last update [d8cf172...692aa7a](https://codecov.io/gh/apache/openwhisk/pull/5072?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] jiangpengcheng commented on a change in pull request #5072: [New Scheduler] Add container counter
Posted by GitBox <gi...@apache.org>.
jiangpengcheng commented on a change in pull request #5072:
URL: https://github.com/apache/openwhisk/pull/5072#discussion_r584465367
##########
File path: core/scheduler/src/main/scala/org/apache/openwhisk/core/scheduler/queue/ContainerCounter.scala
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.openwhisk.core.scheduler.queue
+
+import java.util.concurrent.atomic.AtomicInteger
+
+import akka.actor.{Actor, ActorRef, ActorSystem, Props}
+import org.apache.openwhisk.common.Logging
+import org.apache.openwhisk.core.etcd.EtcdClient
+import org.apache.openwhisk.core.etcd.EtcdKV.ContainerKeys
+import org.apache.openwhisk.core.service.{DeleteEvent, PutEvent, UnwatchEndpoint, WatchEndpoint, WatchEndpointOperation}
+
+import scala.collection.concurrent.TrieMap
+import scala.concurrent.{ExecutionContext, Future}
+
+class ContainerCounter(invocationNamespace: String, etcdClient: EtcdClient, watcherService: ActorRef)(
+ implicit val actorSystem: ActorSystem,
+ ec: ExecutionContext,
+ logging: Logging) {
+ private[queue] var existingContainerNumByNamespace: Int = 0
+ private[queue] var inProgressContainerNumByNamespace: Int = 0
+ private[queue] val references = new AtomicInteger(0)
+ private val watcherName = s"container-counter-$invocationNamespace"
+
+ private val inProgressContainerPrefixKeyByNamespace =
+ ContainerKeys.inProgressContainerPrefixByNamespace(invocationNamespace)
+ private val existingContainerPrefixKeyByNamespace =
+ ContainerKeys.existingContainersPrefixByNamespace(invocationNamespace)
+
+ private val watchedKeys = Seq(inProgressContainerPrefixKeyByNamespace, existingContainerPrefixKeyByNamespace)
+
+ private val watcher =
+ actorSystem.actorOf(Props(new Actor {
+ private var countingKeys = Set.empty[String]
+ private var waitingForCountKeys = Set.empty[String]
+
+ override def receive: Receive = {
+ case operation: WatchEndpointOperation if operation.isPrefix =>
Review comment:
to avoid updating `existingContainerNumByNamespace/inProgressContainerNumByNamespace` in multi threads, it will only query and update `existingContainerNumByNamespace/inProgressContainerNumByNamespace` at one time, during querying and updating, incoming watch events will wait in a `set`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org