You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2022/09/12 21:51:57 UTC
[GitHub] [openwhisk] bdoyle0182 opened a new pull request, #5323: Clean Up Etcd Worker Actor
bdoyle0182 opened a new pull request, #5323:
URL: https://github.com/apache/openwhisk/pull/5323
## Description
I don't think this is the root of my issue with the scheduler not learning that a container has been removed, but nevertheless this is a valuable refactor to make sure everything is thread safe in this actor.
There should be no behavior change to the actor at all, should be 100% refactor.
This pr does a couple things:
1. Move etcd worker to its own file under the etcd folder in commons
2. Clean up the error handling to remove duplicate code
3. Make the retry handling more thread safe by first forwarding the retry message and using the actor's internal timers scheduler rather than the system scheduler
4. Fix non-thread safe access to actor member variable `lease` from future callbacks through 3.
5. Add unit tests for EtcdWorker which previously wasn't covered.
## Related issue and scope
<!--- Please include a link to a related issue if there is one. -->
- [ ] I opened an issue to propose and discuss this change (#????)
## My changes affect the following components
- [ ] API
- [ ] Controller
- [ ] Message Bus (e.g., Kafka)
- [ ] Loadbalancer
- [X] Scheduler
- [X] Invoker
- [ ] Intrinsic actions (e.g., sequences, conductors)
- [ ] Data stores (e.g., CouchDB)
- [ ] Tests
- [ ] Deployment
- [ ] CLI
- [ ] General tooling
- [ ] Documentation
## Types of changes
- [X] Bug fix (generally a non-breaking change which closes an issue).
- [ ] Enhancement or new feature (adds new functionality).
- [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
## Checklist:
- [X] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
- [X] I reviewed the [style guides](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md#coding-standards) and followed the recommendations (Travis CI will check :).
- [X] I added tests to cover my changes.
- [ ] My changes require further changes to the documentation.
- [ ] I updated the documentation where necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on a diff in pull request #5323: Clean Up Etcd Worker Actor
Posted by GitBox <gi...@apache.org>.
style95 commented on code in PR #5323:
URL: https://github.com/apache/openwhisk/pull/5323#discussion_r969287893
##########
tests/src/test/scala/org/apache/openwhisk/common/etcd/EtcdWorkerTests.scala:
##########
@@ -0,0 +1,176 @@
+package org.apache.openwhisk.common.etcd
+
+import akka.actor.{ActorRef, ActorSystem}
+import akka.testkit.{ImplicitSender, TestActor, TestActorRef, TestKit, TestProbe}
+import akka.util.Timeout
+import com.ibm.etcd.api.{DeleteRangeResponse, PutResponse, TxnResponse}
+import common.StreamLogging
+import io.grpc.{Status, StatusRuntimeException}
+import org.apache.openwhisk.core.entity.SchedulerInstanceId
+import org.apache.openwhisk.core.etcd.{EtcdClient, EtcdLeader, EtcdWorker}
+import org.apache.openwhisk.core.service.{
+ AlreadyExist,
+ Done,
+ ElectLeader,
+ ElectionResult,
+ FinishWork,
+ GetLease,
+ InitialDataStorageResults,
+ Lease,
+ RegisterData,
+ RegisterInitialData,
+ WatcherClosed
+}
+import org.junit.runner.RunWith
+import org.scalamock.scalatest.MockFactory
+import org.scalatest.{BeforeAndAfterAll, FlatSpecLike, Matchers}
+import org.scalatest.concurrent.ScalaFutures
+import org.scalatest.junit.JUnitRunner
+
+import scala.concurrent.{ExecutionContext, Future}
+import scala.concurrent.duration._
+
+@RunWith(classOf[JUnitRunner])
+class EtcdWorkerTests
+ extends TestKit(ActorSystem("EtcdWorker"))
+ with ImplicitSender
+ with FlatSpecLike
+ with ScalaFutures
+ with Matchers
+ with MockFactory
+ with BeforeAndAfterAll
+ with StreamLogging {
+
+ implicit val timeout: Timeout = Timeout(5.seconds)
+ implicit val ec: ExecutionContext = system.dispatcher
+ val leaseService = TestProbe()
+ val leaseId = 10
+ val leaseTtl = 10
+ leaseService.setAutoPilot((sender: ActorRef, msg: Any) =>
+ msg match {
+ case GetLease =>
+ sender ! Lease(leaseId, leaseTtl)
+ TestActor.KeepRunning
+
+ case _ =>
+ TestActor.KeepRunning
+ })
+
+ //val dataManagementService = TestProbe()
+ val schedulerId = SchedulerInstanceId("scheduler0")
+ val instanceId = schedulerId
+
+ behavior of "EtcdWorker"
+
+ it should "elect leader and send completion ack to parent" in {
+ val mockEtcd = mock[EtcdClient]
+
+ val key = "testKey"
+ val value = "testValue"
+ val leader = Right(EtcdLeader(key, value, leaseId))
+ val etcdWorker = TestActorRef(EtcdWorker.props(mockEtcd, leaseService.ref), self)
+
+ (mockEtcd
+ .electLeader(_: String, _: String, _: Lease))
+ .expects(key, value, *)
+ .returns(Future.successful(leader))
+
+ etcdWorker ! ElectLeader(key, value, recipient = self)
+
+ expectMsg(ElectionResult(leader))
+ expectMsg(FinishWork(key))
+ }
+
+ it should "register initial data when doesn't exit and send completion ack to parent" in {
+ val mockEtcd = mock[EtcdClient]
+
+ val key = "testKey"
+ val value = "testValue"
+ val etcdWorker = TestActorRef(EtcdWorker.props(mockEtcd, leaseService.ref), self)
+
+ (mockEtcd
+ .putTxn(_: String, _: String, _: Long, _: Long))
+ .expects(key, value, *, *)
+ .returns(Future.successful(TxnResponse.newBuilder().setSucceeded(true).build()))
+
+ etcdWorker ! RegisterInitialData(key, value, recipient = Some(self))
+
+ expectMsg(FinishWork(key))
+ expectMsg(InitialDataStorageResults(key, Right(Done())))
+ }
+
+ it should "attempt to register initial data when exists and send completion ack to parent" in {
+ val mockEtcd = mock[EtcdClient]
+
+ val key = "testKey"
+ val value = "testValue"
+ val etcdWorker = TestActorRef(EtcdWorker.props(mockEtcd, leaseService.ref), self)
+
+ (mockEtcd
+ .putTxn(_: String, _: String, _: Long, _: Long))
+ .expects(key, value, *, *)
+ .returns(Future.successful(TxnResponse.newBuilder().setSucceeded(false).build()))
+
+ etcdWorker ! RegisterInitialData(key, value, recipient = Some(self))
+
+ expectMsg(FinishWork(key))
+ expectMsg(InitialDataStorageResults(key, Left(AlreadyExist())))
+ }
+
+ it should "register data and send completion ack to parent" in {
+ val mockEtcd = mock[EtcdClient]
+
+ val key = "testKey"
+ val value = "testValue"
+ val etcdWorker = TestActorRef(EtcdWorker.props(mockEtcd, leaseService.ref), self)
+
+ (mockEtcd
+ .put(_: String, _: String, _: Long))
+ .expects(key, value, leaseId)
+ .returns(Future.successful(PutResponse.newBuilder().build()))
+
+ etcdWorker ! RegisterData(key, value)
+
+ expectMsg(FinishWork(key))
+ }
+
+ it should "delete data when watcher closed" in {
+ val mockEtcd = mock[EtcdClient]
+
+ val key = "testKey"
+ val etcdWorker = TestActorRef(EtcdWorker.props(mockEtcd, leaseService.ref), self)
+
+ (mockEtcd
+ .del(_: String))
+ .expects(key)
+ .returns(Future.successful(DeleteRangeResponse.newBuilder().build()))
+
+ etcdWorker ! WatcherClosed(key, false)
+
+ expectMsg(FinishWork(key))
+ }
+
+ it should "retry request after failure if lease does not exist" in {
Review Comment:
Can we also add other negative tests for `RegisterData` and `RegisterInitialData`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter commented on pull request #5323: Clean Up Etcd Worker Actor
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #5323:
URL: https://github.com/apache/openwhisk/pull/5323#issuecomment-1244661804
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5323?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5323](https://codecov.io/gh/apache/openwhisk/pull/5323?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2268f1a) into [master](https://codecov.io/gh/apache/openwhisk/commit/138f3d9022610b9d00078db5226e3fd4ec67d64a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (138f3d9) will **decrease** coverage by `76.46%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #5323 +/- ##
==========================================
- Coverage 81.03% 4.56% -76.47%
==========================================
Files 239 240 +1
Lines 14245 14242 -3
Branches 594 616 +22
==========================================
- Hits 11543 650 -10893
- Misses 2702 13592 +10890
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/openwhisk/core/etcd/EtcdWorker.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZXRjZC9FdGNkV29ya2VyLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...openwhisk/core/service/DataManagementService.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvc2VydmljZS9EYXRhTWFuYWdlbWVudFNlcnZpY2Uuc2NhbGE=) | `0.00% <ø> (-44.28%)` | :arrow_down: |
| [...he/openwhisk/core/invoker/FPCInvokerReactive.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9GUENJbnZva2VyUmVhY3RpdmUuc2NhbGE=) | `0.00% <ø> (ø)` | |
| [...rg/apache/openwhisk/core/scheduler/Scheduler.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvU2NoZWR1bGVyLnNjYWxh) | `0.00% <ø> (-10.39%)` | :arrow_down: |
| [.../main/scala/org/apache/openwhisk/core/WarmUp.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2FybVVwLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ain/scala/org/apache/openwhisk/spi/SpiLoader.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL3NwaS9TcGlMb2FkZXIuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...n/scala/org/apache/openwhisk/utils/JsHelpers.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL3V0aWxzL0pzSGVscGVycy5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...scala/org/apache/openwhisk/common/Prometheus.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi9Qcm9tZXRoZXVzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...scala/org/apache/openwhisk/common/time/Clock.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvbW1vbi90aW1lL0Nsb2NrLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...scala/org/apache/openwhisk/core/FeatureFlags.scala](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvRmVhdHVyZUZsYWdzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [207 more](https://codecov.io/gh/apache/openwhisk/pull/5323/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org