You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by "style95 (via GitHub)" <gi...@apache.org> on 2023/08/07 22:23:17 UTC
[GitHub] [openwhisk] style95 opened a new pull request, #5432: Compare invocation namespaces when handling a cycle and recovering a queue
style95 opened a new pull request, #5432:
URL: https://github.com/apache/openwhisk/pull/5432
## Description
This is to fix a bug in that ETCD data for non-existent containers persist forever.
This bug happens when a shared action is invoked.
When the queue manager receives an activation message, it finds a queue based on the `docId` only and this is wrong.
As a result, the activation message could be sent to the wrong queue as long as the `docId` is same.
But the `docId` is same for all shared actions.
ex)
- original action: `/whisk.system/sharedPackage/hello`
- bound action1: `/style95/myPackage/hello` (docId: `/whisk.system/sharedPackage/hello`)
- bound action2: `/bdoyle/yourPackage/hello` (docId: `/whisk.system/sharedPackage/hello`)
So an activation for `/style95/myPackage/hello` could be sent to `/bdoyle/yourPackage/hello`.
Then a memory queue will send the activation to a container for `/style95/myPackage/hello`.
The container is initialized with `/style95/myPackage/hello` and it registers the ETCD data for the running container.
But after executing the activation for `/bdoyle/yourPackage/hello`, it tries to remove the ETCD data for the running container using the key prefix with `/bdoyle/yourPackage/hello` because the key now resides in the container data(`WarmData`).
Accordingly, the original ETCD data for the running container is not deleted forever.
Please refer to the following logs that I found.
(The shared action is `whisk.system/sharedPackage/hello` and I replaced the name of two namespace to `style95` and `bdoyle` from the original logs just to hide proprietary information.)
```
[2023-08-04T16:31:49.392Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [QueueManager] Got activation message 98481e836089419d881e836089d19d00 for Namespace(style95,07e316f4-9393-4777-a1d3-650da22a83f0)/whisk.system/sharedPackage/hello@0.0.38 from kafka.
[2023-08-04T16:31:49.392Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [QueueManager] [98481e836089419d881e836089d19d00] the key whisk/queue/style95/whisk.system/sharedPackage/hello/leader is not in the initRevisionMap. revision: 38-5e7fc51a452c685030fdfcec61e3bdf1
[2023-08-04T16:31:49.392Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [QueueManager] [98481e836089419d881e836089d19d00] send activation to remote queue, key: whisk/queue/style95/whisk.system/sharedPackage/hello/leader revision: 38-5e7fc51a452c685030fdfcec61e3bdf1
[2023-08-04T16:31:49.419Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [QueueManager] add a new actor selection to a map with key: whisk/queue/style95/whisk.system/sharedPackage/hello/leader
[2023-08-04T16:31:49.419Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [QueueManager] Got activation message 98481e836089419d881e836089d19d00 for Namespace(style95,07e316f4-9393-4777-a1d3-650da22a83f0)/whisk.system/sharedPackage/hello@0.0.38 from remote queue manager.
[2023-08-04T16:31:49.419Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [QueueManager] Queue for action whisk.system/sharedPackage/hello@0.0.38 is already recovered, skip
[2023-08-04T16:31:49.419Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [MemoryQueue] [bdoyle:whisk.system/sharedPackage/hello@0.0.38:Running] got a new activation message 98481e836089419d881e836089d19d00
```
As you can see above the activation(`98481e836089419d881e836089d19d00`) is originally for `style95` but it is finally sent to the queue for `bdoyle`.
And the queue sent this activation to a container with an ID(`f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff`).
```
[2023-08-04T16:31:49.686Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [MemoryQueue] [bdoyle:whisk.system/sharedPackage/hello@0.0.38:Running] Get activation request f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff, send one message: 98481e836089419d881e836089d19d00
```
This container was originally created for `bdoyle`.
```
[2023-08-04T16:31:49.623Z] [INFO] [#tid_sid_unknown] [FunctionPullingContainerPool] received a container creation message: f3dc23dbb34c43889c23dbb34c93882e
[2023-08-04T16:31:49.683Z] [INFO] [#tid_sid_unknown] [WatcherService] watch endpoint: WatchEndpoint(whisk/namespace/bdoyle/whisk.system/sharedPackage/hello/38-5e7fc51a452c685030fdfcec61e3bdf1/invoker0/container/f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff,,false,data-management-service,Set(DeleteEvent))
[2023-08-04T16:31:49.685Z] [INFO] [#tid_sid_unknown] [FPCInvokerReactive] Posted success ack of container creation f3dc23dbb34c43889c23dbb34c93882e for bdoyle/whisk.system/sharedPackage/hello@0.0.38
```
But when the same container(`f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff`) is being paused, it tries to unwatch endpoint based on the `style95` key because it executed an activation for `style95` namespace.
```
[2023-08-04T16:31:49.688Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [FunctionPullingContainerProxy] received a message 98481e836089419d881e836089d19d00 for whisk.system/sharedPackage/hello@0.0.38 in ClientCreated
[2023-08-04T16:31:49.688Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [DockerContainer] sending initialization to ContainerId(f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff) ContainerAddress(172.17.0.34,8080)
[2023-08-04T16:31:49.888Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [DockerContainer] initialization result: ok
[2023-08-04T16:31:49.888Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [DockerContainer] sending arguments to /whisk.system/sharedPackage/hello at ContainerId(f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff) ContainerAddress(172.17.0.34,8080)
[2023-08-04T16:31:49.992Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [DockerContainer] running result: ok
[2023-08-04T16:31:49.992Z] [INFO] [#tid_5de45169ee33b5678a8067431ba09ca8] [MessagingActiveAck] posted completion of activation 98481e836089419d881e836089d19d00
[2023-08-04T16:32:00.007Z] [INFO] [#tid_sid_unknown] [FunctionPullingContainerProxy] No more run activation is coming in state: Running, action: ExecutableWhiskAction/whisk.system/sharedPackage/hello@0.0.38, container: ContainerId(f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff)
[2023-08-04T16:32:00.472Z] [INFO] [#tid_sid_invokerNanny] [RuncClient] running /usr/bin/docker-runc pause f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff (timeout: 10 seconds)
[2023-08-04T16:32:00.544Z] [INFO] [#tid_sid_unknown] [WatcherService] watch endpoint: WatchEndpoint(whisk/warmed/style95/whisk.system/sharedPackage/hello/38-5e7fc51a452c685030fdfcec61e3bdf1/invoker/0/container/f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff,,false,data-management-service,Set(DeleteEvent))
[2023-08-04T16:32:00.544Z] [INFO] [#tid_sid_unknown] [WatcherService] unwatch endpoint: UnwatchEndpoint(whisk/namespace/style95/whisk.system/sharedPackage/hello/38-5e7fc51a452c685030fdfcec61e3bdf1/invoker0/container/f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff,false,data-management-service,true)
```
Consequently, the data for `bdoyle`(`whisk/namespace/bdoyle/whisk.system/sharedPackage/hello/38-5e7fc51a452c685030fdfcec61e3bdf1/invoker0/container/f372eb3f960da72c4bff75110d24b0c5b72d6d306d3c31e59f4c10f90ccefdff`) is not removed forever unless the invoker is restarted.
## Related issue and scope
<!--- Please include a link to a related issue if there is one. -->
- [ ] I opened an issue to propose and discuss this change (#????)
## My changes affect the following components
<!--- Select below all system components are affected by your change. -->
<!--- Enter an `x` in all applicable boxes. -->
- [ ] API
- [ ] Controller
- [ ] Message Bus (e.g., Kafka)
- [ ] Loadbalancer
- [x] Scheduler
- [x] Invoker
- [ ] Intrinsic actions (e.g., sequences, conductors)
- [ ] Data stores (e.g., CouchDB)
- [ ] Tests
- [ ] Deployment
- [ ] CLI
- [ ] General tooling
- [ ] Documentation
## Types of changes
<!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
- [x] Bug fix (generally a non-breaking change which closes an issue).
- [ ] Enhancement or new feature (adds new functionality).
- [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
## Checklist:
<!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
- [x] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
- [x] I reviewed the [style guides](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md#coding-standards) and followed the recommendations (Travis CI will check :).
- [ ] I added tests to cover my changes.
- [ ] My changes require further changes to the documentation.
- [ ] I updated the documentation where necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "bdoyle0182 (via GitHub)" <gi...@apache.org>.
bdoyle0182 commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1669670507
I'm not sure why the multiruntime tests are failing now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 merged pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "style95 (via GitHub)" <gi...@apache.org>.
style95 merged PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "bdoyle0182 (via GitHub)" <gi...@apache.org>.
bdoyle0182 commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1668656637
Is there anything that can be unit tested here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "style95 (via GitHub)" <gi...@apache.org>.
style95 commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1676111477
As I shared via the dev list, this change is urgent and not related to anything about multi-runtime tests, I would merge this PR without passing the tests in 24 hours unless there is any objection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "bdoyle0182 (via GitHub)" <gi...@apache.org>.
bdoyle0182 commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1668675872
> > Is there anything that can be unit tested here?
>
> I tried to add a unit test but it seems it's not easy to reproduce the situation based on a unit test.
LGTM then. This is a pretty critical bug since an activation can be delivered to a wrong container so should get this merged as quickly as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "style95 (via GitHub)" <gi...@apache.org>.
style95 commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1668928141
It seems there is a problem in multi-runtime tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1668670055
## [Codecov](https://app.codecov.io/gh/apache/openwhisk/pull/5432?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#5432](https://app.codecov.io/gh/apache/openwhisk/pull/5432?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (791d064) into [master](https://app.codecov.io/gh/apache/openwhisk/commit/54564cbc76771658c14a84dd413856d8b078eeb8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (54564cb) will **decrease** coverage by `72.31%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head 791d064 differs from pull request most recent head 18fb70a. Consider uploading reports for the commit 18fb70a to get more accurate results
```diff
@@ Coverage Diff @@
## master #5432 +/- ##
==========================================
- Coverage 76.81% 4.51% -72.31%
==========================================
Files 241 241
Lines 14630 14632 +2
Branches 616 616
==========================================
- Hits 11238 660 -10578
- Misses 3392 13972 +10580
```
| [Files Changed](https://app.codecov.io/gh/apache/openwhisk/pull/5432?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../openwhisk/core/scheduler/queue/QueueManager.scala](https://app.codecov.io/gh/apache/openwhisk/pull/5432?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvUXVldWVNYW5hZ2VyLnNjYWxh) | `0.00% <0.00%> (-81.51%)` | :arrow_down: |
... and [205 files with indirect coverage changes](https://app.codecov.io/gh/apache/openwhisk/pull/5432/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] rabbah commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "rabbah (via GitHub)" <gi...@apache.org>.
rabbah commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1676134145
Change lgtm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "style95 (via GitHub)" <gi...@apache.org>.
style95 commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1668663957
> Is there anything that can be unit tested here?
I tried to add a unit test but it seems it's not easy to reproduce the situation based on a unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5432: Compare invocation namespaces when handling a cycle and recovering a queue
Posted by "bdoyle0182 (via GitHub)" <gi...@apache.org>.
bdoyle0182 commented on PR #5432:
URL: https://github.com/apache/openwhisk/pull/5432#issuecomment-1668656084
Huge! This was the only remaining bug I had identified from my use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org