You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2022/04/22 04:09:11 UTC
[GitHub] [openwhisk] JesseStutler opened a new pull request, #5221: Change the value of pause-grace for new scheduler
JesseStutler opened a new pull request, #5221:
URL: https://github.com/apache/openwhisk/pull/5221
<!--- Provide a concise summary of your changes in the Title -->
The default value of pause-grace, which is 50 milliseconds by default, is too short for FPCScheduler. If requests' time intervals are longer than 50ms, the container will be paused and can't pull requests from MemoryQueue, doesn't take advantage of the ActivationProxy-ActivationService path, which is the core of FPCScheduler design. So we need to set the pause-grace to a longer value, 10s by default now.
## Description
<!--- Provide a detailed description of your changes. -->
<!--- Include details of what problem you are solving and how your changes are tested. -->
To prove this, I did a test for FPCScheduler. I sent 5 concurrent requests every 5 seconds, the second 5 requests would be successfully picked up by GetActivaion in MemoryQueue. But for subsequent concurrent requests, the SchedulingDecisionMaker would keep matching an exceptional case:
```
// this is an exceptional case, create a container immediately
case (Running, _) if totalContainers == 0 && availableMsg > 0 =>
logging.info(
this,
s"add one container if totalContainers($totalContainers) == 0 && availableMsg($availableMsg) > 0 [$invocationNamespace:$action]")
Future.successful(DecisionResults(AddContainer, 1))
```
It was because MemoryQueue told him that there were no warm containers, but actually, there were some containers already spawned for previous requests, just in paused state, and couldn't pull requests temporarily. And then SchedulingDecisonMaker tried to send container creation message to the invoker, invoker didn't create any container because there are already some warm containers, causing some latencies.
## My changes affect the following components
<!--- Select below all system components are affected by your change. -->
<!--- Enter an `x` in all applicable boxes. -->
- [ ] API
- [ ] Controller
- [ ] Message Bus (e.g., Kafka)
- [ ] Loadbalancer
- [ ] Scheduler
- [x] Invoker
- [ ] Intrinsic actions (e.g., sequences, conductors)
- [ ] Data stores (e.g., CouchDB)
- [ ] Tests
- [ ] Deployment
- [ ] CLI
- [ ] General tooling
- [x] Documentation
## Types of changes
<!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
- [ ] Bug fix (generally a non-breaking change which closes an issue).
- [ ] Enhancement or new feature (adds new functionality).
- [x] Breaking change (a bug fix or enhancement which changes existing behavior).
## Checklist:
<!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
- [x] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
- [x] I reviewed the [style guides](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md#coding-standards) and followed the recommendations (Travis CI will check :).
- [N/A] I added tests to cover my changes.
- [N/A] My changes require further changes to the documentation.
- [x] I updated the documentation where necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1183030021
sorry, long time not to check, I will try to solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1106073906
Actually, I take @style95 's opinion that in their internal system they set 10 seconds as default, but for k8s environment, ow does not pause containers. I can get the point that, in old scheduler, the warm container can be paused for saving resources. For subsequent requests, the invoker can resume the container. But for FPCScheduler, it seems that there is no need to pause the container? Otherwise we can't take advantage of ActivationClientProxy-ActivaionService path if the time interval is longer than any preset pause-grace. But on the other hand, warm containers will waste resources if they keep running and don't handle any requests.
Or under paused state containers also need to send get activation requests? What are your suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on a diff in pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on code in PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#discussion_r855803531
##########
core/invoker/src/main/resources/application.conf:
##########
@@ -154,7 +154,7 @@ whisk {
# The "unusedTimeout" in the ContainerProxy,
#aka 'How long should a container sit idle until we kill it?'
idle-container = 10 minutes
- pause-grace = 50 milliseconds
+ pause-grace = 10 seconds
Review Comment:
> This will affect the old scheduler too. It looks fine to me as I think each downstream would use its own timeout. But worth listening to others' opinions.
Yes I realize later that I have changed application.conf as well, supposed to only change `ansible/README.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1106858499
Important thing to note about the pause grace value is that the container eviction time is pauseGrace + idleTimeout at least in the old scheduler so increasing the pause grace value dramatically will also increase the time that containers sit around unless that behavior is decoupled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1107359719
> Since each downstream can freely change the default while deploying OW in production(and I believe they are already doing it), I think we can increase the default.
I think I need hot configuration swapping so much...Each time I change the resources like `application.conf` I need to rebuild a image and redeploy it. Don't know if ow can achieve like K8S's configmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on a diff in pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on code in PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#discussion_r920753107
##########
ansible/README.md:
##########
@@ -177,6 +177,23 @@ whisk.spi {
.
.
```
+#### Configure pause grace for the scheduler
+Set the value of pause-grace to 10s by default
+
+**core/invoker/src/main/resources/application.conf**
+```
+ container-proxy {
+ timeouts {
+ # The "unusedTimeout" in the ContainerProxy,
+ #aka 'How long should a container sit idle until we kill it?'
+ idle-container = 10 minutes
+ pause-grace = 10 seconds
+ keeping-duration = 1 second
Review Comment:
Could you update this to 10 minutes just like the default value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1120542535
@JesseStutler Could you rebase and resolve the conflicts?
> I think I need hot configuration swapping so much...Each time I change the resources like application.conf I need to rebuild an image and redeploy it. Don't know if ow can achieve like K8S's configmap.
This is possible but I see there is no corresponding configuration available for `pauseGrace`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1195011539
@JesseStutler
Thanks for your effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1107359298
I agree that there is no a priori default value for pause-grace. Maybe some developers will concern about waitTime, so they need a bigger pause-grace, even not to pause, so that unpause or pause introduced latency overhead will be reduced. Maybe some developers will concern about unpausing containers will compete for resources and influence other containers' behaviors or waste resources for VMs, so they may need to pause it. But for now, I think pause-grace is at odds with FPCScheduler's design...
BTW, as @bdoyle0182 's saying, pause-grace is low is most for billing reasons. Can't ow charge only in activation handling cycles? No requests no charge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on a diff in pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on code in PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#discussion_r855789446
##########
core/invoker/src/main/resources/application.conf:
##########
@@ -154,7 +154,7 @@ whisk {
# The "unusedTimeout" in the ContainerProxy,
#aka 'How long should a container sit idle until we kill it?'
idle-container = 10 minutes
- pause-grace = 50 milliseconds
+ pause-grace = 10 seconds
Review Comment:
This will affect the old scheduler too.
It looks fine to me as I think each downstream would use its own timeout.
But worth listening to others' opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1107151487
> Important thing to note about the pause grace value is that the container eviction time is pauseGrace + idleTimeout at least in the old scheduler so increasing the pause grace value dramatically will also increase the time that containers sit around unless that behavior is decoupled
Exactly. The value I see in this PR is to improve the development experience by increasing the default pause-grace.
I think the default value is mostly used when people give it a shot with OW in their local.
They would test each feature, and try to run some actions by hand.
So even if they frequently invoke actions containers would be easily paused in turn the waitTime would increase.
Since each downstream can freely change the default while deploying OW in production(and I believe they are already doing it), I think we can increase the default.
Too small pause-grace is not well aligned with one of the main ideas behind the FPCScheduler and even it could harm the latency in the old scheduler too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on a diff in pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on code in PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#discussion_r855790879
##########
core/invoker/src/main/resources/application.conf:
##########
@@ -154,7 +154,7 @@ whisk {
# The "unusedTimeout" in the ContainerProxy,
#aka 'How long should a container sit idle until we kill it?'
idle-container = 10 minutes
- pause-grace = 50 milliseconds
+ pause-grace = 10 seconds
Review Comment:
I'm personally pro larger pause grace anyways as the default, but maybe needs to be communicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1130943246
It's strange that CI tests are failing because of the swaggerGen task failure.
I retriggered the build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1124480003
Hi @style95, I have already resolved the conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1186399018
Done. It seems to be merged quickly, I found that the master has been modified a lot recently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1179519261
@JesseStutler Could you rebase this to pass all checks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1122072799
I hope the default pause-grace is increased.
Each downstream still can configure its own value for it.
IMHO, keeping warmed containers alive is just for performance improvement and if there is no activation running, users should not be charged.
Maybe one thing we can do is to make it configurable without rebuilding the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1183882619
@style95 Hi, check has passed, please review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on a diff in pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on code in PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#discussion_r922272361
##########
ansible/README.md:
##########
@@ -177,6 +177,23 @@ whisk.spi {
.
.
```
+#### Configure pause grace for the scheduler
+Set the value of pause-grace to 10s by default
+
+**core/invoker/src/main/resources/application.conf**
+```
+ container-proxy {
+ timeouts {
+ # The "unusedTimeout" in the ContainerProxy,
+ #aka 'How long should a container sit idle until we kill it?'
+ idle-container = 10 minutes
+ pause-grace = 10 seconds
+ keeping-duration = 1 second
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1188312593
fine with me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] codecov-commenter commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1119784288
# [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5221?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5221](https://codecov.io/gh/apache/openwhisk/pull/5221?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d95cb7a) into [master](https://codecov.io/gh/apache/openwhisk/commit/3e3414ced7fb6e6d85b95f0f9cdd43a0a68da4fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3e3414c) will **decrease** coverage by `39.98%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #5221 +/- ##
==========================================
- Coverage 44.53% 4.54% -39.99%
==========================================
Files 238 238
Lines 13957 13957
Branches 570 570
==========================================
- Hits 6216 635 -5581
- Misses 7741 13322 +5581
```
| [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5221?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../main/scala/org/apache/openwhisk/core/WarmUp.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvV2FybVVwLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ain/scala/org/apache/openwhisk/spi/SpiLoader.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL3NwaS9TcGlMb2FkZXIuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...scala/org/apache/openwhisk/core/FeatureFlags.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvRmVhdHVyZUZsYWdzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...la/org/apache/openwhisk/http/BasicRasService.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvQmFzaWNSYXNTZXJ2aWNlLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/openwhisk/http/LenientSprayJsonSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvTGVuaWVudFNwcmF5SnNvblN1cHBvcnQuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ache/openwhisk/core/database/DocumentFactory.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvRG9jdW1lbnRGYWN0b3J5LnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...che/openwhisk/core/invoker/LogStoreCollector.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9Mb2dTdG9yZUNvbGxlY3Rvci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/openwhisk/core/invoker/NamespaceBlacklist.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9OYW1lc3BhY2VCbGFja2xpc3Quc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../openwhisk/core/entity/ActivationEntityLimit.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZW50aXR5L0FjdGl2YXRpb25FbnRpdHlMaW1pdC5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...openwhisk/core/database/ActivationStoreLevel.scala](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvQWN0aXZhdGlvblN0b3JlTGV2ZWwuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [118 more](https://codecov.io/gh/apache/openwhisk/pull/5221/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5221?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5221?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3e3414c...d95cb7a](https://codecov.io/gh/apache/openwhisk/pull/5221?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1106074614
>
Let me share the details.
In FPCScheduling, each ContainerProxy keeps sending `FetchRequest` to schedulers to pull activations.
When there is no activation for `pause-grace`, the container would be paused in turn it can't send the `FetchRequest`.
Warmed containers are still counted in the scheduler but they cannot send the request accordingly no activation is pulled.
After then, once a new activation comes to an action queue, it would try to add a new container because there is no running container.
And at this point, it will try to reuse warmed(paused) containers first.
If there are warmed containers in invokers, they are chosen first.
Since the data plane and control plane are separated in FPCScheduler, frequent pausing can aggravate the performance.
It's same in the old scheduler too but in FPCScheduler there is one more step for containers to send FetchRequest so latency could be a little bit more harmed.
And I feel it's a kind of tradeoff for the VM environment. (In K8S, no container is paused.)
No matter which value we choose, if the incoming interval for activations is a little bigger than that, the action will suffer from this. (This paper addressed this well: [Serverless in the wild](https://www.usenix.org/conference/atc20/presentation/shahrad), maybe we can consider taking a similar approach)
Anyway, one thing we can see is a bigger value will prevent frequent actions to be suffered.
Also, let's say unpausing takes around 200~500ms, if the interval is 1s, it's 20~50% overhead of the interval.
But if the interval is 1 min or 1 hour, 200~500ms overhead might be acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] JesseStutler commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
JesseStutler commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1131061376
Oh Failed again? What was the reason? I don't change any code but only the configuration files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 merged pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 merged PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
style95 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1156381504
It requires another rebase to include https://github.com/apache/openwhisk/commit/d88a7d2be8399bea7e55910ed225777d90d8fb9f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1106040642
LGTM but if the container can't pull new activations once it's paused from the scheduler so it appears like there's no available warm containers, what's the point of keeping the container once it is paused with the default pause now being 10 seconds and the default container idle eviction being 10 minutes for the new scheduler. Or am I missing something and there's another mechanism that unpauses the container from the scheduler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] bdoyle0182 commented on pull request #5221: Change the value of pause-grace for new scheduler
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on PR #5221:
URL: https://github.com/apache/openwhisk/pull/5221#issuecomment-1106747287
I agree pause grace should be higher if it's a high percentage of overhead. I asked about this recently in slack and it sounded like the only reason it was low was for the billing cycles being 100ms so public users would not get free compute from async processes after their function completes, but as a default I don't see that as necessary
And understand totally now it works the same as the old scheduler where the scheduler makes a request to the invoker which the invoker proxy will get and unpause the container, just the one additional step of the unpaused container then having to retrieve the activation from the scheduler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org