[GitHub] [openwhisk-package-alarms] machenity opened a new issue #196: [Suggestion] Add delays of a few seconds before firing triggers

[GitBox <gi...@apache.org>]

machenity opened a new issue #196: [Suggestion] Add delays of a few seconds before firing triggers
URL: https://github.com/apache/openwhisk-package-alarms/issues/196
 
 
   I ran 600 alarms with cron `*/1 * * * *` on the provider and the thundering herd arised:
   
   ![스크린샷 2019-08-09 오후 2 11 50](https://user-images.githubusercontent.com/16838736/62931732-cddf9880-bdf9-11e9-9234-865c0f62a3cf.png)
   
   All triggers were fired every minute on the minute, so CPU usage of the provider peaked every minutes. I think it could result in harm on management. To deal with the problem, it is necessary to distribute the firings within every minutes.
   
   I suggest adding a few seconds delay before firing.
   
   In my suggestion, two alterations will be needed in part of parameter and scheduler.
   
   In part of parameter, a new paramter *strict* that takes boolean value is added. 
   
   - If *strict* is true, the alarm will be fired at the reserved time sharp (\*\*:\*\*:00). (same as ever)
   - If false, the alarm will be scheduled with distribution (\*\*:\*\*:00-59).
   
   In part of scheduler, It needs a hash function that takes a name as a key and returns an integer in the range of 0 to 59. When the alarm trigger is created, scheduler injects the value that the hash function returns to `newTrigger.cron`. For example, if the hash function returns 30 and the cron of the trigger is `*/10 * * * *`, the cron will be converted like this: `30 */10 * * * *` . It works because the library `node-cron` which we used supports [sixth field for second](https://github.com/kelektiv/node-cron#available-cron-patterns).
   
   Using hash function can bring some benefits. Because of the characteristics of hash, we can keep the same delay time among hosts or before and after redeployment of providers without using DB or consensus.
   
   I think the level of peaks decreased if the suggestion is implemented, that leads to keep better condition on the machine running providers. Google uses the [similar way](https://queue.acm.org/detail.cfm?id=2745840) to distribute Cron jobs in their systems.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services