You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "tenkian4 (via GitHub)" <gi...@apache.org> on 2023/09/13 07:55:45 UTC
[GitHub] [superset] tenkian4 opened a new pull request, #25280: feat(helm): Support HPA for supersetNode and supersetWorker
tenkian4 opened a new pull request, #25280:
URL: https://github.com/apache/superset/pull/25280
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Support auto-scaling the supersetNode and supersetWorker using HPA with CPU/MEM metrics.
When enabled in helm values, will create HPA objects and omit the number of replicas from the deployments.
HPA templates have been derived from the standard templates generated using `helm create`.
The default is disabled.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
HPA Objects are created after the values are configured and feature enabled:
HPA Is reacting to CPU load and increasing replicas:
Pods created
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
Apply with custom values.yaml and set the following options :
```supersetNode:
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80```
The same options can be set for `supersetWorker`
The helm install/upgrade will then create the HPA objects and start monitoring the resource usage and scale replicas.
To simulate CPU load we added a sidecar which generates load as the HPA averages across all containers in the pods.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda merged pull request #25280: feat(helm): Support HPA for supersetNode and supersetWorker
Posted by "craig-rueda (via GitHub)" <gi...@apache.org>.
craig-rueda merged PR #25280:
URL: https://github.com/apache/superset/pull/25280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on pull request #25280: feat(helm): Support HPA for supersetNode and supersetWorker
Posted by "craig-rueda (via GitHub)" <gi...@apache.org>.
craig-rueda commented on PR #25280:
URL: https://github.com/apache/superset/pull/25280#issuecomment-1719683467
I think you're missing newlines at the end of those files as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(helm): Support HPA for supersetNode and supersetWorker [superset]
Posted by "fzhan (via GitHub)" <gi...@apache.org>.
fzhan commented on PR #25280:
URL: https://github.com/apache/superset/pull/25280#issuecomment-2097449602
Seems flask isn't reading the cache correctly when in HPA mode:
HTTPException
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1788, in dispatch_request
self.raise_routing_exception(req)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1770, in raise_routing_exception
raise request.routing_exception # type: ignore
File "/usr/local/lib/python3.10/site-packages/flask/ctx.py", line 351, in match_request
result = self.url_adapter.match(return_rule=True) # type: ignore
File "/usr/local/lib/python3.10/site-packages/werkzeug/routing/map.py", line 624, in match
raise NotFound() from None
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
2024-05-07 04:57:06,714:WARNING:superset.views.base:HTTPException
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1788, in dispatch_request
self.raise_routing_exception(req)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1770, in raise_routing_exception
raise request.routing_exception # type: ignore
File "/usr/local/lib/python3.10/site-packages/flask/ctx.py", line 351, in match_request
result = self.url_adapter.match(return_rule=True) # type: ignore
File "/usr/local/lib/python3.10/site-packages/werkzeug/routing/map.py", line 624, in match
raise NotFound() from None
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] tenkian4 commented on pull request #25280: feat(helm): Support HPA for supersetNode and supersetWorker
Posted by "tenkian4 (via GitHub)" <gi...@apache.org>.
tenkian4 commented on PR #25280:
URL: https://github.com/apache/superset/pull/25280#issuecomment-1720777082
> I think you're missing newlines at the end of those files as well
Added. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] tenkian4 commented on pull request #25280: feat(helm): Support HPA for supersetNode and supersetWorker
Posted by "tenkian4 (via GitHub)" <gi...@apache.org>.
tenkian4 commented on PR #25280:
URL: https://github.com/apache/superset/pull/25280#issuecomment-1718933101
> Thanks for your contribution! Looks like you're missing License info from the new HPA files, but otherwise LGTM.
We have added the licence info copied from one of the other templates to the 2 new files.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org