You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/07/17 12:13:57 UTC

[GitHub] [superset] mdeshmu opened a new pull request, #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

mdeshmu opened a new pull request, #20733:
URL: https://github.com/apache/superset/pull/20733

   ### SUMMARY
   This PR allows to adjust Gunicorn max_requests and max_requests_jitter which allows to restart the Gunicorn worker process periodically.
   
   This is a simple method to help limit the damage of memory leaks like one reported in [15132](https://github.com/apache/superset/issues/15132). 
   
   For more details, Please refer to: https://docs.gunicorn.org/en/stable/settings.html#max-requests
   
   Default values are set to zero to match gunicorn's default i.e. automatic worker restarts are disabled by default.
   
   Note: This change is non-disruptive, no end user will be impacted unless they turn on settings.
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   
   ![memory1](https://user-images.githubusercontent.com/57723564/179397545-c5a16b87-6588-44a8-aad0-5c15618299cc.png)
   
   
   ### TESTING INSTRUCTIONS
   
   Set the values like: 
   gunicorn --max-requests 1024 --max-requests-jitter 100 ....
   
   ### ADDITIONAL INFORMATION
   - [ ] Has associated issue: [15132](https://github.com/apache/superset/issues/15132)
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] mdeshmu commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

mdeshmu commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1191767780

   I have it enabled for a while now but I don’t see any 504's.
   
   For you, Is it 502 bad gateway or 504 gateway time out?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] low-on-mana commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

low-on-mana commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1193642246

   @mdeshmu Thanks for sharing this, I will check these configuration out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] zhaoyongjie merged pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

zhaoyongjie merged PR #20733:
URL: https://github.com/apache/superset/pull/20733


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] evkuzin commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

evkuzin commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1187577997

   What's the timeline on the docker hub image? How long would it take to release a new image? The one published is broken now. @zhaoyongjie 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] mdeshmu commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

mdeshmu commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1187650356

   @evkuzin updated build is available now. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] z6cr0b3rts commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

z6cr0b3rts commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1186616134

   Seeing this in logs during docker compose (posting in case related): 
   
   `superset_app | gunicorn: error: argument --max-requests: expected one argument`
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] zhaoyongjie commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

zhaoyongjie commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1186707099

   > Seeing this in logs during docker compose (posting in case related):
   > 
   > `superset_app | gunicorn: error: argument --max-requests: expected one argument`
   
   getting default value should be [${parameter:-word}](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html). I will push the fix. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] low-on-mana commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

low-on-mana commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1191860968

   504 gateway timeout @mdeshmu .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] mdeshmu commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

mdeshmu commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1192065667

   @low-on-mana Please make sure you have appropriate jitter parameter value as it is needed to avoid simultaneous restarts of workers. 
   
   Another possibility, Maybe your queries are occasionally taking longer time. Please make sure you are having large enough timeout values at all places.
   
   This is what i have, 
   
   ALB idle timeout (120s) >
   GUNICORN_TIMEOUT (115s) =
   SUPERSET_WEBSERVER_TIMEOUT 
   
   I also have ecs task (gunicorn + superset) directly under alb and was intermittently getting 502 bad gateway because of low keep alive of gunicorn. Then i set it below and all errors were gone.
   
   GUNICORN_KEEPALIVE (125s) >
   ALB idle timeout (120s)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] mdeshmu commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

mdeshmu commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1186500165

   @zhaoyongjie please review and approve. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] mdeshmu commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

mdeshmu commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1186820312

   sorry. my bad, i missed the hyphen. Thanks for reporting @z6cr0b3rts and thanks for the quick fix @zhaoyongjie.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] low-on-mana commented on pull request #20733: fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

Posted by GitBox <gi...@apache.org>.

low-on-mana commented on PR #20733:
URL: https://github.com/apache/superset/pull/20733#issuecomment-1190076062

   We have tried this solution sometime back with custom docker image, but faced lot of intermittent http504s.
   Our setup to serve requests is aws alb -> superset pods in k8.  
   Please share your findings as well @mdeshmu @zhaoyongjie @evkuzin 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org