You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "stelsemeyer-m60 (via GitHub)" <gi...@apache.org> on 2023/07/16 03:57:09 UTC
[GitHub] [airflow] stelsemeyer-m60 opened a new pull request, #32631: Poll interval improvement
stelsemeyer-m60 opened a new pull request, #32631:
URL: https://github.com/apache/airflow/pull/32631
Parametrize the interval in which the Kubernetes pod status is polled when launching a new pod using the already existing `poll_interval` parameter.
When using serverless Kubernetes services like Google GKE Autopilot the pod startup time is sometimes expected to be longer due to a cold start. Therefore the logs might be spammed due to the default checks every second (see below), and a lower check frequency might be desired
```
[2023-05-02, 05:33:22 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:23 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:24 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:25 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:26 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:27 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:28 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:29 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:30 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:31 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:32 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:33 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:34 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:35 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:36 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
[2023-05-02, 05:33:37 UTC] {pod_manager.py:187} WARNING - Pod not yet started: some-pod-he2j8139
...
```
This addresses comments in https://github.com/apache/airflow/pull/31008 (having duplicated logic, not fully functional new implementation).
cc @jedcunningham @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on pull request #32631: Parametrize poll_interval in KubernetesPodOperator
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #32631:
URL: https://github.com/apache/airflow/pull/32631#issuecomment-1668917262
@stelsemeyer-m60 Why was this closed? Is it not applicable anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #32631: Parametrize poll_interval in KubernetesPodOperator
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #32631:
URL: https://github.com/apache/airflow/pull/32631#issuecomment-1670990563
> > I see. The PR was closed due to no activity.
> > How do you plan to follow up on it?
>
> Would be great to get feedback on my last reply (in that PR). Otherwise I can also make an educated guess/decision. My time is limited at the moment, but I still plan to finish that PR.
Small uggestion - in cae it is not clear @stelsemeyer-m60: I suggest (and our CONTRIBUTING docs already explain that) to regular (but not too frequent - use your own judgment taking into account that people look at those in their private time and can be busy/on holidays etc.) gently pinginggwhen you wait for answer.
It's SUPER easy to miss someone's question or explanation, and when you take into accounts that we merge 30-40 PRs a day and have 150 opened PRs and many more issues commented on, the fact that you asked a question might be easily missed. If you look at this from the other side - you have one PR that you care about and maintainers have at the same time 100s of conversations going on, it's quite a bit obvious that it's on the side of author to make sure to follow up if they want to make things happen :). This also shows that you do, actually care, and have not abandoned it (which happens more often than you think).
Just a friendly tip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stelsemeyer-m60 closed pull request #32631: Parametrize poll_interval in KubernetesPodOperator
Posted by "stelsemeyer-m60 (via GitHub)" <gi...@apache.org>.
stelsemeyer-m60 closed pull request #32631: Parametrize poll_interval in KubernetesPodOperator
URL: https://github.com/apache/airflow/pull/32631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stelsemeyer-m60 commented on pull request #32631: Parametrize poll_interval in KubernetesPodOperator
Posted by "stelsemeyer-m60 (via GitHub)" <gi...@apache.org>.
stelsemeyer-m60 commented on PR #32631:
URL: https://github.com/apache/airflow/pull/32631#issuecomment-1670637401
> I see. The PR was closed due to no activity.
>
> How do you plan to follow up on it?
Would be great to get feedback on my last reply. Otherwise I can also make an educated guess/decision. My time is limited at the moment, but I still plan to finish that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stelsemeyer-m60 commented on pull request #32631: Parametrize poll_interval in KubernetesPodOperator
Posted by "stelsemeyer-m60 (via GitHub)" <gi...@apache.org>.
stelsemeyer-m60 commented on PR #32631:
URL: https://github.com/apache/airflow/pull/32631#issuecomment-1670476439
> @stelsemeyer-m60 Why was this closed? Is it not applicable anymore?
This is a duplicate of https://github.com/apache/airflow/pull/31008, with a pending discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on pull request #32631: Parametrize poll_interval in KubernetesPodOperator
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #32631:
URL: https://github.com/apache/airflow/pull/32631#issuecomment-1670600329
I see. The PR was closed due to no activity.
How do you plan to follow up on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stelsemeyer-m60 commented on pull request #32631: Parametrize poll_interval in KubernetesPodOperator
Posted by "stelsemeyer-m60 (via GitHub)" <gi...@apache.org>.
stelsemeyer-m60 commented on PR #32631:
URL: https://github.com/apache/airflow/pull/32631#issuecomment-1672315005
> > > I see. The PR was closed due to no activity.
> > > How do you plan to follow up on it?
> >
> >
> > Would be great to get feedback on my last reply (in that PR). Otherwise I can also make an educated guess/decision. My time is limited at the moment, but I still plan to finish that PR.
>
> Small uggestion - in case it is not clear @stelsemeyer-m60: I suggest (and our CONTRIBUTING docs already explain that) to regular (but not too frequent - use your own judgment taking into account that people look at those in their private time and can be busy/on holidays etc.) gently pinging when you wait for answer.
>
> It's SUPER easy to miss someone's question or explanation, and when you take into accounts that we merge 30-40 PRs a day and have 150 opened PRs and many more issues commented on, the fact that you asked a question might be easily missed. If you look at this from the other side - you have one PR that you care about and maintainers have at the same time 100s of conversations going on, it's quite a bit obvious that it's on the side of author to make sure to follow up if they want to make things happen :). This also shows that you do, actually care, and have not abandoned it (which happens more often than you think).
>
> Just a friendly tip.
Absolutely understandable. Thanks for outlining this, and the tip, @potiuk.
I should have checked CONTRIBUTING doc more carefully. Sometimes mature open source projects are a bit overwhelming, especially when you just want to make a small adjustment that you feel is fair and without risk. Anyways, that shall not be an excuse but maybe rather an additional perspective. I totally get your point and you are absolutely right.
I will follow up on the ticket and go ahead as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org