You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dirrao (via GitHub)" <gi...@apache.org> on 2024/03/18 12:40:07 UTC
[PR] Merge kubernetes pod objects inplace [airflow]
dirrao opened a new pull request, #38259:
URL: https://github.com/apache/airflow/pull/38259
What happened
In the Kubernetes executor, Merge Kubernetes Pod objects (base_worker_pod with 3 containers and 1 init container) is taking a significant amount of time (> 500 ms) due to the recursive deep copy.
What you think should happen instead
We are constructing the base_worker_pod, dynamic_pod, and executor_config_pod for every task. So, we don't need to retain these pod objects. We can merge these pod objects in place and return the merged pod.
I agree with fact that the immutability brings advantages. Even if we want to retain the pod objects, clone them before calling construct_pod/reconcile_pods function.
Close: #37131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Merge kubernetes pod objects inplace [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #38259: Merge kubernetes pod objects inplace
URL: https://github.com/apache/airflow/pull/38259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Merge kubernetes pod objects inplace [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on PR #38259:
URL: https://github.com/apache/airflow/pull/38259#issuecomment-2006497755
> @dirrao in terms of numbers, how much improvement does this bring to the table? Just curious.
>
> Btw, wondering if there's any risk associated here with this.
I have seen the 3x improvement in the scheduler performance. Check the same in the #37131.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Merge kubernetes pod objects inplace [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38259:
URL: https://github.com/apache/airflow/pull/38259#issuecomment-2103631982
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Merge kubernetes pod objects inplace [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on PR #38259:
URL: https://github.com/apache/airflow/pull/38259#issuecomment-2006520847
@hussein-awala / @jedcunningham
This is breaking with recent changes from Kubernetes job operators. I would like to hear from others if it make sense for Kubernetes executor. I will add flag to enable it for Kubernetes executor and disable it for operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Merge kubernetes pod objects inplace [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on PR #38259:
URL: https://github.com/apache/airflow/pull/38259#issuecomment-2017330510
> what do you mean by breaking IMHO it's just broken, not breaking - because it's impossible to foresee how one pod executing will impact another one, and since you have no influence on that, this is indeterministic. Which trades of performance with predictability . Or am I wrong?
When I made these changes sometime back when there was no Kubernetes job operator. These Kubernetes operators started using the Kubernetes utilities. Immutability is mandatory for Kubernetes operators not for Kubernetes executor. Right now the proposed changes don't distinguish between operators and executor path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Merge kubernetes pod objects inplace [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #38259:
URL: https://github.com/apache/airflow/pull/38259#issuecomment-2015984516
> what do you mean by breaking IMHO it's just broken, not breaking - because it's impossible to foresee how one pod executing will impact another one, and since you have no influence on that, this is indeterministic. Which trades of performance with predictability . Or am I wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org