You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/17 14:32:50 UTC
[GitHub] [airflow] BasPH opened a new pull request #20381: Fix kubernetes type hints
BasPH opened a new pull request #20381:
URL: https://github.com/apache/airflow/pull/20381
Was doing work on the KubernetesPodOperator and found several incorrect type hints. Created a separate PR to avoid changing too much in one PR.
Also renamed `_task_status()` to `_pod_state_to_task_state()` for clarity sake. Since it's an "internal" method, I don't think that should cause any issues.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#discussion_r771544019
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_launcher.py
##########
@@ -321,19 +322,24 @@ def _exec_pod_command(self, resp, command: str) -> None:
break
return None
- def process_status(self, job_id: str, status: str) -> str:
- """Process status information for the JOB"""
- status = status.lower()
- if status == PodStatus.PENDING:
+ def process_status(self, pod_name: str, pod_phase: str) -> TaskInstanceState:
+ """
+ Convert a K8S Pod phase to Airflow task state.
+ :param pod_name: Name of the pod
+ :param pod_phase: Phase of the pod
+ :return: Airflow task state corresponding to the pod phase
+ """
+ pod_phase = pod_phase.lower()
+ if pod_phase == PodStatus.PENDING:
return State.QUEUED
Review comment:
I looked at this too and was confused on the direction.
I know mypy expects this to be TaskInstanceState, but State.QUEUED is a member of State class with the field assigned to enum in TaskInstanceState, @potiuk had suggested to use State(State.QUEUED).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH closed pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
BasPH closed pull request #20381:
URL: https://github.com/apache/airflow/pull/20381
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH commented on pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
BasPH commented on pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#issuecomment-997954144
@potiuk ugh bad PR :-( https://github.com/apache/airflow/pull/19572 will make this PR redundant, so I'm going to close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH commented on a change in pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
BasPH commented on a change in pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#discussion_r771851421
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_launcher.py
##########
@@ -321,19 +322,24 @@ def _exec_pod_command(self, resp, command: str) -> None:
break
return None
- def process_status(self, job_id: str, status: str) -> str:
- """Process status information for the JOB"""
- status = status.lower()
- if status == PodStatus.PENDING:
+ def process_status(self, pod_name: str, pod_phase: str) -> TaskInstanceState:
+ """
+ Convert a K8S Pod phase to Airflow task state.
+ :param pod_name: Name of the pod
+ :param pod_phase: Phase of the pod
+ :return: Airflow task state corresponding to the pod phase
+ """
+ pod_phase = pod_phase.lower()
+ if pod_phase == PodStatus.PENDING:
return State.QUEUED
Review comment:
Ah much better! I couldn't think of a good reason for the state conversion anywaysđź‘Ť
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#issuecomment-997292256
Seems like there are some static checks an import errors :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#issuecomment-997242299
@BasPH @subkanthi a big refactor of KPO is underway here: https://github.com/apache/airflow/pull/19572
Would welcome a review if you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
josh-fell commented on pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#issuecomment-996832940
@BasPH There might be some overlap between this and #20321 just in case you need to resolve any conflicts depending on merge timing and/or if you'd like to review that PR. Just a heads up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#discussion_r771544019
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_launcher.py
##########
@@ -321,19 +322,24 @@ def _exec_pod_command(self, resp, command: str) -> None:
break
return None
- def process_status(self, job_id: str, status: str) -> str:
- """Process status information for the JOB"""
- status = status.lower()
- if status == PodStatus.PENDING:
+ def process_status(self, pod_name: str, pod_phase: str) -> TaskInstanceState:
+ """
+ Convert a K8S Pod phase to Airflow task state.
+ :param pod_name: Name of the pod
+ :param pod_phase: Phase of the pod
+ :return: Airflow task state corresponding to the pod phase
+ """
+ pod_phase = pod_phase.lower()
+ if pod_phase == PodStatus.PENDING:
return State.QUEUED
Review comment:
I looked at this too and was confused on the direction.
I know mypy expects the return value to be TaskInstanceState, but State.QUEUED is a member of State class with the field assigned to enum in TaskInstanceState, @potiuk had suggested to use State(State.QUEUED).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #20381: Fix kubernetes type hints
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20381:
URL: https://github.com/apache/airflow/pull/20381#discussion_r771851298
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_launcher.py
##########
@@ -321,19 +322,24 @@ def _exec_pod_command(self, resp, command: str) -> None:
break
return None
- def process_status(self, job_id: str, status: str) -> str:
- """Process status information for the JOB"""
- status = status.lower()
- if status == PodStatus.PENDING:
+ def process_status(self, pod_name: str, pod_phase: str) -> TaskInstanceState:
+ """
+ Convert a K8S Pod phase to Airflow task state.
+ :param pod_name: Name of the pod
+ :param pod_phase: Phase of the pod
+ :return: Airflow task state corresponding to the pod phase
+ """
+ pod_phase = pod_phase.lower()
+ if pod_phase == PodStatus.PENDING:
return State.QUEUED
Review comment:
> I looked at this too and was confused on the direction.
I know mypy expects the return value to be TaskInstanceState, but State.QUEUED is a member of State class with the field assigned to enum in TaskInstanceState, @potiuk had suggested to use State(State.QUEUED).
This is one of the things I address in https://github.com/apache/airflow/pull/19572.
Currently with KPO, we sort of mix up pod phase and task state. We convert pod phase to task state, than look at this converted state to _infer_ conclusions about pod phase. In #19572 I don't convert to state any more and it's much simpler this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org