You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/04 16:15:49 UTC
[GitHub] [airflow] kazanzhy opened a new pull request #20654: Fix MyPy Errors for providers: Tableau, MySQL, IMAP, gRPC, CNCF, Apache
kazanzhy opened a new pull request #20654:
URL: https://github.com/apache/airflow/pull/20654
Fix MyPy Errors for providers: Tableau, MySQL, IMAP, gRPC, CNCF, Apache Hive
related: #19891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780238975
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
That's hell of a union :)
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
The result does not seem to be used (last_log_time = follow_logs() is not used). Seems to be a remnant of previous version of log following. Can we just make the function -> None and do not return anything from it ?
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
~The result does not seem to be used (last_log_time = follow_logs() is not used). Seems to be a remnant of previous version of log following. Can we just make the function -> None and do not return anything from it ?~
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
Scratch that - it is used. but still it seems rather strange as it is only used in calling the follow_logs(since_time=last_log_time) and it's Optional[datetime] there. This Union does not look right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20654: Fix MyPy Errors for providers: Tableau, MySQL, IMAP, gRPC, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#issuecomment-1005971993
some rebase + fix :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780476931
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
As `parse_log_line` method return a pendulum object I thought to add before returning something like
```
if timestamp is not None:
datetime.fromtimestamp(timestamp.timestamp())
```
Then we will have only the `Optional[datetime]` returning type.
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
But I find another solution that reduces the number of types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780240236
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
~The result does not seem to be used (last_log_time = follow_logs() is not used). Seems to be a remnant of previous version of log following. Can we just make the function -> None and do not return anything from it ?~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780517601
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
But I find another solution that reduces the number of types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#issuecomment-1007345959
Seems now all tests have passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#issuecomment-1007345959
Seems now all tests have passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780241115
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
Scratch that - it is used. but still it seems rather strange as it is only used in calling the follow_logs(since_time=last_log_time) and it's Optional[datetime] there. This Union does not look right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780476931
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
As `parse_log_line` method return a pendulum object I thought to add before returning something like
```
if timestamp is not None:
datetime.fromtimestamp(timestamp.timestamp())
```
Then we will have only the `Optional[datetime]` returning type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r779265091
##########
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##########
@@ -56,14 +44,21 @@
)
from airflow.providers.cncf.kubernetes.backcompat.pod_runtime_info_env import PodRuntimeInfoEnv
from airflow.providers.cncf.kubernetes.utils import xcom_sidecar
+from airflow.providers.cncf.kubernetes.utils.pod_manager import PodLaunchFailedException, PodManager, PodPhase
+from airflow.settings import pod_mutation_hook
+from airflow.utils import yaml
+from airflow.utils.context import Context
Review comment:
Need to add this when `TYPE_CHECKING` otherwise the Kubernetes provider will have a minimum Airflow-version dependency on 2.2.3.
##########
File path: airflow/providers/tableau/operators/tableau_refresh_workbook.py
##########
@@ -15,14 +15,11 @@
# specific language governing permissions and limitations
# under the License.
import warnings
-from typing import TYPE_CHECKING, Optional
+from typing import Optional
from airflow.models import BaseOperator
from airflow.providers.tableau.operators.tableau import TableauOperator
-
-if TYPE_CHECKING:
- from airflow.utils.context import Context
-
+from airflow.utils.context import Context
Review comment:
Needs to be in `TYPE_CHECKING` or handled a different way so that the Tableau provider doesn't have a minimum Airflow version requirement of 2.2.3.
##########
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##########
@@ -307,7 +302,7 @@ def __init__(
def _render_nested_template_fields(
self,
content: Any,
- context: Dict,
+ context: Context,
Review comment:
After adding to `TYPE_CHECKING`:
```suggestion
context: "Context",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780240236
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
The result does not seem to be used (last_log_time = follow_logs() is not used). Seems to be a remnant of previous version of log following. Can we just make the function -> None and do not return anything from it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r780238975
##########
File path: airflow/providers/cncf/kubernetes/utils/pod_manager.py
##########
@@ -180,7 +180,7 @@ def follow_container_logs(self, pod: V1Pod, container_name: str) -> None:
So the looping logic is there to let us resume following the logs.
"""
- def follow_logs(since_time: Optional[datetime] = None) -> Optional[datetime]:
+ def follow_logs(since_time: Optional[datetime] = None) -> Union[Date, Time, Duration, datetime, None]:
Review comment:
That's hell of a union :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#issuecomment-1007345959
Seems now tests are passing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a change in pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on a change in pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#discussion_r779703190
##########
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##########
@@ -56,14 +44,21 @@
)
from airflow.providers.cncf.kubernetes.backcompat.pod_runtime_info_env import PodRuntimeInfoEnv
from airflow.providers.cncf.kubernetes.utils import xcom_sidecar
+from airflow.providers.cncf.kubernetes.utils.pod_manager import PodLaunchFailedException, PodManager, PodPhase
+from airflow.settings import pod_mutation_hook
+from airflow.utils import yaml
+from airflow.utils.context import Context
Review comment:
Done
##########
File path: airflow/providers/tableau/operators/tableau_refresh_workbook.py
##########
@@ -15,14 +15,11 @@
# specific language governing permissions and limitations
# under the License.
import warnings
-from typing import TYPE_CHECKING, Optional
+from typing import Optional
from airflow.models import BaseOperator
from airflow.providers.tableau.operators.tableau import TableauOperator
-
-if TYPE_CHECKING:
- from airflow.utils.context import Context
-
+from airflow.utils.context import Context
Review comment:
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #20654:
URL: https://github.com/apache/airflow/pull/20654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on pull request #20654: Fix MyPy Errors for providers: Tableau, CNCF, Apache
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on pull request #20654:
URL: https://github.com/apache/airflow/pull/20654#issuecomment-1007345959
Seems now tests are passing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org