You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/06 07:25:33 UTC

[GitHub] [airflow] dstandish opened a new pull request, #27526: Cleanup backcompat code kpo

dstandish opened a new pull request, #27526:
URL: https://github.com/apache/airflow/pull/27526

   Assumes https://github.com/apache/airflow/pull/27524 will be merged first!
   
   I did a thorough review of the kubernetes system KPO backcompat test.
   
   1. most of the time, no actual backcompat logic was tested and the tests were simply duplicates that can be removed.
   2. sometimes actual backcompat was tested and in this case i moved to the main KPO system test module and indicated it is a test of a backcompat feature, and where possible i parametrized.
   3. in a some cases, tests that needed to be retained were not actually system tests so i either moved to the unit test module or made it a real system test.
   4. in the retained KPO system test module I had to convert it from unittests
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015880123


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -51,7 +50,7 @@
 
 def create_context(task) -> Context:
     dag = DAG(dag_id="dag")
-    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")  # type: ignore

Review Comment:
   It’s because Pendulum imports `timezone` in `__init__.py`, and then later overrides the name to a function:
   
   https://github.com/sdispater/pendulum/blob/cc2db5d573f8936a475148bda3ff92057723fc48/pendulum/__init__.py
   
   It is not unlike how Mypy dislikes this:
   
   ```python
   a = 1
   a = "2"  # type: ignore
   
   def f(x: str) -> None:
       print(x)
   
   f(a)  # error: Argument 1 to "f" has incompatible type "int"; expected "str"
   ```
   
   Pendulum does not really care about strong typing, so everything’s fine from its perspective, but from Mypy’s perspective `pendulum.timezone` is still declared as a module (because that’s what appears first), not a function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015194596


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -79,17 +77,14 @@ def get_kubeconfig_path():
 
 def get_label():
     test = os.environ.get("PYTEST_CURRENT_TEST")
+    test = test.split(" ")[0]  # if invoked from setup, will have suffix ' (setup)', e.g.

Review Comment:
   this change was necessitated by the switch to pytest... the test name shows up differently in diff contexts...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1016988746


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -387,8 +369,53 @@ def test_pod_resources(self):
         }
         assert self.expected_pod == actual_pod
 
-    def test_pod_affinity(self):
-        affinity = {
+    @pytest.mark.parametrize(
+        "val",
+        [
+            param(
+                k8s.V1Affinity(
+                    node_affinity=k8s.V1NodeAffinity(
+                        required_during_scheduling_ignored_during_execution=k8s.V1NodeSelector(
+                            node_selector_terms=[
+                                k8s.V1NodeSelectorTerm(
+                                    match_expressions=[
+                                        k8s.V1NodeSelectorRequirement(
+                                            key="beta.kubernetes.io/os",
+                                            operator="In",
+                                            values=["linux"],
+                                        )
+                                    ]
+                                )
+                            ]
+                        )
+                    )

Review Comment:
   well, this was before my time, but obvs if you're using objects it can be helpful e.g. with autocomplete and validation.  
   but yeah it was actually kind of tricky to use. 
   
   maybe we should only be deprecating the airflow objects, and still support json or k8s object.
   
   but, note that this PR doesn't change anything related to deprecations.  it just gets rid of noise in the "backcompat" test, and moves legitimate "backcompat" tests inline with the main tests. which helps e.g. in this case becasue you can see side-by-side new vs "backcompat". in this particular test, there actually _wasn't_ any test of the "new" behavior previously (maybe because it's such a pain in the ass?) so i added that here.
   
   anyway, if any of you "looky lous" has an inclination to approve that would be welcome 😛 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015227247


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -51,7 +50,7 @@
 
 def create_context(task) -> Context:
     dag = DAG(dag_id="dag")
-    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")  # type: ignore

Review Comment:
   yea... sigh... so... i was getting `module is not callable` error. happened twice  in a row so not flakey.  maybe it was confused by the other `timezone`. it doesn't make any sense. but it's just a test so just ... went the easy route.  and frankly i have no clue how i would fix it otherwise.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015934916


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -51,7 +50,7 @@
 
 def create_context(task) -> Context:
     dag = DAG(dag_id="dag")
-    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")  # type: ignore

Review Comment:
   ok i updated to use `tz`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015204273


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -51,7 +50,7 @@
 
 def create_context(task) -> Context:
     dag = DAG(dag_id="dag")
-    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")  # type: ignore

Review Comment:
   Hmm, why is this needed, is pendulum’s type annotation falty?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015254128


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -51,7 +50,7 @@
 
 def create_context(task) -> Context:
     dag = DAG(dag_id="dag")
-    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")  # type: ignore

Review Comment:
   Ah that makes sense, there _is_ a `pendulum.timezone` subpackage.
   
   Some of our code uses `pendulum.tz.timezone` instead and that seems to work fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1016116743


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -387,8 +369,53 @@ def test_pod_resources(self):
         }
         assert self.expected_pod == actual_pod
 
-    def test_pod_affinity(self):
-        affinity = {
+    @pytest.mark.parametrize(
+        "val",
+        [
+            param(
+                k8s.V1Affinity(
+                    node_affinity=k8s.V1NodeAffinity(
+                        required_during_scheduling_ignored_during_execution=k8s.V1NodeSelector(
+                            node_selector_terms=[
+                                k8s.V1NodeSelectorTerm(
+                                    match_expressions=[
+                                        k8s.V1NodeSelectorRequirement(
+                                            key="beta.kubernetes.io/os",
+                                            operator="In",
+                                            values=["linux"],
+                                        )
+                                    ]
+                                )
+                            ]
+                        )
+                    )

Review Comment:
   yes, quite. makes you wonder whether it really makes sense to deprecate the old way



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
potiuk merged PR #27526:
URL: https://github.com/apache/airflow/pull/27526


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1014781481


##########
airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py:
##########
@@ -62,6 +62,23 @@
     from airflow.utils.context import Context
 
 
+def _task_id_to_pod_name(val):

Review Comment:
   this is from https://github.com/apache/airflow/pull/27524 which should be merged first



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1016523494


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -387,8 +369,53 @@ def test_pod_resources(self):
         }
         assert self.expected_pod == actual_pod
 
-    def test_pod_affinity(self):
-        affinity = {
+    @pytest.mark.parametrize(
+        "val",
+        [
+            param(
+                k8s.V1Affinity(
+                    node_affinity=k8s.V1NodeAffinity(
+                        required_during_scheduling_ignored_during_execution=k8s.V1NodeSelector(
+                            node_selector_terms=[
+                                k8s.V1NodeSelectorTerm(
+                                    match_expressions=[
+                                        k8s.V1NodeSelectorRequirement(
+                                            key="beta.kubernetes.io/os",
+                                            operator="In",
+                                            values=["linux"],
+                                        )
+                                    ]
+                                )
+                            ]
+                        )
+                    )

Review Comment:
   Ugly it is. I agree. What are the benefits in deprecating the old way BTW :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015195525


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -79,17 +77,14 @@ def get_kubeconfig_path():
 
 def get_label():
     test = os.environ.get("PYTEST_CURRENT_TEST")
+    test = test.split(" ")[0]  # if invoked from setup, will have suffix ' (setup)', e.g.
     label = "".join(filter(str.isalnum, test)).lower()
-    return label[-63]
+    return label[-63:]

Review Comment:
   ... as a result, looking at just this one character was unreliable. here we also make it longer so it's more meaningful.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015953165


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -79,17 +77,14 @@ def get_kubeconfig_path():
 
 def get_label():
     test = os.environ.get("PYTEST_CURRENT_TEST")
+    test = test.split(" ")[0]  # if invoked from setup, will have suffix ' (setup)', e.g.

Review Comment:
   ```suggestion
       test = test.split(" ", 1)[0]  # if invoked from setup, will have suffix ' (setup)', e.g.
   ```
   
   No need to do additional splits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015886525


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -51,7 +50,7 @@
 
 def create_context(task) -> Context:
     dag = DAG(dag_id="dag")
-    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")  # type: ignore

Review Comment:
   oh, yeah, makes sense.  bad pendulum.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015953966


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -387,8 +369,53 @@ def test_pod_resources(self):
         }
         assert self.expected_pod == actual_pod
 
-    def test_pod_affinity(self):
-        affinity = {
+    @pytest.mark.parametrize(
+        "val",
+        [
+            param(
+                k8s.V1Affinity(
+                    node_affinity=k8s.V1NodeAffinity(
+                        required_during_scheduling_ignored_during_execution=k8s.V1NodeSelector(
+                            node_selector_terms=[
+                                k8s.V1NodeSelectorTerm(
+                                    match_expressions=[
+                                        k8s.V1NodeSelectorRequirement(
+                                            key="beta.kubernetes.io/os",
+                                            operator="In",
+                                            values=["linux"],
+                                        )
+                                    ]
+                                )
+                            ]
+                        )
+                    )

Review Comment:
   OMG this is ugly (not your fault)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015196760


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -79,17 +77,14 @@ def get_kubeconfig_path():
 
 def get_label():
     test = os.environ.get("PYTEST_CURRENT_TEST")
+    test = test.split(" ")[0]  # if invoked from setup, will have suffix ' (setup)', e.g.
     label = "".join(filter(str.isalnum, test)).lower()
-    return label[-63]
+    return label[-63:]
 
 
 @pytest.mark.execution_timeout(180)
-class TestKubernetesPodOperatorSystem(unittest.TestCase):
-    def get_current_task_name(self):
-        # reverse test name to make pod name unique (it has limited length)
-        return "_" + unittest.TestCase.id(self).replace(".", "_")[::-1]

Review Comment:
   if we're just after "unique" we can use something simpler, like uuid4



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1015864460


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -51,7 +50,7 @@
 
 def create_context(task) -> Context:
     dag = DAG(dag_id="dag")
-    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")  # type: ignore

Review Comment:
   that's so weird though... how does that even work, if there is a function and a package that have the same name?  is it becaues it's a package and not a module that it works? and they just say "tie goes to the function"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27526: Cleanup backcompat code kpo

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27526:
URL: https://github.com/apache/airflow/pull/27526#discussion_r1016988746


##########
kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -387,8 +369,53 @@ def test_pod_resources(self):
         }
         assert self.expected_pod == actual_pod
 
-    def test_pod_affinity(self):
-        affinity = {
+    @pytest.mark.parametrize(
+        "val",
+        [
+            param(
+                k8s.V1Affinity(
+                    node_affinity=k8s.V1NodeAffinity(
+                        required_during_scheduling_ignored_during_execution=k8s.V1NodeSelector(
+                            node_selector_terms=[
+                                k8s.V1NodeSelectorTerm(
+                                    match_expressions=[
+                                        k8s.V1NodeSelectorRequirement(
+                                            key="beta.kubernetes.io/os",
+                                            operator="In",
+                                            values=["linux"],
+                                        )
+                                    ]
+                                )
+                            ]
+                        )
+                    )

Review Comment:
   well, this was before my time, but obvs if you're using objects it can be helpful e.g. with autocomplete and validation.  
   but yeah it was actually kind of tricky to use. 
   
   maybe we should only be deprecating the airflow objects, and still support json or k8s object.
   
   but, note that this PR doesn't change anything related to deprecations.  it just gets rid of noise in the "backcompat" test, and moves legitimate "backcompat" tests inline with the main tests. which helps e.g. in this case becasue you can see side-by-side new vs "backcompat". in this particular test, there actually _wasn't_ any test of the "new" behavior previously (maybe because it's such a pain in the ass?) so i added that here.
   
   anyway, if any of you "looky lous" has an inclination to approve that would be welcome



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org