You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/23 12:44:12 UTC

[GitHub] [airflow] turbaszek opened a new pull request #7833: Remove super().__init__() call in backported hooks

turbaszek opened a new pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833
 
 
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#issuecomment-602666761
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7833?src=pr&el=h1) Report
   > Merging [#7833](https://codecov.io/gh/apache/airflow/pull/7833?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4bde99f1323d72f6c84c1548079d5e98fc0a2a9a&el=desc) will **decrease** coverage by `0.31%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7833/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7833?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7833      +/-   ##
   ==========================================
   - Coverage   86.95%   86.64%   -0.32%     
   ==========================================
     Files         926      926              
     Lines       44974    44975       +1     
   ==========================================
   - Hits        39109    38969     -140     
   - Misses       5865     6006     +141     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7833?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...viders/google/cloud/example\_dags/example\_pubsub.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX3B1YnN1Yi5weQ==) | `100.00% <100.00%> (ø)` | |
   | [airflow/providers/google/cloud/hooks/pubsub.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL3B1YnN1Yi5weQ==) | `94.47% <100.00%> (+0.03%)` | :arrow_up: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
   | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
   | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7833/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.73% <0.00%> (-0.15%)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7833?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7833?src=pr&el=footer). Last update [4bde99f...921bc7c](https://codecov.io/gh/apache/airflow/pull/7833?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#discussion_r396461191
 
 

 ##########
 File path: backport_packages/setup_backport_packages.py
 ##########
 @@ -155,6 +156,11 @@ def pure_airflow_models_filter(node: LN, capture: Capture, filename: Filename) -
         """Check if select is exactly [airflow, . , models]"""
         return len([ch for ch in node.children[1].leaves()]) == 3
 
+    def remove_super_init_call(node: LN, capture: Capture, filename: Filename) -> None:
+        for ch in node.post_order():
+            if hasattr(ch, "value") and ch.value == "super":
 
 Review comment:
   I've added an additional check ✅

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#discussion_r396473010
 
 

 ##########
 File path: backport_packages/setup_backport_packages.py
 ##########
 @@ -155,6 +155,12 @@ def pure_airflow_models_filter(node: LN, capture: Capture, filename: Filename) -
         """Check if select is exactly [airflow, . , models]"""
         return len([ch for ch in node.children[1].leaves()]) == 3
 
+    def remove_super_init_call(node: LN, capture: Capture, filename: Filename) -> None:
+        for ch in node.post_order():
+            if isinstance(ch, Leaf) and ch.value == "super":
+                if any(c.value for c in ch.parent.post_order() if isinstance(c, Leaf)):
+                    ch.parent.remove()
 
 Review comment:
   argghhh, in that case, removal is better I agree 👍 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#discussion_r396442318
 
 

 ##########
 File path: backport_packages/setup_backport_packages.py
 ##########
 @@ -155,6 +156,11 @@ def pure_airflow_models_filter(node: LN, capture: Capture, filename: Filename) -
         """Check if select is exactly [airflow, . , models]"""
         return len([ch for ch in node.children[1].leaves()]) == 3
 
+    def remove_super_init_call(node: LN, capture: Capture, filename: Filename) -> None:
+        for ch in node.post_order():
+            if hasattr(ch, "value") and ch.value == "super":
 
 Review comment:
   can we have something else than super().init() removed here? I do not see check that it is actually an init() method?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#discussion_r396470979
 
 

 ##########
 File path: backport_packages/setup_backport_packages.py
 ##########
 @@ -155,6 +155,12 @@ def pure_airflow_models_filter(node: LN, capture: Capture, filename: Filename) -
         """Check if select is exactly [airflow, . , models]"""
         return len([ch for ch in node.children[1].leaves()]) == 3
 
+    def remove_super_init_call(node: LN, capture: Capture, filename: Filename) -> None:
+        for ch in node.post_order():
+            if isinstance(ch, Leaf) and ch.value == "super":
+                if any(c.value for c in ch.parent.post_order() if isinstance(c, Leaf)):
+                    ch.parent.remove()
 
 Review comment:
   The problem is in signature:
   
   ```[2020-03-23 09:16:06,791] {taskinstance.py:887} INFO - Executing <Task(CloudFunctionDeployFunctionOperator): gcf_deploy_task> on 2020-03-22T00:00:00+00:00
   [2020-03-23 09:16:06,795] {standard_task_runner.py:53} INFO - Started process 587 to run task
   [2020-03-23 09:16:07,040] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: example_gcp_function.gcf_deploy_task 2020-03-22T00:00:00+00:00 [running]> bac65bcc36b9
   [2020-03-23 09:16:07,078] {taskinstance.py:1128} ERROR - __init__() missing 1 required positional argument: 'source'
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 966, in _run_raw_task
       result = task_copy.execute(context=context)
     File "/usr/local/lib/python3.6/site-packages/airflow/providers/google/cloud/operators/functions.py", line 194, in execute
       hook = CloudFunctionsHook(gcp_conn_id=self.gcp_conn_id, api_version=self.api_version)
     File "/usr/local/lib/python3.6/site-packages/airflow/providers/google/cloud/hooks/functions.py", line 50, in __init__
       super().__init__(gcp_conn_id, delegate_to)
     File "/usr/local/lib/python3.6/site-packages/airflow/providers/google/cloud/hooks/base.py", line 158, in __init__
       super().__init__()
   TypeError: __init__() missing 1 required positional argument: 'source'
   [2020-03-23 09:16:07,085] {taskinstance.py:1185} INFO - Marking task as FAILED.dag_id=example_gcp_function, task_id=gcf_deploy_task, execution_date=20200322T000000, start_date=20200323T091606, end_date=20200323T091607
   ```
   
   The 1.10 signature:
   https://github.com/apache/airflow/blob/655f0cd0e640830e12dc4bf94725b5704d31ec16/airflow/hooks/base_hook.py#L37-L46

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on issue #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#issuecomment-602868408
 
 
   The CI is green but the status was not updated :< @potiuk should we merge?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#discussion_r396465256
 
 

 ##########
 File path: backport_packages/setup_backport_packages.py
 ##########
 @@ -155,6 +155,12 @@ def pure_airflow_models_filter(node: LN, capture: Capture, filename: Filename) -
         """Check if select is exactly [airflow, . , models]"""
         return len([ch for ch in node.children[1].leaves()]) == 3
 
+    def remove_super_init_call(node: LN, capture: Capture, filename: Filename) -> None:
+        for ch in node.post_order():
+            if isinstance(ch, Leaf) and ch.value == "super":
+                if any(c.value for c in ch.parent.post_order() if isinstance(c, Leaf)):
+                    ch.parent.remove()
 
 Review comment:
   Do you think instead of removing we can change it to **super(BashOperator, self)** format which is compatible in both Py2 and Py3 ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#discussion_r396444798
 
 

 ##########
 File path: backport_packages/setup_backport_packages.py
 ##########
 @@ -155,6 +156,11 @@ def pure_airflow_models_filter(node: LN, capture: Capture, filename: Filename) -
         """Check if select is exactly [airflow, . , models]"""
         return len([ch for ch in node.children[1].leaves()]) == 3
 
+    def remove_super_init_call(node: LN, capture: Capture, filename: Filename) -> None:
+        for ch in node.post_order():
+            if hasattr(ch, "value") and ch.value == "super":
 
 Review comment:
   ```suggestion
               if isinstaance((ch, Leaf) and ch.value == "super":
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833#discussion_r396471415
 
 

 ##########
 File path: backport_packages/setup_backport_packages.py
 ##########
 @@ -155,6 +155,12 @@ def pure_airflow_models_filter(node: LN, capture: Capture, filename: Filename) -
         """Check if select is exactly [airflow, . , models]"""
         return len([ch for ch in node.children[1].leaves()]) == 3
 
+    def remove_super_init_call(node: LN, capture: Capture, filename: Filename) -> None:
+        for ch in node.post_order():
+            if isinstance(ch, Leaf) and ch.value == "super":
+                if any(c.value for c in ch.parent.post_order() if isinstance(c, Leaf)):
+                    ch.parent.remove()
 
 Review comment:
   On actual master the `super().__init__()` calls LoggingMixin

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj merged pull request #7833: Remove super().__init__() call in backported hooks

Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #7833: Remove super().__init__() call in backported hooks
URL: https://github.com/apache/airflow/pull/7833
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services