You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/09 10:58:47 UTC

[GitHub] [airflow] turbaszek opened a new pull request #10827: Fix chain methods for XComArg

turbaszek opened a new pull request #10827:
URL: https://github.com/apache/airflow/pull/10827


   __lshift__ and __rshift__ methods should return other not self.
   This PR fixes XComArg implementation  to support chain like this one:
   BaseOprator >> XComArg >> BaseOperator
   
   Related to: #10153
   
   Kudos to @yuqian90 for spotting this issue! 🚀 
   
   <img width="728" alt="Screenshot 2020-09-09 at 12 57 21" src="https://user-images.githubusercontent.com/9528307/92590228-2357d480-f29c-11ea-9dcd-9ad14e18a697.png">
   I
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] yuqian90 commented on a change in pull request #10827: Fix chain methods for XComArg

Posted by GitBox <gi...@apache.org>.
yuqian90 commented on a change in pull request #10827:
URL: https://github.com/apache/airflow/pull/10827#discussion_r485542846



##########
File path: airflow/example_dags/example_xcomargs.py
##########
@@ -43,11 +44,26 @@ def print_value(value):
     default_args={'owner': 'airflow'},
     start_date=days_ago(2),
     schedule_interval=None,
-    tags=['example']
+    tags=['example'],
 ) as dag:
     task1 = PythonOperator(
         task_id='generate_value',
         python_callable=generate_value,
     )
 
     print_value(task1.output)
+
+
+with DAG(
+    "example_xcom_args_with_operators",
+    default_args={'owner': 'airflow'},
+    start_date=days_ago(2),
+    schedule_interval=None,
+    tags=['example'],
+) as dag2:
+    bash_op1 = BashOperator(task_id="c", bash_command="echo c")
+    bash_op2 = BashOperator(task_id="d", bash_command="echo c")
+    xcom_args_a = print_value("first!")
+    xcom_args_b = print_value("second!")
+
+    bash_op1 >> xcom_args_a >> xcom_args_b >> bash_op2

Review comment:
       Thanks for the quick fix. One other issue is that if we do this at this line:
   ```
   xcom_args_a >> bash_op1 >>  xcom_args_b
   ```
   
   Is this supposed to work? i have not tried yet, but looking at the code, I think `BaseOperator.__rshift__` will complain that it can't handle `xcom_args`. That's a different issue though so the fix can be done separately.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] turbaszek commented on pull request #10827: Fix chain methods for XComArg

Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #10827:
URL: https://github.com/apache/airflow/pull/10827#issuecomment-691907889


   @casassg @yuqian90 I've added missing method. I'm wondering about making `BaseTaskOperation` class that will implement all those methods so we can use it in `XComArg`, `BaseOperator` and `TaskGroup` WDYT? @ashb @dimberman @potiuk 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on pull request #10827: Fix chain methods for XComArg

Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #10827:
URL: https://github.com/apache/airflow/pull/10827#issuecomment-691927501


   > @casassg @yuqian90 I've added missing method. I'm wondering about making `BaseTaskOperation` class that will implement all those methods so we can use it in `XComArg`, `BaseOperator` and `TaskGroup` WDYT? @ashb @dimberman @potiuk
   
   Sounds like a prime use of a mixin to me, rather than a base class. But some DRYing here now we have it three places definitely sounds good.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] turbaszek commented on pull request #10827: Fix chain methods for XComArg

Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #10827:
URL: https://github.com/apache/airflow/pull/10827#issuecomment-689487363


   @casassg would like to take a look?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] turbaszek merged pull request #10827: Fix chain methods for XComArg

Posted by GitBox <gi...@apache.org>.
turbaszek merged pull request #10827:
URL: https://github.com/apache/airflow/pull/10827


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] turbaszek commented on a change in pull request #10827: Fix chain methods for XComArg

Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #10827:
URL: https://github.com/apache/airflow/pull/10827#discussion_r485548810



##########
File path: airflow/example_dags/example_xcomargs.py
##########
@@ -43,11 +44,26 @@ def print_value(value):
     default_args={'owner': 'airflow'},
     start_date=days_ago(2),
     schedule_interval=None,
-    tags=['example']
+    tags=['example'],
 ) as dag:
     task1 = PythonOperator(
         task_id='generate_value',
         python_callable=generate_value,
     )
 
     print_value(task1.output)
+
+
+with DAG(
+    "example_xcom_args_with_operators",
+    default_args={'owner': 'airflow'},
+    start_date=days_ago(2),
+    schedule_interval=None,
+    tags=['example'],
+) as dag2:
+    bash_op1 = BashOperator(task_id="c", bash_command="echo c")
+    bash_op2 = BashOperator(task_id="d", bash_command="echo c")
+    xcom_args_a = print_value("first!")
+    xcom_args_b = print_value("second!")
+
+    bash_op1 >> xcom_args_a >> xcom_args_b >> bash_op2

Review comment:
       Same DAG is used in tests to make sure that it works. `BaseOperator._set_relatives` handles XComArgs properly. However, `XComArgs` is missing in type hints due to cyclic imports. I think that once we have your PR merged we may consider something like `TaskType` that will be union of types that are "task-like"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org