You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/05 11:02:48 UTC

[GitHub] [airflow] bhavaniravi opened a new issue #18734: DAG Serialization doesn't have mro context for Operators

bhavaniravi opened a new issue #18734:
URL: https://github.com/apache/airflow/issues/18734


   ### Apache Airflow version
   
   2.1.3
   
   ### Operating System
   
   Debian GNU/Linux 10 (buster)
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   Breeze
   
   ### What happened
   
   Dag serialization loses the MRO for the operators 
   
   
   
   ### What you expected to happen
   
   _No response_
   
   ### How to reproduce
   
   Example DAG
   
   ```
   from airflow import DAG
   from airflow.operators.python import PythonOperator
   from airflow.utils.dates import days_ago
   from datetime import datetime
   
   DAG_NAME = 'example_subdag_operator'
   
   args = {
       'owner': 'airflow',
   }
   with DAG(
       dag_id=DAG_NAME, default_args=args, start_date=days_ago(2), schedule_interval="@once", tags=['example']
   ) as dag:
   
       tn = PythonOperator(
               task_id=f'simple_python',
               python_callable=simple_python,  # make sure you don't include the () of the function
               provide_context=True,
       )
   ```
   
   
   Serialization script 
   
   ```
   import datetime
   from airflow.operators.dummy import DummyOperator
   from airflow.models import DAG
   from airflow.serialization.serialized_objects import SerializedDAG
   import rich
   
   from example_subdag import dag, PythonOperator, CustomPythonOperator
   
   task = dag.get_task("simple_python")
   print ((isinstance(task, PythonOperator)))
   
   
   output_dict = SerializedDAG.to_dict(dag)
   new_dag = SerializedDAG.from_dict(output_dict)
   
   
   task = new_dag.get_task("simple_python")
   print ((isinstance(task, PythonOperator))) # Returns false
   ```
   
   ### Anything else
   
   Related issues
   
   #18492 
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #18734: DAG Serialization doesn't have mro context for Operators

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18734:
URL: https://github.com/apache/airflow/issues/18734#issuecomment-934345341


   (The fix is as mik-laj suggests, but we don't need this issue as we already have #18492 )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #18734: DAG Serialization doesn't have mro context for Operators

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18734:
URL: https://github.com/apache/airflow/issues/18734#issuecomment-934344520


   Yeah -- this is by design. A serialized DAG is not the same as a real dag.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #18734: DAG Serialization doesn't have mro context for Operators

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #18734:
URL: https://github.com/apache/airflow/issues/18734#issuecomment-934340103


   I think it is expected.  We don't want to load the user code on the web server.
   
   If we need to do some MRO check, we should do it during serialization.  We are already doing something similar for the DummyOperator.
   https://github.com/apache/airflow/blob/b0442f30e296eae5e1f38fb74d0fbb6b1c5cef54/airflow/serialization/serialized_objects.py#L483-L484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb closed issue #18734: DAG Serialization doesn't have mro context for Operators

Posted by GitBox <gi...@apache.org>.
ashb closed issue #18734:
URL: https://github.com/apache/airflow/issues/18734


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org