You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/10 10:16:43 UTC

[GitHub] [airflow] turbaszek opened a new issue #11393: Scheduler does not support custom operators inheriting from DummyOperator

turbaszek opened a new issue #11393:
URL: https://github.com/apache/airflow/issues/11393


   **Apache Airflow version**:
   v2.0.0
   
   **What happened**:
   
   Custom operators inheriting from DummyOperator are stuck in a scheduled state.
   
   **What you expected to happen**:
   
   All operators should be successfully executed.
   
   **How to reproduce it**:
   
   ```py
   from datetime import datetime
   from airflow import DAG
   from airflow.operators.dummy_operator import DummyOperator
   
   class MyJob(DummyOperator):
       template_fields_renderers = {
           "body": "json"
       }
       template_fields = ("body",)
   
       def __init__(self, body, *args, **kwargs):
           super().__init__(*args, **kwargs)
           self.body = body
   
   with DAG(
       dag_id="test",
       schedule_interval=None,
       start_date=datetime(2020, 8, 13),
   ) as dag:
       MyJob(task_id="aaaa", body={"aaa": "bbb"})
   
   ```
   
   **Anything else we need to know**:
   This is a known problem that we will fix, see: https://github.com/apache/airflow/pull/10956#discussion_r502492775


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #11393: Optimize subclasses of DummyOperator be set straight to success instead of being run

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11393:
URL: https://github.com/apache/airflow/issues/11393#issuecomment-956357863


   > This optimization currently does not function correctly with the `pre_execute` and `post_execute` parameters introduced (on an experimental basis) in #17576.
   
   There is no point in wasting a worker slot for `DummyOperators`, there are exceptions too where a DummyOperator has callbacks or not. On a large scale this makes a huge difference and overhead of sending task to Workers if it is not needed.
   
   
   The `pre/post_execute` are still "experimental" for a good reason. There still aren't that many use-cases for it. We might rip it of if that does not serves a purpose tbh.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #11393: Scheduler does not support custom operators inheriting from DummyOperator

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11393:
URL: https://github.com/apache/airflow/issues/11393#issuecomment-707256699


   We subclass DummyOperator for ExternalTaskMarker


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] malthe commented on issue #11393: Optimize subclasses of DummyOperator be set straight to success instead of being run

Posted by GitBox <gi...@apache.org>.
malthe commented on issue #11393:
URL: https://github.com/apache/airflow/issues/11393#issuecomment-956326552






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil closed issue #11393: Optimize subclasses of DummyOperator be set straight to success instead of being run

Posted by GitBox <gi...@apache.org>.
kaxil closed issue #11393:
URL: https://github.com/apache/airflow/issues/11393


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #11393: Optimize subclasses of DummyOperator be set straight to success instead of being run

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11393:
URL: https://github.com/apache/airflow/issues/11393#issuecomment-956357863


   > This optimization currently does not function correctly with the `pre_execute` and `post_execute` parameters introduced (on an experimental basis) in #17576.
   
   There is no point in wasting a worker slot for `DummyOperators`, there are exceptions too where a DummyOperator has callbacks or not. On a large scale this makes a huge difference and overhead of sending task to Workers if it is not needed.
   
   
   The `pre/post_execute` are still "experimental" for a good reason. There still aren't that many use-cases for it. We might rip it of if that does not serves a purpose tbh.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] malthe commented on issue #11393: Optimize subclasses of DummyOperator be set straight to success instead of being run

Posted by GitBox <gi...@apache.org>.
malthe commented on issue #11393:
URL: https://github.com/apache/airflow/issues/11393#issuecomment-956326552


   This optimization currently does not function correctly with the `pre_execute` and `post_execute` parameters introduced (on an experimental basis) in #17576.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org