You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/28 16:07:12 UTC
[GitHub] [airflow] ZaxR opened a new issue #16697: Assigning operator to DAG via bitwise composition does not pickup default args
ZaxR opened a new issue #16697:
URL: https://github.com/apache/airflow/issues/16697
**Description**
In v2.1.0, when attempting to set the dag for an Operator, default_args aren't being honored for start_date.
This relates to an issue in 1.x where using the bitshift operator with Operators that don't specify a dag (or are not within a DAG's context), and a dag where start_date is set via `default_args`, the bitshift operation will error with: `AirflowException: Task is missing the start_date parameter`. That issue was raised several times in the past (most recently in #7996), and @ashb had a solution in #5598 , but it was put on hold since support for bitshift operators was being debated for Airflow 2.x.
Example:
```
from datetime import datetime
from airflow import DAG
from airflow.operators.dummy import DummyOperator
default_args = {
'owner': 'airflow',
'start_date': datetime(2017, 2, 1)
}
dag = DAG('my_dag', default_args=default_args)
dummy = DummyOperator(task_id='dummy')
dag.add_task(dummy)
dummy2 = DummyOperator(task_id='dummy2')
dag.add_task(dummy2)
dummy >> dummy2
```
Setting start_date directly instead of from default_args works as expected:
```
...
dag = DAG('my_dag', start_date=datetime(2017, 2, 1), default_args=default_args)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16697: Assigning operator to DAG via bitwise composition does not pickup default args
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869812710
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZaxR commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ZaxR commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-870695302
From what I've seen in the wild, people heavily use default_args for everything. At companies I've worked at, there's even been utility functions to get "global" default_args as a fallback, which include start_date. So based on the heavy usage and support for using default_args for start_date, I'd argue the fix is justified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869862332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-876365601
> From what I've seen in the wild, people heavily use default_args for everything. At companies I've worked at, there's even been utility functions to get "global" default_args as a fallback, which include start_date. So based on the heavy usage and support for using default_args for start_date, I'd argue the fix is justified.
Setting start date via default_args is the one that bugs me most. Set it on the dag directly instead :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZaxR edited a comment on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ZaxR edited a comment on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869864770
Updated the details above. The error happens on `dag.add_task(dummy)`, where the `dummy` task to add doesn't have its own start_date (or other config for that matter), but rather it would expect it to take on the dag's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZaxR commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ZaxR commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869864770
Updated by issue. The error happens on `dag.add_task(dummy)`, where the `dummy` task to add doesn't have its own start_date (or other config for that matter), but rather it would expect it to take on the dag's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZaxR edited a comment on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ZaxR edited a comment on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869864770
Updated the details above. The error happens on `dag.add_task(dummy)`, where the `dummy` task to add doesn't have its own start_date (or other config for that matter), but rather it would expect it to take on the dag's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-870716928
Personally I wish Airflow never had `default_args`, it’s too magical with a logical native Python approach (`**kwargs`). But I guess that ship sailed way back and we should make it work mroe closely to what users expect in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-870503909
I'm of two minds wether or not we fix this -- as 99% of use cases od `default_args` should just be set directly as attributes on DAG -- it's just that our examples show using default args so many people copy that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16697: Assigning operator to DAG via bitwise composition does not pickup default args
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869812710
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869862332
Are both tasks missing the start date?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZaxR commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
ZaxR commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-869864770
Updated by issue. The error happens on `dag.add_task(dummy)`, where the `dummy` task to add doesn't have its own start_date (or other config for that matter), but rather it would expect it to take on the dag's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16697: DAG.add_task doesn't pick up start_date default arg
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16697:
URL: https://github.com/apache/airflow/issues/16697#issuecomment-870852498
> Personally I wish Airflow never had `default_args`, it’s too magical with a logical native Python approach (`**kwargs`). But I guess that ship sailed way back and we should make it work mroe closely to what users expect in this case.
Side comment - It does save you a bit of hassle when you define dags, but I agree it is a bit too implicit. And yeah. the ship has sailed :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org