You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/09 09:03:00 UTC
[GitHub] [airflow] ana-carolina-januario opened a new issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
ana-carolina-januario opened a new issue #10824:
URL: https://github.com/apache/airflow/issues/10824
**Apache Airflow version**:1.10.5
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): NA
**Environment**: CentOS Linux release 7.7.1908 (Core), without Airflow environment defined variables.
- **Cloud provider or hardware configuration**: Azure, B4ms standard (4 cpu, 16GiB memory, 30GiB ssd disk)
- **OS** (e.g. from /etc/os-release): CentOS Linux release 7.7.1908 (Core)
- **Kernel** (e.g. `uname -a`): Linux prd-azure-ddp-airflow 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**:
$ pip list
Package Version
---------------------------- ---------
alembic 1.2.1
apache-airflow 1.10.5
apispec 3.0.0
attrs 19.3.0
azure-common 1.1.25
azure-cosmos 3.1.2
azure-datalake-store 0.0.48
azure-mgmt-containerinstance 1.5.0
azure-mgmt-hdinsight 1.4.0
azure-mgmt-resource 9.0.0
azure-nspkg 3.0.2
azure-storage 0.36.0
azure-storage-blob 2.1.0
azure-storage-common 2.1.0
Babel 2.7.0
boto 2.49.0
boto3 1.12.31
botocore 1.15.31
cached-property 1.5.1
certifi 2019.9.11
cffi 1.13.0
chardet 3.0.4
Click 7.0
colorama 0.4.1
colorlog 4.0.2
configparser 3.5.3
croniter 0.3.30
cryptography 2.8
defusedxml 0.6.0
dill 0.2.9
docutils 0.15.2
dumb-init 1.2.2
elasticsearch 5.5.3
elasticsearch-dsl 5.4.0
Flask 1.1.1
Flask-Admin 1.5.3
Flask-AppBuilder 1.13.1
Flask-Babel 0.12.2
Flask-Caching 1.3.3
Flask-JWT-Extended 3.24.0
Flask-Login 0.4.1
Flask-OpenID 1.2.5
Flask-SQLAlchemy 2.4.1
flask-swagger 0.2.13
Flask-WTF 0.14.2
funcsigs 1.0.0
future 0.16.0
gunicorn 19.9.0
idna 2.8
importlib-metadata 0.23
iso8601 0.1.12
itsdangerous 1.1.0
JayDeBeApi 1.1.1
Jinja2 2.10.3
jmespath 0.9.5
JPype1 0.6.3
json-merge-patch 0.2
jsonschema 3.1.1
lazy-object-proxy 1.4.2
lockfile 0.12.2
Mako 1.1.0
Markdown 2.6.11
MarkupSafe 1.1.1
marshmallow 2.19.5
marshmallow-enum 1.5.1
marshmallow-sqlalchemy 0.19.0
more-itertools 7.2.0
mysql-connector 2.2.9
mysqlclient 1.3.14
ndg-httpsclient 0.5.1
numpy 1.17.3
ordereddict 1.1
pandas 0.25.2
pendulum 1.4.4
pip 20.2
prison 0.1.0
psutil 5.6.3
psycopg2 2.8.4
pyasn1 0.4.7
pycparser 2.19
Pygments 2.4.2
PyJWT 1.7.1
pyOpenSSL 19.0.0
pyrsistent 0.15.4
python-daemon 2.1.2
python-dateutil 2.8.0
python-editor 1.0.4
python3-openid 3.1.0
pytz 2019.3
pytzdata 2019.3
PyYAML 5.1.2
requests 2.22.0
s3transfer 0.3.3
setproctitle 1.1.10
setuptools 41.4.0
six 1.12.0
SQLAlchemy 1.3.10
tabulate 0.8.5
tenacity 4.12.0
termcolor 1.1.0
text-unidecode 1.2
thrift 0.11.0
tzlocal 1.5.1
unicodecsv 0.14.1
urllib3 1.25.6
Werkzeug 0.16.0
wheel 0.33.6
WTForms 2.2.1
zipp 0.6.0
zope.deprecation 4.4.0
- **Others**:
**What happened**: I have a dag scheduled to be triggered weekly using '@weekly' in dag creation but the dag was never triggered according to this schedule.
Before the weekly schedule, I've also tried to triggered this same DAG monthly with "0 3 1 * *" which didn't work as well.
**What you expected to happen**: I've expected the referred DAG to be triggered.
**How to reproduce it**:
This can be reproduced by simply create a couple of DAGs scheduled to be triggered with the referred schedule intervals(weekly and monthly).
**Anything else we need to know**:
I don't thing the content of the DAGs impact this issue but let me know if you think they impact.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1050117293
> 'start_date': airflow.utils.dates.days_ago(1),
Don't use "days_ago" -> this is confusing *and you just experienced this very confusion*. we are actuallt deprecating days_ago and already removed it from all examples. Used fixed date for it (6/12/2021) - see our examples.
The way you specified it "days_ago(1)" is yesterday. EVERY SINGLE DAY IT IS `YESTERDAY`.
This means that start_date literally changes every single day.
When you want a dag that should start running 6th of December - make it so explicitly - look at our fresh examples where date is fixed - and set it to 6/12/2021 explicitly. See examples how to it for example here: https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html?highlight=start_date
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ana-carolina-januario commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
ana-carolina-januario commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-699880078
Hi there,
I ended up figuring this out. Basically the was in the value of "start_date" DAG's field. It should be pointing to a date previous to the (first) execution_date (which depends on schedule_interval).In case you want to trigger a DAGs on a daily basis, your schedule_interval will be something like "@daily", your start_date should be 1 day ago. For a schedule_interval @weekly", your start_date should be 8 days ago (= week +1day), etc.
I hope this helps.
Cheers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1050117293
> 'start_date': airflow.utils.dates.days_ago(1),
Don't use "days_ago" -> this is confusing *and you just experienced this very confusion*. We are actually deprecating days_ago and already removed it from all examples. Used fixed date for it (6/12/2021) - see our examples.
The way you specified it "days_ago(1)" is yesterday. EVERY SINGLE DAY IT IS `YESTERDAY`.
This means that start_date literally changes every single day.
When you want a dag that should start running 6th of December - make it so explicitly - look at our fresh examples where date is fixed - and set it to 6/12/2021 explicitly. See examples how to it for example here: https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html?highlight=start_date
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1050677007
> won't it have a impact if I set a specific start date as the dag execution depends on schedule_interval which we pass as a cron which is different for every dag.
What Impact do you think it will have? I do not understand? using days_ago in this case is just wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1054165319
> ok. But is there a workaround for now or it takes timezone as UTC by default?
No idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akanshajainnice edited a comment on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
akanshajainnice edited a comment on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1050539706
won't it have a impact if I set a specific start date as the dag execution depends on schedule_interval which we pass as a cron which is different for every dag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akanshajainnice commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
akanshajainnice commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1049762274
Hi I am facing the same problem. Can you help me understand how to set start date for monthly jobs like u said above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akanshajainnice commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
akanshajainnice commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1050539706
I don't want to set a specific start date as the dag execution depends on schedule_interval which we pass as a cron which is different for every dag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akanshajainnice commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
akanshajainnice commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1053876580
I am using version 1.10.10 for airflow. Using fixed start date with tz argument gave me below error. I couldn't find any document for this version stating if it supports this parameter. Can you please help with the valid format for using fixed date.
'start_date': pendulum.datetime(2022, 1, 1, tz="UTC")
TypeError: __new__() got an unexpected keyword argument 'tz'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akanshajainnice edited a comment on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
akanshajainnice edited a comment on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1049762274
Hi I am facing the same problem. Can you help me understand how to set start date for monthly jobs like u said above.
One of the dags having monthly cron set as "0 11 1 * *" but it does not trigger as per scheduled interval. The dag was created in 06/12/ 2021. With cron schedule, the job was expected to trigger automatically on 1st Jan 2022 and 1st Feb 2022. But it did not trigger on time.
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': airflow.utils.dates.days_ago(1),
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akanshajainnice commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
akanshajainnice commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1054095501
ok. But is there a workaround for now or it takes timezone as UTC by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ana-carolina-januario closed issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
ana-carolina-januario closed issue #10824:
URL: https://github.com/apache/airflow/issues/10824
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-689431077
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akanshajainnice commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
akanshajainnice commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1050699560
mixed it with execution_date. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1054055747
PLease migrate to Airflow 2. Airflow 1.10 reached end-of-life and is not supported for over 8 months already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #10824: Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *".
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #10824:
URL: https://github.com/apache/airflow/issues/10824#issuecomment-1054165319
> ok. But is there a workaround for now or it takes timezone as UTC by default?
No idea. This is standard pendulum library - check with version you have and read the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org