You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/17 02:26:33 UTC
[GitHub] [airflow] imamdigmi opened a new issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
imamdigmi opened a new issue #18312:
URL: https://github.com/apache/airflow/issues/18312
### Apache Airflow version
2.1.3 (latest released)
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
Some DAGs that are running have a sudden error, and after I checked in the container scheduler, there is an error like this
### What you expected to happen
Scheduler runs smoothly without any errors in the log
### How to reproduce
1. Deploy airflow using Helm Chart
2. Run a DAG
3. DAG Error
4. Check on scheduler container log
### Anything else
Every time
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] imamdigmi edited a comment on issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
imamdigmi edited a comment on issue #18312:
URL: https://github.com/apache/airflow/issues/18312#issuecomment-922277685
Thanks for answering @uranusjr . After some exploring, I'm guessing this problem has to do with schedule intervals. I want to share the DAG, but unfortunately I can't share the DAG code that I created due to privacy reasons from my company, but I'm happy to explain about the DAG I made, this is the schedule interval I use (but mostly we use `@daily`):
1. `@daily`
1. `@hourly`
1. `0 */3 * * *`
1. `0 8 * * *`
1. `0 7 * * *`
1. `0 11 * * *`
1. `0 21 * * *`
1. `30 12 * * *`
1. `0 12 * * *`
1. `0 13 * * *`
These DAGs are generated dynamically with a custom generator that generates hundreds of DAG Objects with configurable schedule intervals, and here is the interval map that I made to support that:
```py
def interval_map(alias):
"""Translate the initials of the schedule interval."""
import string
from datetime import timedelta # noqa: F401
res = {"alias": alias, "interval": None}
default = {
"1h": "@hourly",
"1d": "@daily",
"1w": "@weekly",
"1m": "@monthly",
"1q": "@quarterly",
"1y": "@yearly",
}
if alias is None:
res["alias"] = "manual"
res["interval"] = None
return res
alias = alias.translate(str.maketrans("", "", string.whitespace)).lower()
_number, _type = alias[:-1], alias[-1:] # noqa: F841
if alias == "o":
res["alias"] = "once"
res["interval"] = "@once"
return res
if intrv := default.get(alias):
res["alias"] = alias
res["interval"] = intrv
return res
if "h" in alias:
res["alias"] = alias
res["interval"] = f"0 */{_number} * * *"
return res
if "d" in alias:
res["alias"] = alias
res["interval"] = f"0 0 */{_number} * *"
return res
raise ValueError("Unrecognized interval type")
```
I use the above function to find a schedule interval from an alias, then with the `dict` returned from that function i create a DAG object something like:
```py
table_conf = {"name": "....", "interval": "1d"}
intrv = interval_map(table_conf.get("interval"))
table_conf["interval"] = intrv["interval"]
table_conf["alias"] = intrv["alias"]
dag_obj = DAG(
dag_id=dag_id,
default_args=args,
max_active_runs=1,
description="...",
schedule_interval=table_conf["interval"],
tags=list(set(tags)),
)
```
however I haven't found any hint based on the code in this line:
https://github.com/apache/airflow/blob/v2-1-stable/airflow/models/dag.py#L492
I hope my explanation can help you find the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18312:
URL: https://github.com/apache/airflow/issues/18312#issuecomment-926215866
I had no luck replicating this issue either, unfortunately. My suspicion is there’s a botched DAG deserialisation somewhere that turned `dag.schedule_interval` into a dict (a timedelta interval is serialised into a dict internally) and cuased this issue.
Since we are unlikely to release another 2.1 patch version, and have significantly rewrote this part of the logic for 2.2, I’m inclined to put this on hold until 2.2 is out and you could try replicating on it. Or, if you’re feeling adventurous, we have recently released the second beta for 2.2 as well…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] imamdigmi edited a comment on issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
imamdigmi edited a comment on issue #18312:
URL: https://github.com/apache/airflow/issues/18312#issuecomment-922277685
Thanks for answering @uranusjr . After some exploring, I'm guessing this problem has to do with schedule intervals. I want to share the DAG, but unfortunately I can't share the DAG code that I created due to privacy reasons from my company, but I'm happy to explain about the DAG I made, this is the schedule interval I use (but mostly we use `@daily`):
1. `@daily`
1. `@hourly`
1. `0 */3 * * *`
1. `0 8 * * *`
1. `0 7 * * *`
1. `0 11 * * *`
1. `0 21 * * *`
1. `30 12 * * *`
1. `0 12 * * *`
1. `0 13 * * *`
These DAGs are generated dynamically with a custom generator that generates hundreds of DAG Objects with configurable schedule intervals, and here is the interval map that I made to support that:
```py
def interval_map(alias):
"""Translate the initials of the schedule interval."""
import string
from datetime import timedelta # noqa: F401
res = {"alias": alias, "interval": None}
default = {
"1h": "@hourly",
"1d": "@daily",
"1w": "@weekly",
"1m": "@monthly",
"1q": "@quarterly",
"1y": "@yearly",
}
if alias is None:
res["alias"] = "manual"
res["interval"] = None
return res
alias = alias.translate(str.maketrans("", "", string.whitespace)).lower()
_number, _type = alias[:-1], alias[-1:] # noqa: F841
if alias == "o":
res["alias"] = "once"
res["interval"] = "@once"
return res
if intrv := default.get(alias):
res["alias"] = alias
res["interval"] = intrv
return res
if "h" in alias:
res["alias"] = alias
res["interval"] = f"0 */{_number} * * *"
return res
if "d" in alias:
res["alias"] = alias
res["interval"] = f"0 0 */{_number} * *"
return res
raise ValueError("Unrecognized interval type")
```
I use the above function to find a schedule interval from an alias, then with the `dict` returned from that function i create a DAG object something like:
```py
table_conf = {"name": "....", "interval": "1d"}
intrv = interval_map(table_conf.get("interval"))
table_conf["interval"] = intrv["interval"]
table_conf["alias"] = intrv["alias"]
dag_obj = DAG(
dag_id=dag_id,
default_args=args,
max_active_runs=1,
description="...",
schedule_interval=table_conf["interval"],
tags=list(set(tags)),
)
```
however I haven't found any hint based on the code in this line:
https://github.com/apache/airflow/blob/v2-1-stable/airflow/models/dag.py#L492
I hope my explanation can help you find the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] imamdigmi commented on issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
imamdigmi commented on issue #18312:
URL: https://github.com/apache/airflow/issues/18312#issuecomment-922277685
Thanks for answering @uranusjr . After some exploring, I'm guessing this problem has to do with schedule intervals. I want to share the DAG, but unfortunately I can't share the DAG code that I created due to privacy reasons from my company, but I'm happy to explain about the DAG I made, this is the schedule interval I use (but mostly we use `@daily`):
1. `@daily`
1. `@hourly`
1. `0 */3 * * *`
1. `0 8 * * *`
1. `0 7 * * *`
1. `0 11 * * *`
1. `0 21 * * *`
1. `30 12 * * *`
1. `0 12 * * *`
1. `0 13 * * *`
These DAGs are generated dynamically with a custom generator that generates hundreds of DAG Objects with configurable schedule intervals, and here is the interval map that I made to support that:
```py
def interval_map(alias):
"""Translate the initials of the schedule interval."""
import string
from datetime import timedelta # noqa: F401
res = {"alias": alias, "interval": None}
default = {
"1h": "@hourly",
"1d": "@daily",
"1w": "@weekly",
"1m": "@monthly",
"1q": "@quarterly",
"1y": "@yearly",
}
if alias is None:
res["alias"] = "manual"
res["interval"] = None
return res
alias = alias.translate(str.maketrans("", "", string.whitespace)).lower()
_number, _type = alias[:-1], alias[-1:] # noqa: F841
if alias == "o":
res["alias"] = "once"
res["interval"] = "@once"
return res
if intrv := default.get(alias):
res["alias"] = alias
res["interval"] = intrv
return res
if "h" in alias:
res["alias"] = alias
res["interval"] = f"0 */{_number} * * *"
return res
if "d" in alias:
res["alias"] = alias
res["interval"] = f"0 0 */{_number} * *"
return res
raise ValueError("Unrecognized interval type")
```
then if we use the above function with:
```py
print(interval_map('4h'))
print(interval_map('1w'))
print(interval_map('2d'))
print(interval_map('1d'))
print(interval_map('o'))
print(interval_map(None))
print(interval_map('1q'))
print(interval_map('1m'))
print(interval_map('1y'))
```
the result is:
```
{'alias': '4h', 'interval': '0 */4 * * *'}
{'alias': '1w', 'interval': '@weekly'}
{'alias': '2d', 'interval': '0 0 */2 * *'}
{'alias': '1d', 'interval': '@daily'}
{'alias': 'once', 'interval': '@once'}
{'alias': 'manual', 'interval': None}
{'alias': '1q', 'interval': '@quarterly'}
{'alias': '1m', 'interval': '@monthly'}
{'alias': '1y', 'interval': '@yearly'}
```
however I haven't found any hint based on the code in this line:
https://github.com/apache/airflow/blob/v2-1-stable/airflow/models/dag.py#L492
I hope my explanation can help you find the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] imamdigmi commented on issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
imamdigmi commented on issue #18312:
URL: https://github.com/apache/airflow/issues/18312#issuecomment-925601552
hi @uranusjr , is there any continuation regarding this? or any good news regarding this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] imamdigmi commented on issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
imamdigmi commented on issue #18312:
URL: https://github.com/apache/airflow/issues/18312#issuecomment-933281271
Hi @uranusjr i will close this issue since i already got the culprit, my DAG has dictionary object passing to schedule interval. Thank you for your help, appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18312:
URL: https://github.com/apache/airflow/issues/18312#issuecomment-921511544
Please provide a DAG that reproduces the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] imamdigmi closed issue #18312: Scheduler TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
Posted by GitBox <gi...@apache.org>.
imamdigmi closed issue #18312:
URL: https://github.com/apache/airflow/issues/18312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org