You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/29 03:20:39 UTC
[GitHub] [airflow] covaliov opened a new issue #19869: Custom Timetable Import Error
covaliov opened a new issue #19869:
URL: https://github.com/apache/airflow/issues/19869
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Darwin Kernel Version 21.1.0 RELEASE_ARM64_T8101 arm64
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
python_version | 3.9.7 (default, Sep 16 2021, 23:53:23) [Clang 12.0.0 ]
### What happened
The following error is displayed in Web UI:
```
Broken DAG: [<EDITED>/scripts/airflow/dags/master/sample_dag/sample_dag.py] Traceback (most recent call last):
File "<EDITED>/miniconda3/envs/dev-airflow/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 271, in serialize_to_json
serialized_object[key] = _encode_timetable(value)
File "<EDITED>/miniconda3/envs/dev-airflow/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 152, in _encode_timetable
raise _TimetableNotRegistered(importable_string)
airflow.serialization.serialized_objects._TimetableNotRegistered: Timetable class 'workday.AfterWorkdayTimetable' is not registered
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<EDITED>/miniconda3/envs/dev-airflow/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 937, in to_dict
json_dict = {"__version": cls.SERIALIZER_VERSION, "dag": cls.serialize_dag(var)}
File "<EDITED>/miniconda3/envs/dev-airflow/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 849, in serialize_dag
raise SerializationError(f'Failed to serialize DAG {dag.dag_id!r}: {e}')
airflow.exceptions.SerializationError: Failed to serialize DAG 'learning_example_workday_timetable': Timetable class 'workday.AfterWorkdayTimetable' is not registered
```
### What you expected to happen
For the custom timetable to be implemented and used by DAG.
### How to reproduce
Following instructions from [Custom DAG Scheduling with Timetables](https://airflow.apache.org/docs/apache-airflow/stable/howto/timetable.html) with following new DAG to implement:
```python
import datetime
from airflow import DAG
from airflow import plugins_manager
from airflow.operators.dummy import DummyOperator
from workday import AfterWorkdayTimetable
with DAG(
dag_id="learning_example_workday_timetable",
start_date=datetime.datetime(2021,11,20),
timetable=AfterWorkdayTimetable(),
tags=["example","learning","timetable"],
) as dag:
DummyOperator(task_id="run_this")
```
### Anything else
I have tried digging through the code and believe the issue is in this line:
https://github.com/apache/airflow/blob/fb478c00cdc5e78d5e85fe5ac103707c829be2fb/airflow/serialization/serialized_objects.py#L149
Perhaps the [Custom DAG Scheduling with Timetables](https://airflow.apache.org/docs/apache-airflow/stable/howto/timetable.html) expects an `__eq__` implemented in the `AfterWorkdayTimetable` class but it would appear that the `AfterWorkdayTimetable` class imported through the DAG and the `AfterWorkdayTimetable` class imported through `plugin_manager` have different `id()`'s:
https://github.com/apache/airflow/blob/fb478c00cdc5e78d5e85fe5ac103707c829be2fb/airflow/serialization/serialized_objects.py#L129
The only way I could get it to import successfully was via the following sequence of import statements since [_get_registered_timetable](https://github.com/apache/airflow/blob/fb478c00cdc5e78d5e85fe5ac103707c829be2fb/airflow/serialization/serialized_objects.py#L124) uses a lazy import:
```python
import datetime
from airflow import DAG
from airflow import plugins_manager
from airflow.operators.dummy import DummyOperator
plugins_manager.initialize_timetables_plugins()
from workday import AfterWorkdayTimetable
```
I also had the webserver and scheduler restarted and confirmed the plugin is seen via cli:
```bash
airflow plugins ✔ dev-airflow 19:01:43
name | source
=========================+===========================
workday_timetable_plugin | $PLUGINS_FOLDER/workday.py
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19869: Custom Timetable Import Error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19869:
URL: https://github.com/apache/airflow/issues/19869#issuecomment-981255229
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] toddy86 commented on issue #19869: Custom Timetable Import Error
Posted by GitBox <gi...@apache.org>.
toddy86 commented on issue #19869:
URL: https://github.com/apache/airflow/issues/19869#issuecomment-981574073
We are experiencing this issue as well.
Copying and pasting the `AfterWorkdayTimetable` code into our plugins directory, then restarting our services yields the exact same error.
We have several other plugins in the same directory and they are loaded fine. Just these new timetables that have a serialization error. Might try your workaround for the time being @covaliov
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19869: Custom Timetable Import Error
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19869:
URL: https://github.com/apache/airflow/issues/19869#issuecomment-981524121
`__eq__` is for _instance_ equality, but the line you referenced compares the _type_ itself (i.e. the timetable class). The Python interpreter guarantees class identity, as far as I know, so the line is correct.
A possible cause to this is your `workday` module has different identities in the DAG and in the plugin. Airflow adds the plugin directory on-load, and DAG files lazily when they are parsed, so if you put the timetable module in the DAG directory, it will be reloaded each time DAGs are parsed and have different identities. This is one of the reasons the example tells you to put the timetable class in the plugin’s module—identities of modules in this directory are stable (during an interpreter session).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19869: Custom Timetable Import Error
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19869:
URL: https://github.com/apache/airflow/issues/19869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org