You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/28 18:51:36 UTC
[GitHub] [airflow] szymonorz opened a new issue, #26024: Triggerer service doesn't reload custom triggers after changes to their code.
szymonorz opened a new issue, #26024:
URL: https://github.com/apache/airflow/issues/26024
### Apache Airflow version
main (development)
### What happened
After updating a custom trigger the triggerer service is still using the cached version and requires a restart to load the updated trigger.
### What you think should happen instead
There should be some sort of mechanism that determines whether the trigger has changed and then update the `trigger_cache`.
### How to reproduce
1. Create any custom trigger based on BaseTrigger
Sample code
```python
from airflow.triggers.base import BaseTrigger, TriggerEvent
from airflow.utils.decorators import apply_defaults
import asyncio
class CustomTrigger(BaseTrigger):
@apply_defaults
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def serialize(self):
return ("test.custom_trigger.CustomTrigger", {})
async def run(self):
while True:
from random import randint
number = randint(0,9)
if number % 2:
yield TriggerEvent({"status":"success", "message": "Number {0} is odd".format(number)})
break
else:
await asyncio.sleep(10)
```
2. Create a Deferrable Operator using this trigger
3. Run the task once (Not sure if it has to be successful)
4. Change the code of the custom trigger you've made without changing its name
```python
async def run(self):
while True:
from random import randint
number = randint(0,9)
if not number % 2:
yield TriggerEvent({"status":"success", "message": "Number {0} is even".format(number)})
break
else:
await asyncio.sleep(10)
```
5. Run the same task again. It should now only succeed if the number is even, but the trigger never gets updated in the triggerer so it still succeeds for odd numbers.
The use of `random` is not necessary. You can even just change have a simple trigger that only yields TriggerEvent and change the message after run.
Logs:
```shell
(airflow) airflow | > airflow triggerer
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
//Before changes
[2022-08-28 20:44:59,833] {triggerer_job.py:100} INFO - Starting the triggerer
[2022-08-28 20:45:09,856] {triggerer_job.py:358} INFO - Trigger <test.custom_trigger.CustomTrigger > (ID 37) starting
[2022-08-28 20:45:19,857] {triggerer_job.py:361} INFO - Trigger <test.custom_trigger.CustomTrigger > (ID 37) fired: TriggerEvent<{'status': 'success', 'message': 'Number 5 is odd'}>
//After changes
[2022-08-28 20:46:36,967] {triggerer_job.py:358} INFO - Trigger <test.custom_trigger.CustomTrigger > (ID 38) starting
[2022-08-28 20:46:36,967] {triggerer_job.py:361} INFO - Trigger <test.custom_trigger.CustomTrigger > (ID 38) fired: TriggerEvent<{'status': 'success', 'message': 'Number 3 is odd'}>
```
### Operating System
Arch Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-sqlite==3.2.0
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
Not sure if it is a bug or an intended feature. Couldn't find anything in the documentation.
By looking at the code I think the method that is at fault is [triggerer_job#get_trigger_by_classpath](https://github.com/apache/airflow/blob/aa877637f40ddbf3b74f99847606b52eb26a92d9/airflow/jobs/triggerer_job.py#L416)
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26024: Triggerer service doesn't reload custom triggers after changes to their code.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26024:
URL: https://github.com/apache/airflow/issues/26024#issuecomment-1233442804
Following discussion in https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1661974787629479 -> this is extremely hard (and likely very difficult to hot-reload custom trigger code and we chose not to do it at least for now)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26024: Triggerer service doesn't reload custom triggers after changes to their code.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26024:
URL: https://github.com/apache/airflow/issues/26024#issuecomment-1229530060
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26024: Triggerer service doesn't reload custom triggers after changes to their code.
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26024: Triggerer service doesn't reload custom triggers after changes to their code.
URL: https://github.com/apache/airflow/issues/26024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26024: Triggerer service doesn't reload custom triggers after changes to their code.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26024:
URL: https://github.com/apache/airflow/issues/26024#issuecomment-1233438372
cc: @andrewgodwin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org