You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "victorrgez (via GitHub)" <gi...@apache.org> on 2023/12/31 11:51:46 UTC

[I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

victorrgez opened a new issue, #36510:
URL: https://github.com/apache/airflow/issues/36510

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.6.3
   
   ### What happened?
   
   We need to create custom triggers [1] and use them in our Production environment with Cloud Composer (`composer-2.5.3-airflow-2.6.3`). We normally work with packaged dags in zip format, which have all the required dependencies for each DAG.
   
   The issue we have is that, once our custom operators are deferred, the tasks fail since the triggerer cannot create the Custom Trigger class. After investigating, it seems our issue is related to the fact that the triggerer does not have the code from our zip files (furthermore, creating the libraries in the DAG folder outside the zip files did not help us either --> `Because the DAGs folder is not synchronized with the Airflow triggerer, the inlined trigger code is missing when the trigger is executed. ` [2])
   
   In our case, instead of the message in [2] (`ImportError: Module "PACKAGE_NAME" does not define a "CLASS_NAME" attribute/
   class`), what we get is `ModuleNotFoundError: No module named 'common'`.
   
   The only solution we have found so far is to create a Custom PyPI package and install it in the whole Cloud Composer Environment as specified in [2], however; we cannot afford to restart the whole Cloud Composer environment to update the PyPI dependencies every time we update the code, fix a bug, etc.
   
   Is there any way of being able to install/update the custom trigger code without needing an environment restart in production? Would it be possible to syncronise the DAGs folder also with the triggerer? We have not found much documentation about this so any help with be greatly appreciated!
   
   [1]:
   https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#writing-triggers
   [2]:
   https://cloud.google.com/composer/docs/composer-2/troubleshooting-triggerer#trigger_class_not_found
   
   ### What you think should happen instead?
   
   There should be an easy and quick way of updating the code of Custom Triggers without needing to restart the production environment to update a PyPI package every time a line of the code changes.
   
   ### How to reproduce
   
   1. Create a custom trigger and an operator that uses that trigger in deferrable mode (Attaching an example below)
   2. Put the code for that trigger in either a packaged DAG (zip file) or directly in the Dags folder
   3. When the task is deferred, the triggerer fails with Module not found as the class code is not synchronized with the Triggerer
   
   ### Operating System
   
   composer-2.5.3-airflow-2.6.3
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow==2.6.3
   apache-airflow-providers-google==10.10.1
   
   ### Deployment
   
   Google Cloud Composer
   
   ### Deployment details
   
   Attaching sample code from one of our triggers (censoring provider name as it is confidential):
   
   ```
   from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
   from airflow.triggers.base import BaseTrigger, TriggerEvent
   
   
   class BigQueryPartitionExistsTrigger(BaseTrigger):
   
       def __init__(self, *, project_id: str, dataset_id: str, table_id: str, partition_date: str,
                    gcp_conn_id="google_cloud_default", use_legacy_sql: bool = False,
                    location: str = "EU", defer_seconds: int = 300
                    ) -> None:
           super().__init__()
           self.project_id = project_id
           self.dataset_id = dataset_id
           self.table_id = table_id
           self.partition_date = partition_date
           self.gcp_conn_id = gcp_conn_id
           self.use_legacy_sql = use_legacy_sql
           self.location = location
           self.defer_seconds = defer_seconds
   
       def serialize(self) -> str:
           """
           Method that specifies the instructions for reconstructing the class inside Airflow Triggerer process.
           It needs to know the classpath and the list of kwargs to be provided to the `__init__` method.
   
           This trigger is not working at the moment as the DAGs folder is not synchronised with the Triggerer. You need to
           create a new package and install it inside composer with PyPI
           """
           return ("common.airflow.providers.<CENSORED>.custom_triggers.BigQueryTriggers.BigQueryPartitionExistsTrigger",
                   {"project_id": self.project_id,
                    "dataset_id": self.dataset_id,
                    "table_id": self.table_id,
                    "partition_date": self.partition_date,
                    "gcp_conn_id": self.gcp_conn_id,
                    "use_legacy_sql": self.use_legacy_sql,
                    "location": self.location,
                    "defer_seconds": self.defer_seconds})
   
       async def run(self):
           from asyncio import sleep as async_sleep
           print(f"About to create hook: {self.gcp_conn_id=}")
           bq_hook = BigQueryHook(gcp_conn_id=self.gcp_conn_id,
                                  use_legacy_sql=self.use_legacy_sql,
                                  location=self.location)
   
           while not bq_hook.table_partition_exists(
                       project_id=self.project_id,
                       dataset_id=self.dataset_id,
                       table_id=self.table_id,
                       partition_id=self.partition_date
                   ):
               await async_sleep(self.defer_seconds)
           yield TriggerEvent({"status": "success"})
   ```
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

Posted by "Ronoman (via GitHub)" <gi...@apache.org>.
Ronoman commented on issue #36510:
URL: https://github.com/apache/airflow/issues/36510#issuecomment-3961247834

   The workaround I'm currently using in composer is to use `@task.sensor`. This limits you to having time-based trigger logic instead of custom Trigger logic from local files, but it at least allows you to defer tasks and not take up worker slots if you're doing simple waiting tasks.
   
   If you want this implemented in Cloud Composer, please star and comment on https://issuetracker.google.com/issues/487349966 so that Google will implement `dags/` syncing to the triggerer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #36510:
URL: https://github.com/apache/airflow/issues/36510#issuecomment-1872929748

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

Posted by "mperesson (via GitHub)" <gi...@apache.org>.
mperesson commented on issue #36510:
URL: https://github.com/apache/airflow/issues/36510#issuecomment-2236911046

   @victorrgez did you find a solution on composer?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36510:
URL: https://github.com/apache/airflow/issues/36510#issuecomment-1872931603

   Raise this issue with composer team.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36510:
URL: https://github.com/apache/airflow/issues/36510#issuecomment-1872934038

   Just to add a bit more context:
   * https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html#basic-airflow-architecture - shows the basic architecture of Airflow where DAG folder is synchronized and used with Triggerer (this image has been updated recently in 2.8.0 but it reflects how it should work)
   
   * Triggerrer is classified as "untrusted" component - i.e one that is supposed to run the "Dag Author" code  and in the future - even in multi-tennant architecture it should read both DAG files and folder. You can see it in the next diagram https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html#separate-dag-processing-architecture - where we explained our security model and how we plan to implement tenancy isolation. Trigger will be part of the tenant and it will need access to DAG files, it will also communicate via internal API with the DB. So there is no particular reason for Trigger NOT to have access to the DAG files
   
   * When you update new Triggers in the DAG folder, you should restart Triggerer to pick those changes. This is something that should be deployment controlled - for example you should either be able to restart the triggerer manually or it should have a file watcher (if designated folders contain Trigger code provided by the user). But this is a deployment setup not airflow feature.
   
   So all-in-all: 
   
   * Composer's Triggerer should have access to DAG files
   * Composer's Triggerer should be restartable when Trigger sources change - either manually or via some kind of watching mechanism
   
   Nothing to do in Airlfow - you should raise an issue to Composer team about it. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

Posted by "victorrgez (via GitHub)" <gi...@apache.org>.
victorrgez commented on issue #36510:
URL: https://github.com/apache/airflow/issues/36510#issuecomment-1873800909

   Will do as suggested. Thank you very much for pointing us in the right direction!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #36510: Unable to use Custom Triggers in Cloud Composer in Production as Dags folder is not synchronized with the Triggerer
URL: https://github.com/apache/airflow/issues/36510


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org