You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "ecodina (via GitHub)" <gi...@apache.org> on 2026/04/27 11:12:17 UTC

[I] Triggerer not starting [airflow]

ecodina opened a new issue, #65945:
URL: https://github.com/apache/airflow/issues/65945

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.2.1
   
   ### What happened and how to reproduce it?
   
   I upgraded from 3.1.8 to 3.2.1 and the Triggerer didn't start properly for my prod instance. 
   
   My staging instance worked well (and has worked well for over a week). It uses the same exact Docker image as prod, and the same airflow.cfg (except the value of parallelism).
   
   The triggerer was sending the heartbeat, but the logs only showed:
   `[info] Starting the triggerer [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:201`
   
   I changed the log level to debug and this error appeared. All other logs looked as if it was working correctly:
   
   ```
   Stack (most recent call last):
     File "/usr/python/lib/python3.12/threading.py", line 1032, in _bootstrap
       self._bootstrap_inner()
       self.run()
     File "/usr/python/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
     File "/usr/python/lib/python3.12/threading.py", line 1012, in run
       self._target(*self._args, **self._kwargs)
     File "/usr/python/lib/python3.12/asyncio/base_events.py", line 645, in run_forever
       self._run_once()
     File "/usr/python/lib/python3.12/asyncio/events.py", line 88, in _run
       handle._run()
     File "/usr/python/lib/python3.12/asyncio/base_events.py", line 1999, in _run_once
       self._context.run(self._callback, *self._args)
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 353, in start_lifespan
       await cm.enter_async_context(app.router.lifespan_context(app))
     File "/usr/python/lib/python3.12/contextlib.py", line 659, in enter_async_context
       result = await _enter(cm)
     File "/usr/python/lib/python3.12/contextlib.py", line 210, in __aenter__
       return await anext(self.gen)
     File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 216, in merged_lifespan
       async with original_context(app) as maybe_original_state:
     File "/home/airflow/.local/lib/python3.12/site-packages/svcs/fastapi.py", line 66, in __call__
       async with self.registry, cm(app, self.registry) as state:
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 96, in lifespan
       registry.register_factory(JWTGenerator, _jwt_generator)
     File "/home/airflow/.local/lib/python3.12/site-packages/svcs/_core.py", line 294, in register_factory
     File "/usr/python/lib/python3.12/logging/__init__.py", line 1527, in debug
       log.debug(
       self._log(DEBUG, msg, args, **kwargs)
     File "/usr/python/lib/python3.12/logging/__init__.py", line 1684, in _log
     File "/usr/python/lib/python3.12/logging/__init__.py", line 999, in format
     File "/usr/python/lib/python3.12/logging/__init__.py", line 1160, in emit
       msg = self.format(record)
       self.emit(record)
     File "/usr/python/lib/python3.12/logging/__init__.py", line 1028, in handle
     File "/usr/python/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
       self.callHandlers(record)
     File "/usr/python/lib/python3.12/logging/__init__.py", line 1700, in handle
       self.handle(record)
       hdlr.handle(record)
       return fmt.format(record)
   ```
   
   Airflow sent the tasks to the triggerer, but the trigger code wasn't executed.
   
   I had to downgrade to 3.1.8, which left the task instances and dag runs corrupted, so I had to recover the backup.
   
   ### Environment
   
   Deployed on docker swarm using the official image (with some requirements installed)
   We use LocalExecutor
   
   
   
   ### What you think should happen instead?
   
   _No response_
   
   ### Operating System
   
   _No response_
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   All latest versions existing the 27/april/2026
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   The Dockerfile installs the following requirements:
   
   ```
   apache-airflow-client
   apache-airflow-core[gunicorn]>=3.2.0
   apache-airflow[async,graphviz,otel,standard]
   apache-airflow[postgres,ftp,http,imap,sftp,smtp,ssh,redis,keycloak,git]
   ```
   
   It also installs our provider, that contains the deferrable operator code and custom logging:
   
   log_config.py:
   ```
   LOGGING_CONFIG["handlers"]["task"] = {
       "class": "example.airflow.provider.logging.example_logs.SlurmSharedFileTaskHandler",
       "formatter": "airflow",
       "base_log_folder": BASE_LOG_FOLDER,
       "filters": ["mask_secrets_core"],
   }
   ```
   The `SlurmSharedFileTaskHandler` inherits from `FileTaskHandler` and `LoggingMixin` and only overwrites the `_read()` method (which I believe it is only used from the api server).
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org