Re: [I] Large string logs from trigger are not written to log file intermittently. [airflow]

["mrvarmazyar (via GitHub)" <gi...@apache.org>]

mrvarmazyar commented on issue #66158:
URL: https://github.com/apache/airflow/issues/66158#issuecomment-4807416452

   @tirkarthi We can reproduce this consistently on Airflow 3.2.1 with KubernetesExecutor. Here's how:
   
   Setup:
   - Executor: KubernetesExecutor
   - Triggerer: 1 replica
   - Remote logging: GCS enabled
   - Deferrable operator pattern: task defers to a custom trigger that polls an XCom key written by an upstream task (watcher pattern)
   
   Reproduction steps:
   1. Create a deferrable task that calls `self.defer(trigger=MyTrigger(...))` mid-execution
   2. Open the Airflow UI → task log while the task is in deferred state
   3. Observe: logs stop exactly at the `self.defer()` call, nothing from the triggerer polling phase appears
   
   What you see:
   - Task log shows initial setup lines, then cuts off at deferral
   - No triggerer polling output visible in UI during deferred state
   - After task completes: logs remain cut off (triggerer phase still missing)
   - Task `state=success` in DB, normal execution duration, the task DID run correctly
   
   What makes it consistent for us:
   - Happens on every run, not intermittent
   - Triggerer pod is healthy, port 8794 is exposed and log server is running (verified via `curl http://<triggerer-pod-ip>:8794`)
   - GCS log files exist but the triggerer portion (`.log.trigger.*.log`) is absent or empty
   
   **My hypothesis:** the socket pipeline between triggerer subprocess and structlog is dropping messages before they reach the file writer, consistent with what this issue describes. The missing file on GCS (not just missing in UI) confirms it's not a rendering issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org