You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2025/03/05 21:20:11 UTC

[I] EXECUTE_TASKS_NEW_PYTHON_INTERPRETER setting no play nice with elasticsearch [airflow]

dstandish opened a new issue, #47420:
URL: https://github.com/apache/airflow/issues/47420

   ### Body
   
   Elasticsearch (when configured to output json to stdout) requires, naturally, that the logs are sent as json to stdout.
   
   Currently when running with  `EXECUTE_TASKS_NEW_PYTHON_INTERPRETER` set to true, we use `check_output`, and we do nothing with the output.
   
   See `celery_executor_utils.py` method `_execute_in_subprocess`.
   
   I have a POC hack to "fix" this for specifically this use case:
   
   ```
   def _run_and_stream(cmd, env):
       import subprocess
   
       process = subprocess.Popen(
           cmd,
           stdout=sys.__stdout__,
           stderr=sys.__stderr__,
           text=True,
           env=env,
           close_fds=True,
       )
   
       while True:
           if process.poll() is not None:
               break
           time.sleep(.5)
   
       return process.poll()
   
   
   def _execute_in_subprocess(command_to_exec: CommandType, celery_task_id: str | None = None) -> None:
       env = os.environ.copy()
       if celery_task_id:
           env["external_executor_id"] = celery_task_id
       try:
           _run_and_stream(command_to_exec, env=env)
       except subprocess.CalledProcessError as e:
           log.exception("[%s] execute_command encountered a CalledProcessError", celery_task_id)
           log.error(e.output)
           msg = f"Celery command failed on host: {get_hostname()} with celery_task_id {celery_task_id}"
           raise AirflowException(msg)
   ```
   
   But testing would need to be done to ensure it doesn't have unintended consequences or break other configurations.
   
   In that snippet I use `__stdout__` instead of `stdout` because the latter is replaced by a logging proxy in the celery context, which introduces a prefix `ForkPoolWorker-1` or sometihng, which would make it no longer ndjson, which I suspect might prevent the line from being parsed properly.
   
   Anyway, this is a path to pursue, if it is valuable enough to make this combination work.
   
   Last note, there's already some similar code in standard task runner if you look at CAN_FORK -- it's used when the OS cannot fork.
   
   ### Committer
   
   - [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] EXECUTE_TASKS_NEW_PYTHON_INTERPRETER setting no play nice with elasticsearch [airflow]

Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 closed issue #47420: EXECUTE_TASKS_NEW_PYTHON_INTERPRETER setting no play nice with elasticsearch
URL: https://github.com/apache/airflow/issues/47420


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org