You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2025/05/01 05:43:04 UTC

Re: [PR] [v2-11-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

jason810496 commented on code in PR #45914:
URL: https://github.com/apache/airflow/pull/45914#discussion_r2069874691


##########
airflow/jobs/scheduler_job_runner.py:
##########
@@ -872,16 +872,17 @@ def _process_executor_events(self, executor: BaseExecutor, session: Session) ->
 
                     task_log_reader = TaskLogReader()
                     if task_log_reader.supports_read:
-                        metadata: dict[str, Any] = {}
-                        logs, metadata = task_log_reader.read_log_chunks(ti, ti.try_number, metadata)
-                        if ti.hostname in dict(logs[0]):
-                            message = str(dict(logs[0])[ti.hostname]).replace("\\n", "\n")
+                        hosts, log_streams, metadata = task_log_reader.read_log_chunks(
+                            ti, ti.try_number, None
+                        )
+                        if ti.hostname in hosts:
+                            message = str(next(iter(log_streams[0])))
                             while metadata["end_of_log"] is False:
-                                logs, metadata = task_log_reader.read_log_chunks(
+                                hosts, log_streams, metadata = task_log_reader.read_log_chunks(

Review Comment:
   > For example, in this case where the return type of task_log_reader.read_log_chunks has changed.
   
   
   I think changing the return type of `read_log_chunks` or other internal methods isn't a problem, since the user-facing interface is the REST API. As long as the REST API response remains consistent, users won’t be affected or need to be aware of the refactor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org