You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2025/07/11 03:46:30 UTC

[PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

jason810496 opened a new pull request, #53167:
URL: https://github.com/apache/airflow/pull/53167

   
   related: #49470
   
   ## What
   
   Backport [Resolve OOM When Reading Large Logs in Webserver #49470](#49470) to `v3-0-test` branch.
   
   The only conflict when cherry-pick:
   ```
   Merge Changes
   └── airflow-core/src/airflow
       ├── api_fastapi/core_api/routes/public
       │   └── log.py                   [9+, !]
       ├── utils/log
       │   ├── file_task_handler.py     [4, !]
       │   └── log_reader.py            [!]
   └── providers
       ├── amazon/tests/unit/amazon/aws/log
       │   └── test_cloudwatch_task_handler.py [!]
       ├── elasticsearch/src/airflow/providers/elasticsearch/log
       │   └── es_task_handler.py       [!]
       └── opensearch/src/airflow/providers/opensearch/log
           └── os_task_handler.py      [!]
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.

jason810496 closed pull request #53167: [v3-0-test] Resolve OOM When Reading Large Logs in Webserver
URL: https://github.com/apache/airflow/pull/53167


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.

jason810496 commented on PR #53167:
URL: https://github.com/apache/airflow/pull/53167#issuecomment-3065543055

> > @jason810496 is backport something went wrong? could see many files
>
> Fixed it

Thanks @kaxil!
I just check this backport PR modify the same files by the following script.
```bash
REPO=apache/airflow
PR1=53167
PR2=49470

# Get the list of changed files from each PR
FILES_PR1=$(gh pr view "$PR1" --repo "$REPO" --json files --jq '.files[].path' | sort)
FILES_PR2=$(gh pr view "$PR2" --repo "$REPO" --json files --jq '.files[].path' | sort)

TMP1=$(mktemp)
TMP2=$(mktemp)

echo "$FILES_PR1" > "$TMP1"
echo "$FILES_PR2" > "$TMP2"

# Compare the two sorted lists
if cmp -s "$TMP1" "$TMP2"; then
echo "✅ PR #$PR1 and PR #$PR2 modify the exact same set of files."
else
echo "❌ PR #$PR1 and PR #$PR2 modify different sets of files."

echo -e "\n📄 Files only in PR #$PR1:"
comm -23 "$TMP1" "$TMP2"

echo -e "\n📄 Files only in PR #$PR2:"
comm -13 "$TMP1" "$TMP2"
fi

rm -f "$TMP1" "$TMP2"
```

I have also test the behavior using `breeze k8s` with the following setup, the first 2 work well

1. kubernetes executor
2. celery executor
3. kubernetes executor + remote logging with Google Cloud Storage
4. celery executor + remote logging with Google Cloud Storage

but the setup** with remote logging with Google Cloud Storage result in errors**.

I just found the root cause and will raise another PR to fix it, mark this PR to draft to void mislead merging.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk commented on PR #53167:
URL: https://github.com/apache/airflow/pull/53167#issuecomment-3070652979

   cool!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.

jason810496 commented on PR #53167:
URL: https://github.com/apache/airflow/pull/53167#issuecomment-3065862741

   After configuring the remote logging setup correctly, all the behavior of getting task instance logs page work well.
   Ready to merge if this CI run is green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.

jason810496 merged PR #53167:
URL: https://github.com/apache/airflow/pull/53167


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.

gopidesupavan commented on PR #53167:
URL: https://github.com/apache/airflow/pull/53167#issuecomment-3063937841

   @jason810496 is backport something went wrong? could see many files 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-0-test] Resolve OOM When Reading Large Logs in Webserver [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.

kaxil commented on PR #53167:
URL: https://github.com/apache/airflow/pull/53167#issuecomment-3063961038

   > @jason810496 is backport something went wrong? could see many files
   
   Fixed it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org