You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "deepak4babu (via GitHub)" <gi...@apache.org> on 2024/09/12 12:24:49 UTC
[I] Airflow triggerer memory is increasing continously [airflow]
deepak4babu opened a new issue, #42195:
URL: https://github.com/apache/airflow/issues/42195
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.8.3
### What happened?
We started to use deferrable operators and sensors from 2 weeks back. As part of it, we started using triggerer. The triggerer container memory is increasing steadily, even when no triggers are running.
<img width="951" alt="airflow-triggerer-memory-leak-2 8 3" src="https://github.com/user-attachments/assets/74342e88-74a8-4703-a753-6204ae58e268">
### What you think should happen instead?
Memory consumption of idle triggerer should be less and does not increase
### How to reproduce
Deploy triggerer in kubernetes with airflow version 2.8.3. Monitor for 24hrs
### Operating System
kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.6.1
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.7.1
### Deployment
Other
### Deployment details
Airflow version: 2.8.3
Python version: 3.9.x
Database: Postgres 13
Kubernetes details:
Azure kubernetes service
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
k8s Server Version: v1.29.4
Dags are packaged as python packages and installed in the airflow base image. Airflow triggerer is deployed using our custom helm chart, which is just a modified version of official one.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "jasgok (via GitHub)" <gi...@apache.org>.
jasgok commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-2604179378
check the container "triggerer-log-groomer", its memory usage increases when log size increases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "arkadiuszbach (via GitHub)" <gi...@apache.org>.
arkadiuszbach commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-2633714040
Had the same issue with Triggerer and Scheduler on AKS and apache/airflow:2.10.2-python3.9:
- At first I thought it is something with Log Groomer as I didn't use it before, but even though it was disabled, memory usage was still growing and it was growing on main container not on the groomer itself.
- So then I thought maybe memory leak is within airflow processes runing in the container. I saved memory usage by all processes within container with `ps aux`, waited for two days and checked its output again - it didn't change
- Then I searched the internet for how is it possible that memory usage for the container is growing even though for the processes it is not, it was clear that something is leaking in the kernel. I found that I can check `/sys/fs/cgroup/memory.stat` for memory usage statistics within container and **I found that slab memory is growing**.
- Checked internet how I can find what is leaking in the slabs, found `slabtop` and `/proc/slabinfo` but when tried using it I got 'Permisison denied' error, so **I started the container as root user instead of airflow and when I did the memory leak was gone**
- When it was resolved when running as root i thought maybe that is because on root all linux capabilities are available and it manages memory differently, I tried enabling all capabilities when running with airflow user but it didnt fix the issue.
- So to figure out what exactly is leaking I enabled sudo within the container (to still run as airflow and be able to `sudo su`):
```
USER root
RUN usermod -aG sudo airflow
RUN echo "airflow:airflow" | chpasswd
USER airflow
```
- Then wrote bash script to take dumps of /proc/slabinfo, didn't wait for long, but seemed that **dentry memory usage was growing**, checked the internet, chatgpt and it said that it may be growing when there are a lot of directory operations and files operations with dynamic names. Somehow I stumbled into `strace` which can display system calls specific process is making, i.e. open file, create file, directories system calls, memory allocation, etc. So I started checking Airflow processes with strace (` strace -T -e trace=desc,open,mkdir,rmdir,unlink,rename --decode-fds=all -p {process_id})` to verify if they are making system calls related with files and directories - they didn't, except **liveness probe which was making a lot of file related system calls as it is started over and over again, so I disabled it in helm chart and memory leak was gone:**
```
livenessProbe:
command:
- "/bin/true"
```
- I went through strace output a little bit for liveness probe and messages like below got my attention:
```
Line 19609: mkdir("/opt/airflow/config/__pycache__", 0777) = -1 EACCES (Permission denied) <0.000028>
Line 22139: openat(AT_FDCWD</opt/airflow>, "/usr/local/lib/python3.9/concurrent/futures/__pycache__/process.cpython-39.pyc.140482203662256", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0644) = -1 EACCES (Permission denied) <0.000035>
Line 22155: openat(AT_FDCWD</opt/airflow>, "/usr/local/lib/python3.9/multiprocessing/__pycache__/queues.cpython-39.pyc.140482203662768", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0644) = -1 EACCES (Permission denied) <0.000033>
Line 22622: openat(AT_FDCWD</opt/airflow>, "/usr/local/lib/python3.9/__pycache__/sched.cpython-39.pyc.140482203590416", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0644) = -1 EACCES (Permission denied) <0.000035>
```
There are only few entries like this, but maybe this somehow is causing the leak as airflow user does not have access, don't know, files names seems dynamic.
Got an idea to do chmod -R 777 on /opt/airflow/config and /usr/local/lib/python3.9 to verify, but didn't want to spend more time on this as **seems like python leaves something behind when started over and over again** and the liveness probe in python always felt too heavy to me(i can see increase in CPU by 20% of one vcore and memory usage by 100mb-200mb when it starts, it takes 4-5 secs to execute), so I rewrote it to golang instead with chatgpt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3599576041
As described in https://github.com/apache/airflow/issues/58509#issuecomment-3599568800 - an easy way to check it with minimum number of changes will be to either run the container with root, or recursively change `/usr/local/lib` to be group writable (and owned by group 0) .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "rurouni-najeeb (via GitHub)" <gi...@apache.org>.
rurouni-najeeb commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3450802257
Hi 👋
I was wondering if this issue is still open or has been resolved in version 3.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "arkadiuszbach (via GitHub)" <gi...@apache.org>.
arkadiuszbach commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3596781641
I checked the following Docker images:
- apache/airflow.1.10.15
- apache/airflow:2.2.0
- apache/airflow:2.10.2-python3.12
- apache/airflow:3.1.3-python3.12
- apache/airflow:3.0.2-python3.12
In version 1.10.15, I noticed that some .pyc files were present under /usr/local/lib/python3.xx, but the other images didn’t have any .pyc files at all.
Initially, I thought the Airflow 3 images contained some .pyc files, but it turns out they don’t. It’s possible that .pyc files were generated only when I was running the container as root, since once the scheduler starts, it may compile the modules at runtime.
While Python normally generates .pyc files during make install, it looks like these files are immediately deleted afterward in the Airflow images:
https://github.com/apache/airflow/blob/8545d3c578af5e254e230e38f8a5035fac045342/Dockerfile#L427
I also checked some of the official Python images that Airflow was previously based on, and they include the same logic to delete .pyc files after the make step. This behavior goes back at least eight years — it may have been there since the very beginning:
https://github.com/docker-library/python/blob/7891fcc63609dfd43ae2e8de13a40e838b5e7608/3.6/stretch/slim/Dockerfile#L114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "boris-tumbev (via GitHub)" <gi...@apache.org>.
boris-tumbev commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-2470455949
> @deepak4babu there are some optimizations made around this area, it will be available in coming airflow release 2.10.3. FYI #40487.
It does not fix the problem unfortunately, in our setup the memory still get higher and higher and the triggerer eventually restarts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3453700274
> Hi 👋 I was wondering if this issue is still open or has been resolved in version 3.x?
We did remove **some** memory issues in 3.1.1 and **some** are still to be fixed in 3.1.2 (for example https://github.com/apache/airflow/pull/57334 ) - so ... maybe,... I think likely answer can be given by those who upgrade and observed it before. Highly recomended @rurouni-najeeb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-2505060314
> > @deepak4babu there are some optimizations made around this area, it will be available in coming airflow release 2.10.3. FYI #40487.
>
> It does not fix the problem unfortunately, in our setup the memory still get higher and higher and the triggerer eventually restarts
Can you do some memory check using https://github.com/bloomberg/memray for example and see what causes the memory leak?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3590394654
I wonder how it happens that we do not have .pyc files in there. I wonder if the new way we are building docker images, will be a different. Docker images of ours as of recently are not using the `python` images, but we compile and install Python on our own. Maybe we can check if the new image has all .pyc files @arkadiuszbach ?
It should be possible to build generally even older python versions using the latest Dockerfile - > https://airflow.apache.org/docs/docker-stack/build.html#customizing-the-image -> maybe you can try this one :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-2633749046
HEy @arkadiuszbach -> thanks A LOT for such a detailed investigation. This is a great find and I think that one is fully actionable on our side - we might want to rethink on how our liveness probe is implemented and recommended in our helm chart. You are a hero!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3602470385
Regardles from the tests. I tested how much the size of the image increases if we leave .pyc and .pyo files .... And it's <0.5%
So I went ahead and created https://github.com/apache/airflow/pull/58944 and we should merge it for 3.1.4 which is about to be released. I will provisionally close those as "fixed" but please double-check when we have RC or final release and conirm (or not- then I will reopen it).
Very, very good analysis @arkadiuszbach !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3590336907
WHOA... What a deep dive... I will take a look shortly. That needs a focused deep-dive :). cc: @ashb ... Another interesting case for deep engineering dive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-2453363000
@deepak4babu there are some optimizations made around this area, it will be available in coming airflow release 2.10.3. FYI #40487. Please check https://github.com/apache/airflow/issues/43441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "arkadiuszbach (via GitHub)" <gi...@apache.org>.
arkadiuszbach commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3586845274
I gave it another shot and i think was able to figure this out
This is happening: https://lwn.net/Articles/814535/:
- airflow user does not have permission to write into __pycache__ subdirectories under: `/usr/local/lib/python3.12/`
- for most of the libararies there are already precompiled .pyc files, but if there isn't then when importing/starting, python will try creating such
- first it checks if compiled .pyc file exists, if so, it tries to read it:
`openat(AT_FDCWD</opt/airflow>, "/usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000152>`
- if it didnt exists then creates the .pyc file with number suffix after .pyc, some intermediate compiled file (**This creates negative dentry**):
`openat(AT_FDCWD</opt/airflow>, "/usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc.136405097451568", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0644) = 3</usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc.136405097451568> <0.004813>
`
- then it renames the file (removes the number suffix), probably so that next runs of other python processes won't have to compile the same file again when importing
```
newfstatat(3</usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc.136405097451568>, "", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0 <0.000136>
write(3</usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc.136405097451568>, "\313\r\r\n\0\0\0\0\277\306Hh\374\26\0\0\343\0\0\0\0\0\0\0\0\0\0\0\0\6\0\0"..., 5781) = 5781 <0.000161>
close(3</usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc.136405097451568>) = 0 <0.000091>
rename("/usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc.136405097451568", "/usr/local/lib/python3.12/encodings/__pycache__/__init__.cpython-312.pyc") = 0 <0.000239>
```
`__init__.cpython-312.pyc.136405097451568` was renamed to `__init__.cpython-312.pyc`
- airflow user is not able to create the file, each time healthprobe is triggered python generates different intermediate name for the .pyc file, each gets its entry in the Dentries, hence the slow memory increase
Possible fixes:
- disable precompilation entirely via https://docs.python.org/3/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE - but the probe is already heavy, so doesn't seem like a good idea
- precompile the files within Docker image: sudo python3 -m compileall /usr/local/lib/python3.12
- give airflow user write permission to all of the __pycache__ directories under /usr/local/lib/python3.12
The commands i used to figure this out:
- `cat /sys/fs/cgroup/memory.stat | grep slab` - container slab memory usage (dentry is part of slab)
- `cat /proc/sys/fs/dentry-state` - https://www.kernel.org/doc/html/v6.6/admin-guide/sysctl/fs.html#dentry-state, used to track negative dentries count
- `strace -T -e trace=desc,open,mkdir,rmdir,unlink,rename --decode-fds=all python` - display file operations by command, search for dynamic file names
- trace dentries add/kill requests via bpftrace
```
# required by bpftrace
sudo mkdir -p /sys/kernel/tracing
sudo mount -t tracefs nodev /sys/kernel/tracing
sudo mkdir -p /sys/kernel/debug
sudo mount -t debugfs nodev /sys/kernel/debug
sudo bpftrace -e '
#define TARGET_COMM "python"
kprobe:d_alloc
/(comm == TARGET_COMM)/
{
$dentry = (struct dentry *)arg0;
$parent_name = str($dentry->d_parent->d_name.name);
$parent_parent_name = str($dentry->d_parent->d_parent->d_name.name);
@add_process[comm] = count();
printf("ADD | PARENT: %s/%s | FILE: %s | Negative?: %p | Dentry Addr: %p | Caller: %s(%d)\n",
$parent_parent_name,
$parent_name,
str($dentry->d_name.name),
$dentry->d_inode,
arg0,
comm,
pid);
}
kprobe:__dentry_kill
/(comm == TARGET_COMM)/
{
$main = (struct dentry *)arg0;
@delete_process[comm] = count();
printf("KILL %s; negative?: %p; a: %p; caller: %s(%d)\n",
str($main->d_name.name), $main->d_inode, arg0, comm, pid);
}
'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42195:
URL: https://github.com/apache/airflow/issues/42195#issuecomment-3590399708
https://github.com/apache/airflow/blob/main/Dockerfile#L332 -s where our installation code for Python is :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow triggerer memory is increasing continously [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #42195: Airflow triggerer memory is increasing continously
URL: https://github.com/apache/airflow/issues/42195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org