You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kiranzo (via GitHub)" <gi...@apache.org> on 2025/05/21 19:00:38 UTC
[D] How to setup Airflow to use local file system for XComs? [airflow]
GitHub user kiranzo created a discussion: How to setup Airflow to use local file system for XComs?
My XComs sometimes exceed 1 Gb of data, so I needed a custom backend. At first, I tried to use Minio S3 backend, but I inexplicably got 'This class is not fork-safe' ffspec error when doing xcom.push - I didn't use any multiprocessing, and it doesn't reproduce when I copy task code to a test DAG (setting mp_start_method to 'spawn' didn't work, either). So I tried using local file system, like it says [here](https://airflow.apache.org/docs/apache-airflow-providers-common-io/stable/xcom_backend.html), that's my docker-compose now:
```
AIRFLOW__CORE__XCOM_BACKEND: 'airflow.providers.common.io.xcom.backend.XComObjectStorageBackend'
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH: 'file://xcoms'
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD: 0
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION: 'gzip'
```
First error I got was that Airflow Scheduler hanged forever because it couldn't find gzipped XCom file from the previous task
`FileNotFoundError: [Errno 2] No such file or directory '/opt/airflow/xcoms/mydag/manual__2025-05-21T16:10:47+00:00/mytask/0abd8fcf-e171-48ba-9eff-ad47168bcac5.gz'` because Worker wrote the file in its own Docker container, and Scheduler lives in another one.
So I need to create a shared volume '/opt/airflow/xcoms' for all Airflow services, right? Except now I have 'Permission denied' error when it tries to write a directory with a DAG name inside of /opt/airflow/xcoms.
My Docker file currently looks something like this:
```
### extended_airflow:1.6.0
FROM apache/airflow:2.10.5-python3.10
COPY requirements.txt /requirements.txt
RUN pip install --upgrade pip
RUN pip install "cython<3.0.0" wheel
RUN pip install --no-cache-dir -r /requirements.txt
USER root
# apt install extra utils here
RUN rm /requirements.txt
RUN mkdir /opt/airflow/xcoms
RUN chown -R airflow:0 /opt/airflow/xcoms
VOLUME /opt/airflow/xcoms
USER airflow
```
And I have that volume in docker-compose.yml:
```
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./xcoms:/opt/airflow/xcoms
```
`chown` or even `chmod 777` don't work, so how should this whole setup look for filesystem XComs to work?
(Also, I wouldn't mind help with S3 backend, but I doubt the issue is easy to reproduce).
GitHub link: https://github.com/apache/airflow/discussions/50926
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] How to setup Airflow to use local file system for XComs? [airflow]
Posted by "kiranzo (via GitHub)" <gi...@apache.org>.
GitHub user kiranzo added a comment to the discussion: How to setup Airflow to use local file system for XComs?
[Okay, tl;dr](https://stackoverflow.com/a/72343078) - when I used bind mounted volume, `docker compose` took over as a `root` user and overwrote whatever definitions I made in the Docker file, so it had permissions 755, and `airflow` user wasn't allowed to write there. I solved it by turning my XComs directory into a named volume, so the way I created the XCom directory remained the same during the container creation.
My Docker file ended up looking like this:
```
### extended_airflow:1.6.0
FROM apache/airflow:2.10.5-python3.10
COPY requirements.txt /requirements.txt
RUN pip install --upgrade pip
RUN pip install "cython<3.0.0" wheel
RUN pip install --no-cache-dir -r /requirements.txt
USER root
# apt install extra utils here
RUN rm /requirements.txt
RUN mkdir -m 777 /opt/airflow/xcoms
RUN chown -R airflow:0 /opt/airflow/xcoms
USER airflow
```
And my [docker-compose from the official documentation](https://airflow.apache.org/docs/apache-airflow/2.10.5/docker-compose.yaml) had these changes:
```
x-airflow-common:
...
environment:
AIRFLOW__CORE__XCOM_BACKEND: 'airflow.providers.common.io.xcom.backend.XComObjectStorageBackend'
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH: 'file:///opt/airflow/xcoms'
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD: 100000000
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION: 'gzip'
...
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- xcoms:/opt/airflow/xcoms
...
volumes:
airflow-cache-volume:
airflow-protected-volume:
postgres-db-volume:
git-sync-volume:
xcoms:
```
GitHub link: https://github.com/apache/airflow/discussions/50926#discussioncomment-13249843
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] How to setup Airflow to use local file system for XComs? [airflow]
Posted by "kiranzo (via GitHub)" <gi...@apache.org>.
GitHub user kiranzo added a comment to the discussion: How to setup Airflow to use local file system for XComs?
Still, I'd like to know if anyone else received 'This class is not fork-safe' error when trying to push to S3 storage.
GitHub link: https://github.com/apache/airflow/discussions/50926#discussioncomment-13249880
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org