You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/29 17:07:42 UTC
[GitHub] [airflow] ImadYIdrissi opened a new issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
ImadYIdrissi opened a new issue #17320:
URL: https://github.com/apache/airflow/issues/17320
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.1.0
**Environment**:
- **Cloud provider or hardware configuration**: GCP Compute Engine - e2-standard-4 (4 vCPUs, 16 GB memory)
- **OS** (e.g. from /etc/os-release): Ubuntu 18.04
- **Kernel** (e.g. `uname -a`): `Linux lamachine-preprod 5.4.0-1049-gcp #53~18.04.1-Ubuntu SMP Thu Jul 15 11:32:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux`
**What happened**:
When trying to run `$ sudo docker-compose run airflow-init bash`
```
Creating be-api_airflow-init_run ... done
....................
ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 34, in <module>
from airflow import settings
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 35, in <module>
from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf # NOQA F401
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 1115, in <module>
conf = initialize_config()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 836, in initialize_config
with open(AIRFLOW_CONFIG, 'w') as file:
PermissionError: [Errno 13] Permission denied: '/home/airflow/airflow.cfg'
ERROR: 1
```
**What you expected to happen**:
I expected to see a correct initialization of the container with the proper file permissions for the specified UID in the `docker-compose.yml` file, with an output that resembles this :
```
Creating be-api_airflow-init_run ... done
BACKEND=postgresql+psycopg2
DB_HOST=postgres
DB_PORT=5432
DB: postgresql+psycopg2://airflow:***@postgres/airflow
[2021-07-29 16:25:03,687] {db.py:695} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
Upgrades done
airflow already exist in the db
airflow@7a15c956e187:/opt/airflow$ cd /home/airflow/
airflow@7a15c956e187:~$
```
P.S : This output is achieved by using the `UID=50000` within the `.env` file that is attached to the `docker-compose.yml` file.
<!-- What do you think went wrong? -->
When using a different UID (i.e 1001 in my case), in order to match the file permissions for the `./dags`, `./logs`, `./plugins`, the error occurs. I think the `UID=50000` was enforced at some point in the `DockerFile` of the Airflow image, and is not correctly substituted when `docker-compose.yml` tries to change this value, so the `/home/airflow` files are still created with owner as `UID:50000` while the sub-directories `./dags`, `./logs`, `./plugins` will have the UID/GID of the host system.
There are 2 major issues with the approach of using a fixed UID:
1. If I have to create and use a single `UID=50000` that will handle all airflow operations, then my airflow file system within the host cannot be operated properly with different users, e.g. devs when pulling new changes from git...
2. Even if this works properly and we can use another UID than 50000, it still restricts the actions to a singular user, that is binded with the `GID=0` (This is a requirement from airflow). The result is that we will have the same limitation as mentionned earlier, i.e. only 1 UID will be able to change the host file system. (Maybe I need to create a separate issue for this)
**How to reproduce it**:
Create a project with the following structure
```
custom-project
┣ src
┃ ┣ dags
┃ ┃ ┗ hello_geeks.py
┃ ┣ logs
┃ ┗ plugins
┣ .env
┣ README.md
┗ docker-compose.yml
```
Use the following files with the following command `sudo docker-compose run airflow-init bash`
docker-compose.yml file :
```yaml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:|version|
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
# Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
# Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW_HOME: '${AIRFLOW_HOME:-/opt/airflow}'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./src/dags:${AIRFLOW_HOME:-/opt/airflow}/dags
- ./src/logs:${AIRFLOW_HOME:-/opt/airflow}/logs
- ./src/plugins:${AIRFLOW_HOME:-/opt/airflow}/plugins
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflow"]
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
expose:
- 6379
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 9999:8080
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
interval: 10s
timeout: 10s
retries: 5
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
airflow-scheduler:
<<: *airflow-common
command: scheduler
healthcheck:
test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
interval: 10s
timeout: 10s
retries: 5
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- "CMD-SHELL"
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
airflow-cli:
<<: *airflow-common
profiles:
- debug
environment:
<<: *airflow-common-env
CONNECTION_CHECK_MAX_COUNT: "0"
# Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
command:
- bash
- -c
- airflow
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
interval: 10s
timeout: 10s
retries: 5
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
volumes:
postgres-db-volume:
```
.env file :
```
AIRFLOW_UID=1001
AIRFLOW_GID=0
AIRFLOW_HOME=/home/airflow
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed. The causes of this issue are still not clearly identified, or confirmed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and trying to understand it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
I thought I did set `AIRFLOW_UID` to 1001 and `AIRFLOW_GID` to 0 (manually) in the `.env` file.
Is it not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ?
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001). I hope I am not badly mistaken about this, but if so, could you kindly clarify the misconception ?
> The solution you copied is to manually build image and there **manually** your UID as the user.
Could you clarify what you mean by **manually**? Is it because I put it in the `.env` myself? As I have asked earlier, wouldn't the above mentionned command do just that?
I will follow your recommendation about adding the user to the docker group to avoid using sudo. Thank you.
P.S : Please find the `.env` file content under the `docker-compose.yml` which was enclosed in the initial post, but might have been easily missed as the main content is bigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and trying to understand it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
OK so setting my UID manually in the `.env` file is not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ??
Wouldn't that just relace my `.env` file and remove my `AIRFLOW_HOME`?? plus my UID (1001) is already set within it? BTW the GID is already set to 0 **as a group** within it aswell...
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001).
> The solution you copied is to manually build image and there **manually** your UID as the user.
I don't understand what you mean by **manually**, is it because I put it in the `.env` myself? Wouldn't your command do just that?
P.S : Please find the `.env` file content under the `docker-compose.yml` which was enclosed in the initial post, it might have been easily missed as the main content is bigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and challenging it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
OK so setting my UID manually in the `.env` file is not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ??
Wouldn't that just relace my `.env` file and remove my `AIRFLOW_HOME`?? plus my UID (1001) is already set within it? BTW the GID is already set to 0 **as a group** within it aswell...
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001).
> The solution you copied is to manually build image and there **manually** your UID as the user.
I don't understand what you mean by **manually**, is it because I put it in the `.env` myself? Wouldn't your command do just that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889886049
The reason it was closed, because you indicated that you expect that things will work out-of-the-box even when you indicated yourself you tried to change users (first 50000 and then 1001) and that it has changed since previous time, which indicated that you expected more from the docker compose than what it is intended to.
The "quick start" is really to get you quick-started and if you want to change anything (like change the user and experiment with the setting), there is not much we can do to "solve" the issue you raised as bug. From your message it seems that you had a history of using this setup and that you "expect" it to work under this different circumstances (one of the problems was that you used sudo to run the docker compose which is guaranteed not to work).
The Quick-start documentation is just that - quick-start. No more, no less. You have to strictly follow it to get it working, if you do any deviation from that, it might or might not work. When you open an issue indicating "bug" I think you expect it to be fixed. There is no action anyone can take here to "fix" the problems you were experiencing. That's why the issue was closed - because there is no reasonable action anyone can take here to fix the problems you experienced.
The best thing You can do is to wipe out all your setup and start from scratch and precisely follow the quick-start with no deviations and see if it works. If it does not, please report it here with all information that you can and I will be happy to reopen the issue (if it willl indicate that our quick start instructions are wrong. Or even better - open a PR fixing it straight away.
If you want to discuss the usage and results of your experiments you can use "GitHub discussions" or discuss it in slack. But opening a "bug" in this case is clearly a mistake resulting from misuse and not following the instructions, rather than bug in airflow. This is precisely why it was marked as "invalid" and "closed".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889886049
The reason it was closed, because you indicated that things will work out-of-the-box when you indicated yourself you tried to change users (first 50000 and then 1001) and that it has changed since previous time, which indicated that you expected more from the docker compose than what it is intended to.
The "quick start" is really to get you quick-started and if you want to change anything (like change the user and experiment with the setting, there is not much we can do to "solve" the issue you raised as bug. From your message it seems that you had a history of using this setup and that you "expect" it to work under this different circumstances (one of the problems was that you used sudo to run the docker compose which is guaranteed not to work).
The Quick-start documentation is just that - quick-start. No more, no less. You have to strictly follow it to get it working, if you do any deviation from that, it might or might not work. When you open an issue indicating "bug" I think you expect it to be fixed. There is no action anyone can take here to "fix" the problems you were experiencing. That's why the issue was closed - because there is no reasonable action anyone can take here to fix the problems you experienced.
The best thing You can do is to wipe out all your setup and start from scratch and precisely follow the quick-start with no deviations and see if it works. If it does not, please report it here with all information that you can and I will be happy to open the issue (if it willl indicate that our quick start instructions are wrong. Or even better - open a PR fixing it straight away.
If you want to discuss the usage and results of your experiments you can use "GitHub discussions" or discuss it in slack. But opening a "bug" in this case is clearly a mistake resulting from misuse and not following the instructions, rather than bug in airflow. This is precisely why it was marked as "invalid" and "closed".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889886049
The reason it was closed, because you indicated that you expect that things will work out-of-the-box even when you indicated yourself you tried to change users (first 50000 and then 1001) and that it has changed since previous time, which indicated that you expected more from the docker compose than what it is intended to.
The "quick start" is really to get you quick-started and if you want to change anything (like change the user and experiment with the setting), there is not much we can do to "solve" the issue you raised as bug. From your message it seems that you had a history of using this setup and that you "expect" it to work under this different circumstances (one of the problems was that you used sudo to run the docker compose which is guaranteed not to work).
You also indicated "If I have to create and use a single UID=50000 that will handle all airflow operations, then my airflow file system within the host cannot be operated properly with different users, e.g. devs when pulling new changes from git..." which means that you wanted to make the docker compose works for many users, but this is definitely not the intention of our docker compose. It is there to make single user to be able to quick-start and run airflow on that user's machine. That's it. There are other ways to make airflow works for multiple users for development environment, but the quick start docker compose is not one of them. That was the "production" use i was referring to, which was indeed a bit to narrow, "multi-user" would be more appropriate. It's not designed to be used by "multiple users". So it is not a bug that it does not work this way.
The Quick-start documentation is just that - quick-start. No more, no less. You have to strictly follow it to get it working, if you do any deviation from that, it might or might not work. When you open an issue indicating "bug" I think you expect it to be fixed. There is no action anyone can take here to "fix" the problems you were experiencing. That's why the issue was closed - because there is no reasonable action anyone can take here to fix the problems you experienced.
The best thing You can do is to wipe out all your setup and start from scratch and precisely follow the quick-start with no deviations and see if it works. If it does not, please report it here with all information that you can and I will be happy to reopen the issue (if it willl indicate that our quick start instructions are wrong. Or even better - open a PR fixing it straight away.
If you want to discuss the usage and results of your experiments you can use "GitHub discussions" or discuss it in slack. But opening a "bug" in this case is clearly a mistake resulting from misuse and not following the instructions, rather than bug in airflow. This is precisely why it was marked as "invalid" and "closed".
And if you want to propose new feature, change how docker compose works - feel free. We actually have an open feature to make a more "versatile" docker-compose setup with more examples and possibly wizard-like generation https://github.com/apache/airflow/issues/16031 It would be great if you could contribute to that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and trying to understand it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
OK so setting my UID manually in the `.env` file is not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ??
Wouldn't that just relace my `.env` file and remove my `AIRFLOW_HOME`?? plus my UID (1001) is already set within it? BTW the GID is already set to 0 **as a group** within it aswell...
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001).
> The solution you copied is to manually build image and there **manually** your UID as the user.
I don't understand what you mean by **manually**, is it because I put it in the `.env` myself? Wouldn't your command do just that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889886049
The reason it was closed, because you indicated that you expect that things will work out-of-the-box even when you indicated yourself you tried to change users (first 50000 and then 1001) and that it has changed since previous time, which indicated that you expected more from the docker compose than what it is intended to.
The "quick start" is really to get you quick-started and if you want to change anything (like change the user and experiment with the setting, there is not much we can do to "solve" the issue you raised as bug. From your message it seems that you had a history of using this setup and that you "expect" it to work under this different circumstances (one of the problems was that you used sudo to run the docker compose which is guaranteed not to work).
The Quick-start documentation is just that - quick-start. No more, no less. You have to strictly follow it to get it working, if you do any deviation from that, it might or might not work. When you open an issue indicating "bug" I think you expect it to be fixed. There is no action anyone can take here to "fix" the problems you were experiencing. That's why the issue was closed - because there is no reasonable action anyone can take here to fix the problems you experienced.
The best thing You can do is to wipe out all your setup and start from scratch and precisely follow the quick-start with no deviations and see if it works. If it does not, please report it here with all information that you can and I will be happy to reopen the issue (if it willl indicate that our quick start instructions are wrong. Or even better - open a PR fixing it straight away.
If you want to discuss the usage and results of your experiments you can use "GitHub discussions" or discuss it in slack. But opening a "bug" in this case is clearly a mistake resulting from misuse and not following the instructions, rather than bug in airflow. This is precisely why it was marked as "invalid" and "closed".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed. The causes of this issue are still not clearly identified, or confirmed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and trying to understand it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
OK so setting my UID manually in the `.env` file is not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ??
Wouldn't that just relace my `.env` file and remove my `AIRFLOW_HOME`?? plus my UID (1001) is already set within it? BTW the GID is already set to 0 **as a group** within it aswell...
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001).
> The solution you copied is to manually build image and there **manually** your UID as the user.
I don't understand what you mean by **manually**, is it because I put it in the `.env` myself? Wouldn't your command do just that?
P.S : Please find the `.env` file content under the `docker-compose.yml` which was enclosed in the initial post, but might have been easily missed as the main content is bigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889886049
The reason it was closed, because you indicated that things will work out-of-the-box when you indicated yourself you tried to change users (first 50000 and then 1001) and that it has changed since previous time, which indicated that you expected more from the docker compose than what it is intended to.
The "quick start" is really to get you quick-started and if you want to change anything (like change the user and experiment with the setting, there is not much we can do to "solve" the issue you raised as bug. From your message it seems that you had a history of using this setup and that you "expect" it to work under this different circumstances (one of the problems was that you used sudo to run the docker compose which is guaranteed not to work).
The Quick-start documentation is just that - quick-start. No more, no less. You have to strictly follow it to get it working, if you do any deviation from that, it might or might not work. When you open an issue indicating "bug" I think you expect it to be fixed. There is no action anyone can take here to "fix" the problems you were experiencing. That's why the issue was closed - because there is no reasonable action anyone can take here to fix the problems you experienced.
The best thing You can do is to wipe out all your setup and start from scratch and precisely follow the quick-start with no deviations and see if it works. If it does not, please report it here with all information that you can and I will be happy to reopen the issue (if it willl indicate that our quick start instructions are wrong. Or even better - open a PR fixing it straight away.
If you want to discuss the usage and results of your experiments you can use "GitHub discussions" or discuss it in slack. But opening a "bug" in this case is clearly a mistake resulting from misuse and not following the instructions, rather than bug in airflow. This is precisely why it was marked as "invalid" and "closed".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889406349
BTW. I think you are not aware that your user changes when you run sudo. DO NOT use sudo when you run docker compose or docker commands, because then they will be used as "root" user (UID=0) which is likely the root cause of your problem. Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem. Make sure you do what docker installation suggests https://docs.docker.com/engine/install/linux-postinstall/ - add your user to docker group, so you do not have to use sudo to run docker command. Then check what permissions/ownership you have for the dags/ logs/ etc. folders (and files inside). Change them to be owned by your user rather than root.
And then follow the steps from the quick start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889403431
1. Docker-Compose of Airlow is not production ready. You should use it only for development and testing, if you want more production setup I recommend to use the official Helm Chart of the communiy: https://airflow.apache.org/docs/helm-
chart/stable/index.html and K8S.
2. If you mount your local folders as volumes to Airlfow, you should make sure that you use your HOST_ID as user_id and GID=0 as user. This is very clear in the "Initialize the environment section of the docker compose" https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initializing-environment quick start documentation. Apparently you mised that step. so let me copy it here (you need to run it once in the host in the directory where you have docker compose.
```
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
```
How it works - it changes the user that is used by Airflow to be the same as your host user, and sets the group ID to be 0 (which is the best practice followed from OpenShift to make possible to run container image as an arbitrary user).
3. No other configuration is supported for docker compose when you mount your local folder on linux. This is is a Docker limitation, not airflow nor image limitation. Airflow image is defined according to best practices of OpenShift and allows to run as arbitrary user id (, but when you mount local volume from host, the user from your host volume is an owner. There are various ways YOU can deal with the problem when you run the container (one of the solutions is the very one that Airlfow proposes - where you can define and use HOST UID and GID=0 to run the image. You can read more here: https://airflow.apache.org/docs/docker-stack/entrypoint.html#allowing-arbitrary-user-to-run-the-container
3. The solution you copied is to manually build image and there manually your UID as the user. Which (obviously) cannot be done in the public image, because we do not know your user id when we prepare the image and each user has a different UID. It's just impossible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889886049
The reason it was closed, because you indicated that you expect that things will work out-of-the-box even when you indicated yourself you tried to change users (first 50000 and then 1001) and that it has changed since previous time, which indicated that you expected more from the docker compose than what it is intended to.
The "quick start" is really to get you quick-started and if you want to change anything (like change the user and experiment with the setting), there is not much we can do to "solve" the issue you raised as bug. From your message it seems that you had a history of using this setup and that you "expect" it to work under this different circumstances (one of the problems was that you used sudo to run the docker compose which is guaranteed not to work).
You also indicated "If I have to create and use a single UID=50000 that will handle all airflow operations, then my airflow file system within the host cannot be operated properly with different users, e.g. devs when pulling new changes from git..." which means that you wanted to make the docker compose works for many users, but this is definitely not the intention of our docker compose. It is there to make single user to be able to quick-start and run airflow on that user's machine. That's it. There are other ways to make airflow works for multiple users for development environment, but the quick start docker compose is not one of them. That was the "production" use i was referring to, which was indeed a bit to narrow, "multi-user" would be more appropriate. It's not designed to be used by "multiple users". So it is not a bug that it does not work this way.
The Quick-start documentation is just that - quick-start. No more, no less. You have to strictly follow it to get it working, if you do any deviation from that, it might or might not work. When you open an issue indicating "bug" I think you expect it to be fixed. There is no action anyone can take here to "fix" the problems you were experiencing. That's why the issue was closed - because there is no reasonable action anyone can take here to fix the problems you experienced.
The best thing You can do is to wipe out all your setup and start from scratch and precisely follow the quick-start with no deviations and see if it works. If it does not, please report it here with all information that you can and I will be happy to reopen the issue (if it willl indicate that our quick start instructions are wrong. Or even better - open a PR fixing it straight away.
If you want to discuss the usage and results of your experiments you can use "GitHub discussions" or discuss it in slack. But opening a "bug" in this case is clearly a mistake resulting from misuse and not following the instructions, rather than bug in airflow. This is precisely why it was marked as "invalid" and "closed".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi commented on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi commented on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889320599
I did find [this thread](https://github.com/puckel/docker-airflow/issues/224) that deals with this issue, but I believe it should be incorporated to the public community image directly, instead of forking with a custom image. I.e. This custom image, by puckle, which solves this issue is for airflow 1.10.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889896217
> I thought I did set `AIRFLOW_UID` to 1001 and `AIRFLOW_GID` to 0 (manually) in the `.env` file.
> Is it not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ?
>
This looks good. As mentioned above - I recommend you to wipe it out, restart and see if you still have problems. In your earlier comments you mentioned that previously you used different USER and you used sudo to run docker command (which is guaranteed not to work because then your container are run as root rather than your own user. So I guess (as I wrote) that you missed the .env setup and run it without it as sudo user (which would create those directory as root user if you did). That was my line of thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi commented on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi commented on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and challenging it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
OK so setting my UID manually in the `.env` file is not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ??
Wouldn't that just relace my `.env` file and remove my `AIRFLOW_HOME`?? plus my UID (1001) is already set within it? BTW the GID is already set to 0 **as a group** within it aswell...
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001).
> The solution you copied is to manually build image and there **manually** your UID as the user.
I don't understand what you mean by **manually**, is it because I put it in the `.env` myself? Wouldn't your command do just that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889894166
> Could you clarify what you mean by **manually**? Is it because I put it in the `.env` myself? As I have asked earlier, wouldn't the above mentionned command do just that?
Manually means added to Dockerfile and specify it verbatim at the time of docker building. This is at least one of the solutions in the thread that you mentioned (you did not tell which one it was, so I picked the one that concluded the thread).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and challenging it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
OK so setting my UID manually in the `.env` file is not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ??
Wouldn't that just relace my `.env` file and remove my `AIRFLOW_HOME`?? plus my UID (1001) is already set within it? BTW the GID is already set to 0 **as a group** within it aswell...
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001).
> The solution you copied is to manually build image and there **manually** your UID as the user.
I don't understand what you mean by **manually**, is it because I put it in the `.env` myself? Wouldn't your command do just that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed. The causes of this issue are still not clearly identified, or confirmed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and trying to understand it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
I thought I did set `AIRFLOW_UID` to 1001 and `AIRFLOW_GID` to 0 (manually) in the `.env` file.
Is it not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ?
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001). I hope I am not badly mistaken about this, but if so, could you kindly clarify the misconception ?
> The solution you copied is to manually build image and there **manually** your UID as the user.
Could you clarify what you mean by **manually**? Is it because I put it in the `.env` myself? As I have asked earlier, wouldn't the above mentionned command do just that?
P.S : Please find the `.env` file content under the `docker-compose.yml` which was enclosed in the initial post, but might have been easily missed as the main content is bigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed. The causes of this issue are still not clearly identified, or confirmed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and trying to understand it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
I thought I did set to `AIRFLOW_UID` to 1001 and `AIRFLOW_GID` to 0 (manually) in the `.env` file.
Is it not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ?
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001). I hope I am not badly mistaken about this, but if so, could you kindly clarify the misconception ?
> The solution you copied is to manually build image and there **manually** your UID as the user.
Could you clarify what you mean by **manually**? Is it because I put it in the `.env` myself? As I have asked earlier, wouldn't the above mentionned command do just that?
P.S : Please find the `.env` file content under the `docker-compose.yml` which was enclosed in the initial post, but might have been easily missed as the main content is bigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ImadYIdrissi edited a comment on issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
ImadYIdrissi edited a comment on issue #17320:
URL: https://github.com/apache/airflow/issues/17320#issuecomment-889526816
I don't understand why this thread was closed.
> Docker-Compose of Airlow is not production ready. You should use it only for development and testing
I am not trying to use it for production, we are merely testing this approach, and trying to understand it.
> you should make sure that you use your HOST_ID as user_id and GID=0 as user.
OK so setting my UID manually in the `.env` file is not the same as doing `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env` ??
Wouldn't that just relace my `.env` file and remove my `AIRFLOW_HOME`?? plus my UID (1001) is already set within it? BTW the GID is already set to 0 **as a group** within it aswell...
> BTW. I think you are not aware that your user changes when you run sudo... Most likely your logs/dags/ etc. file have been created as owned by that root user and this is causing all your permission problem.
My dags, logs etc... are not root owned, they're user owned (UID : 1001).
> The solution you copied is to manually build image and there **manually** your UID as the user.
I don't understand what you mean by **manually**, is it because I put it in the `.env` myself? Wouldn't your command do just that?
P.S : Please find the `.env` file content under the `docker-compose.yml` which was enclosed in the initial post, but might have been easily missed as the main content is bigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17320: PermissionDenied : Unable to use docker-compose due to UID conflicts
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17320:
URL: https://github.com/apache/airflow/issues/17320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org