You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/05/21 07:06:52 UTC
[GitHub] [druid] tazle opened a new issue #11282: Permission issues with docker-compose setup
tazle opened a new issue #11282:
URL: https://github.com/apache/druid/issues/11282
### Affected Version
0.21.0
### Description
I tried to start up the docker-compose setup from https://github.com/apache/druid/blob/0.21.0/distribution/docker/docker-compose.yml but all the druid containers get permission errors when they try to access the filesystem:
e.g. the broker
```
broker | mkdir: can't create directory 'var/tmp': Permission denied
broker | mkdir: can't create directory 'var/druid/': Permission denied
broker | mkdir: can't create directory 'var/druid/': Permission denied
broker | mkdir: can't create directory 'var/druid/': Permission denied
broker | mkdir: can't create directory 'var/druid/': Permission denied
broker | mkdir: can't create directory 'var/druid/': Permission denied
```
Initially I thought the problem was that the "./storage" directory got created as root:root:
```
drwxr-xr-x 2 root root 4.0K May 21 09:54 storage
```
But even after creating it manually as my own user (with id 1000 like the druid user at https://github.com/apache/druid/blob/0.21.0/distribution/docker/Dockerfile#L48), the druid containers still get the same permission errors.
I think the trouble isn't with the storage volume (at /opt/data) but actually https://github.com/apache/druid/blob/0.21.0/distribution/docker/Dockerfile#L58 which mounts a new anonymous volume at /opt/druid/var. This volume is owned by root.
The docker-compose replaces it with a named volume, but that's still owned by root. The chown and chmod done in the Dockerfile at https://github.com/apache/druid/blob/0.21.0/distribution/docker/Dockerfile#L51-L52 don't help because they only apply to the container filesystem as it was before the volumes got mounted.
The problem seems similar to #11278.
To be honest, I don't see how the docker-compose setup could ever have worked, and I'm not sure how to fix it either, except by having the container start as root, and having an entrypoint script handle the user creation and setting permissions on the target volumes. This is how e.g. the postgres container works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #11282:
URL: https://github.com/apache/druid/issues/11282#issuecomment-846348374
@tazle Thanks for the testing. Indeed, a named volume works because the volume is created by docker and its permission is correctly setup, while the permission of a volume mapped to an absolute path on host is managed by the host OS.
One disadvantage of a named volume is that its directory is managed by docker itself and usually is located at /var directory on the host, since data dir is used to store segments and task logs, its size might grow very fast to make the disk shortage. We have to document this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] tazle commented on issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
tazle commented on issue #11282:
URL: https://github.com/apache/druid/issues/11282#issuecomment-845954351
I'm on Linux.
Looks like 0.20.2 indeed works out-of-the-box with the storage from host. I don't quite understand why that is. Other than that it's related to the COPY which overwrites ownerships on the container filesystem, but I thought the volume mounts would still appear as root:root. Clearly I don't quite know what's going on here.
Apparently docker-compose checks the ownership of the first container in which it mounts a volume when creating it, and sets the ownership accordingly. I can also put /opt/data on a named volume if I change the storage mount from `./storage:/opt/data` to `storage:/opt/data` on the druid containers and add the following service as the first thing in the composefile:
```
test:
container_name: test
build: .
volumes:
- storage:/opt/data
command: ['sleep', '100000']
```
with Dockerfile being:
```
FROM ubuntu:20.04
RUN addgroup --system --gid 1000 test \
&& adduser --system --uid 1000 --no-create-home --home /opt/ --shell /bin/sh --gid 1000 test \
&& mkdir -p /opt/data \
&& chown -R test:test /opt
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #11282:
URL: https://github.com/apache/druid/issues/11282#issuecomment-852648740
Closed via #11299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #11282:
URL: https://github.com/apache/druid/issues/11282#issuecomment-847757225
@tazle The problem of `storage` directory was reported last week on the 0.21.1-RC1, we have been discussing this problem on the dev mails and I'm working on a PR to fix it. It's also tracked by #11298 .
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 closed issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
FrankChen021 closed issue #11282:
URL: https://github.com/apache/druid/issues/11282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
clintropolis commented on issue #11282:
URL: https://github.com/apache/druid/issues/11282#issuecomment-845718399
Hi @tazle , we're aware of this issue, it was a regression with the 0.21 release. We are working on a 0.21.1 patch release, see #11249, and hope to have it released soon.
The current RC1 should fix the permissions issue on the directories created by the Dockerfile, but still has an issue with the storage directory created by docker-compose, so we might or might not need to do an RC2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #11282:
URL: https://github.com/apache/druid/issues/11282#issuecomment-845728254
@tazle If you're on macOS or Linux, I think the upcoming RC1 should have fixed this issue. If you're looking for a workaround, please take a look at the [release note](https://github.com/apache/druid/releases)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] tazle commented on issue #11282: Permission issues with docker-compose setup
Posted by GitBox <gi...@apache.org>.
tazle commented on issue #11282:
URL: https://github.com/apache/druid/issues/11282#issuecomment-846383379
Indeed. I'm not asking for the default docker-compose to be changed to use a named volume, but I'd like to see the Dockerfile enable that by creating and setting up the permissions of the mountpoint directory so that a separate container is not needed for that.
This may be bit of a tangent from my original issue, so perhaps I could create a new issue (and PR) for this change, unless you are opposed to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org