You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/05/20 20:07:06 UTC
[GitHub] [druid] mattmassicotte opened a new issue #11278: Unable to use AWS Fargate bind mounts with Dockerfile
mattmassicotte opened a new issue #11278:
URL: https://github.com/apache/druid/issues/11278
### Affected Version
0.21.1-rc1
### Description
So far, I've never been able to use AWS Fargate bind mounts (ie Docker volumes) with Druid. I have a workaround of using EFS mounts, but it is a real pain to use. I was really excited to see that 0.21.1-rc1 included a change to the Dockefile, so I decided to check it out. Unfortunately, it still does not work.
The druid process fails to start with the following logging:
```
mkdir: can't create directory 'var/tmp': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
```
I'm neither a Fargate, Druid, or Docker expert. But, from what I can gather, Fargate is very sensitive to the nature of the path in the `VOLUME` command in the Dockerfile.
Some details here:
https://github.com/aws/containers-roadmap/issues/938#issuecomment-812590686
An AWS employee actually looked the the Druid Dockefile and identified the symlinking as the issue. I figured I'd open this bug since it is indeed still not quite compatible with Fargate and this area was just touched.
If I can help in some way, I'd really love to, as this is currently affecting my ability to easily scale Druid. Each new instance requires a special EFS mount point, which it turn requires a special ECS task definition. Regular docker volume support would be awesome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] mattmassicotte commented on issue #11278: Unable to use AWS Fargate bind mounts with Dockerfile
Posted by GitBox <gi...@apache.org>.
mattmassicotte commented on issue #11278:
URL: https://github.com/apache/druid/issues/11278#issuecomment-845990954
Thanks so much for looking into this, and for the suggestions.I'm attempting to do this.
Unfortunately, I'm on an arm Mac, and I'm having a very hard time building the image. I also kinda don't know what I'm doing, but I'm fairly sure the issues I'm hitting are platform-related.
I'm going to try to get set up on an amd64 machine and give it another go. Will report back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] mattmassicotte commented on issue #11278: Unable to use AWS Fargate bind mounts with Dockerfile
Posted by GitBox <gi...@apache.org>.
mattmassicotte commented on issue #11278:
URL: https://github.com/apache/druid/issues/11278#issuecomment-846051502
Ok, so that indeed seems to have resolved the issue! I was able to make and test the change from the druid-0.21.1-rc1 tag, and successfully start up a coordinator. I have not tried a full system. But, this is major progress. Thank you all so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #11278: Unable to use AWS Fargate bind mounts with Dockerfile
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #11278:
URL: https://github.com/apache/druid/issues/11278#issuecomment-845726818
This problem is very interesting. I agree that we eliminate the symlink to make a try.
@mattmassicotte Could you do a test by changing the line
https://github.com/apache/druid/blob/fcb4eaa3d481793d4f73ea67f4eabbc83f1745e1/distribution/docker/Dockerfile#L44
to
```
&& mv /opt/apache-druid-${VERSION} /opt/druid
```
and build docker images by your own to see if it works ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #11278: Unable to use AWS Fargate bind mounts with Dockerfile
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #11278:
URL: https://github.com/apache/druid/issues/11278#issuecomment-845590508
Hi @mattmassicotte, thank you for the report! Do you have a fix in your mind that could work? I'm wondering if the fix would be just like renaming the directory instead of creating a symlink, but am not an expert of Fargate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis closed issue #11278: Unable to use AWS Fargate bind mounts with Dockerfile
Posted by GitBox <gi...@apache.org>.
clintropolis closed issue #11278:
URL: https://github.com/apache/druid/issues/11278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org