You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "AlexMercedCoder (via GitHub)" <gi...@apache.org> on 2024/05/10 21:36:46 UTC
[I] Quick Start Permission Issue: Can't Ingest (druid)
AlexMercedCoder opened a new issue, #16432:
URL: https://github.com/apache/druid/issues/16432
Supposedly, this was was fixed by https://github.com/apache/druid/pull/11299/files
But I am getting the same issues of ingestion failing due to not being able to create a directory in opt shared:
here is my docker-compose.yml
```
version: "3"
services:
# Nessie Catalog Server Using In-Memory Store
nessie:
image: projectnessie/nessie:latest
container_name: nessie
networks:
dremio-druid-superset:
ports:
- 19120:19120
# Minio Storage Server
minio:
image: minio/minio:latest
container_name: minio
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
- MINIO_DOMAIN=storage
- MINIO_REGION_NAME=us-east-1
- MINIO_REGION=us-east-1
networks:
dremio-druid-superset:
ports:
- 9001:9001
- 9000:9000
command: ["server", "/data", "--console-address", ":9001"]
# Dremio
dremio:
platform: linux/x86_64
image: dremio/dremio-oss:latest
ports:
- 9047:9047
- 31010:31010
- 32010:32010
container_name: dremio
environment:
- DREMIO_JAVA_SERVER_EXTRA_OPTS=-Dpaths.dist=file:///opt/dremio/data/dist
networks:
dremio-druid-superset:
# Apache Druid
postgres:
container_name: postgres
image: postgres:latest
ports:
- "5433:5432"
volumes:
- metadata_data:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=FoolishPassword
- POSTGRES_USER=druid
- POSTGRES_DB=druid
networks:
dremio-druid-superset:
# Need 3.5 or later for container nodes
zookeeper:
container_name: zookeeper
image: zookeeper:3.5.10
ports:
- "2181:2181"
environment:
- ZOO_MY_ID=1
networks:
dremio-druid-superset:
coordinator:
image: apache/druid:29.0.1
container_name: coordinator
volumes:
- druid_shared:/opt/shared
- coordinator_var:/opt/druid/var
depends_on:
- zookeeper
- postgres
ports:
- "8081:8081"
command:
- coordinator
env_file:
- environment
networks:
dremio-druid-superset:
broker:
image: apache/druid:29.0.1
container_name: broker
volumes:
- broker_var:/opt/druid/var
depends_on:
- zookeeper
- postgres
- coordinator
ports:
- "8082:8082"
command:
- broker
env_file:
- environment
networks:
dremio-druid-superset:
historical:
image: apache/druid:29.0.1
container_name: historical
volumes:
- druid_shared:/opt/shared
- historical_var:/opt/druid/var
depends_on:
- zookeeper
- postgres
- coordinator
ports:
- "8083:8083"
command:
- historical
env_file:
- environment
networks:
dremio-druid-superset:
middlemanager:
image: apache/druid:29.0.1
container_name: middlemanager
volumes:
- druid_shared:/opt/shared
- middle_var:/opt/druid/var
depends_on:
- zookeeper
- postgres
- coordinator
ports:
- "8091:8091"
- "8100-8105:8100-8105"
command:
- middleManager
env_file:
- environment
networks:
dremio-druid-superset:
router:
image: apache/druid:29.0.1
container_name: router
volumes:
- router_var:/opt/druid/var
depends_on:
- zookeeper
- postgres
- coordinator
ports:
- "8888:8888"
command:
- router
env_file:
- environment
networks:
dremio-druid-superset:
# Superset
superset:
image: apache/superset
container_name: superset
networks:
dremio-druid-superset:
ports:
- 8088:8088
networks:
dremio-druid-superset:
volumes:
metadata_data:
middle_var:
historical_var:
broker_var:
coordinator_var:
router_var:
druid_shared:
```
I pretty much copied what was in the repo here https://github.com/apache/druid/blob/29.0.1/distribution/docker/docker-compose.yml. It all starts up fine but I try to do a simple "paste data" ingestion of some comma seperated data:
````
day, amount
1,1
2,2
3,4
4,8
5,16
6,32
```
I get an error and see this in the middlemanager logs
```
middlemanager | 2024-05-10T21:20:59,508 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Logging task query-64c248f2-1516-422c-ba60-79d01f644de8-worker0_0 output to: var/druid/task/slot1/query-64c248f2-1516-422c-ba60-79d01f644de8-worker0_0/log
middlemanager | 2024-05-10T21:20:59,509 DEBUG [qtp514556983-73] org.apache.druid.jetty.RequestLog - 192.168.48.4 GET //192.168.48.7:8091/druid-internal/v1/worker?counter=8&hash=1715376059501&timeout=180000 HTTP/1.1 204
middlemanager | 2024-05-10T21:20:59,554 DEBUG [SegmentChangeRequestHistory] org.apache.druid.jetty.RequestLog - 192.168.48.4 GET //192.168.48.7:8091/druid-internal/v1/worker?counter=9&hash=1715376059503&timeout=180000 HTTP/1.1 200
middlemanager | 2024-05-10T21:21:22,349 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution
middlemanager | org.apache.druid.java.util.common.IOE: Cannot create directory [/opt/shared/indexing-logs]
middlemanager | at org.apache.druid.java.util.common.FileUtils.mkdirp(FileUtils.java:488) ~[druid-processing-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.common.tasklogs.FileTaskLogs.pushTaskLog(FileTaskLogs.java:53) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner.waitForTaskProcessToComplete(ForkingTaskRunner.java:517) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:404) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:171) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-32.0.1-jre.jar:?]
middlemanager | at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) ~[guava-32.0.1-jre.jar:?]
middlemanager | at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-32.0.1-jre.jar:?]
middlemanager | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
middlemanager | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
middlemanager | at java.lang.Thread.run(Thread.java:840) ~[?:?]
middlemanager | 2024-05-10T21:21:22,355 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/slot1/query-64c248f2-1516-422c-ba60-79d01f644de8-worker0_0
middlemanager | 2024-05-10T21:21:22,449 DEBUG [SegmentChangeRequestHistory] org.apache.druid.jetty.RequestLog - 192.168.48.4 GET //192.168.48.7:8091/druid-internal/v1/worker?counter=10&hash=1715376059553&timeout=180000 HTTP/1.1 200
middlemanager | 2024-05-10T21:21:22,471 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task [query-64c248f2-1516-422c-ba60-79d01f644de8-worker0_0] completed with status [FAILED].
middlemanager | 2024-05-10T21:21:24,053 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution
middlemanager | org.apache.druid.java.util.common.IOE: Cannot create directory [/opt/shared/indexing-logs]
middlemanager | at org.apache.druid.java.util.common.FileUtils.mkdirp(FileUtils.java:488) ~[druid-processing-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.common.tasklogs.FileTaskLogs.pushTaskLog(FileTaskLogs.java:53) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner.waitForTaskProcessToComplete(ForkingTaskRunner.java:517) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:404) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:171) ~[druid-indexing-service-29.0.1.jar:29.0.1]
middlemanager | at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-32.0.1-jre.jar:?]
middlemanager | at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) ~[guava-32.0.1-jre.jar:?]
middlemanager | at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-32.0.1-jre.jar:?]
middlemanager | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
middlemanager | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
middlemanager | at java.lang.Thread.run(Thread.java:840) ~[?:?]
middlemanager | 2024-05-10T21:21:24,055 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/slot0/query-64c248f2-1516-422c-ba60-79d01f644de8
middlemanager | 2024-05-10T21:21:24,132 DEBUG [SegmentChangeRequestHistory] org.apache.druid.jetty.RequestLog - 192.168.48.4 GET //192.168.48.7:8091/druid-internal/v1/worker?counter=11&hash=1715376082442&timeout=180000 HTTP/1.1 200
middlemanager | 2024-05-10T21:21:24,148 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task [query-64c248f2-1516-422c-ba60-79d01f644de8] completed with status [FAILED].
middlemanager | 2024-05-10T21:24:19,253 DEBUG [SegmentChangeRequestHistory] org.apache.druid.jetty.RequestLog - 192.168.48.4 GET //192.168.48.7:8091/druid-internal/v1/worker?counter=12&hash=1715376084130&timeout=180000 HTTP/1.1 200
middlemanager | 2024-05-10T21:24:19,257 DEBUG [qtp514556983-82] org.apache.druid.jetty.RequestLog - 192.168.48.4 GET //192.168.48.7:8091/druid-internal/v1/worker?counter=13&hash=1715376259251&timeout=180000 HTTP/1.1 204
```
I'm using the exact same version of the environment file from the repo here: https://github.com/apache/druid/blob/29.0.1/distribution/docker/environment
Is there something I'm missing? Cause I've read everything I could find and supposedly the named volumes were changed to fix this, but I seem to still be seeing the same problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Quick Start Permission Issue: Can't Ingest (druid)
Posted by "AlexMercedCoder (via GitHub)" <gi...@apache.org>.
AlexMercedCoder commented on issue #16432:
URL: https://github.com/apache/druid/issues/16432#issuecomment-2106084755
I was able to get this working by adding
user:root to the middlemanager, not an ideal solution but for anyone else who runs into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Quick Start Permission Issue: Can't Ingest (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16432:
URL: https://github.com/apache/druid/issues/16432#issuecomment-2661702955
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Quick Start Permission Issue: Can't Ingest (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16432: Quick Start Permission Issue: Can't Ingest
URL: https://github.com/apache/druid/issues/16432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Quick Start Permission Issue: Can't Ingest (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16432:
URL: https://github.com/apache/druid/issues/16432#issuecomment-2727746262
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org