You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/01/27 10:29:39 UTC
[GitHub] [druid] clintropolis opened a new pull request #9262: add docker
tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
clintropolis opened a new pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262
### Description
This PR updates the example `docker-compose.yml` file to be more 'quickstart' friendly, opting to configure it to mount a local directory to share between containers as deep storage so that it can potentially run out of the box, as well as run all of the services on the default port. I also removed the overlord since the coordinator was also configured to be an overlord.
Additionally, it adds Java 11 version of the Dockerfile, and I updated the Docker README.md to include reference to it, though mentioning it as experimental. I ran some tests with it and everything appeared to function for me, though I only ran through the native batch quick-start and ran several queries.
Try it out and let me know what you find:
```
docker build -t apache/druid:0.18.0-SNAPSHOT-java11 -f distribution/docker/Dockerfile.java11 .
```
and modify the `docker-compose.yml` to use the `0.18.0-SNAPSHOT-java11` or whatever you call your tag.
Finally, it adds a rough 'getting started' with Docker page to the website documentation:
<img width="1291" alt="Screen Shot 2020-01-27 at 1 33 09 AM" src="https://user-images.githubusercontent.com/1577461/73166570-f3f8ac80-40aa-11ea-9710-f4436d2e826e.png">
Some of the content was stolen from `tutorials/index.md`, but maybe is a good enough start to elaborate on in future PRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #9262: add docker tutorial,
friendlier docker-compose.yml, experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
jihoonson merged pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] sthetland commented on a change in pull request #9262: add
docker tutorial, friendlier docker-compose.yml,
experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
sthetland commented on a change in pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#discussion_r372727632
##########
File path: docs/tutorials/docker.md
##########
@@ -0,0 +1,81 @@
+---
+id: docker
+title: "Docker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In this quickstart, we will download the Druid docker image from [Apache Druid on Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data
+after completing this initial setup.
+
+Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the
+[ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended.
+
+## Prerequisites
+
+* Docker
+
+## Getting started
+
+The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/master/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments.
+
+### Compose file
+
+The example `docker-compose.yml` will create a container for each Druid service, as well as Zookeeper and a PostgreSQL container as the metadata store. Deep storage will be a local directory, by default configured as `./storage` relative to your `docker-compose.yml` file, and will be mounted as `/opt/data` and shared between Druid containers which require access to deep storage. The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/master/distribution/docker/environment).
+
+### Configuration
+
+Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md)
+
+Special environment variables:
+
+* `JAVA_OPTS` -- set java options
+* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim
+* `DRUID_LOG_LEVEL` -- override the default log level in default log4j
+* `DRUID_XMX` -- set Java `Xmx`
+* `DRUID_XMS` -- set Java `Xms`
+* `DRUID_MAXNEWSIZE` -- set Java max new size
+* `DRUID_NEWSIZE` -- set Java new size
+* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size
+* `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties
+* `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties
+
+In addition to the special environment variables, the script which launches Druid in the container will also attempt to use any environment variable starting with the `druid_` prefix as a command-line configuration. For example, an environment variable
+
+```druid_metadata_storage_type=postgresql```
+
+would be translated into
+
+```-Ddruid.metadata.storage.type=postgresql```
+
+for the Druid process in the container.
+
+The Druid `docker-compose.yml` example utilizes a single environment file to specify the complete Druid configuration, however in production use cases we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored service specific environment files.
+
+## Launching the cluster
+
+Run `docker-compose up` to launch the cluster will a shell attached, or `docker-compose up -d` to run the cluster in the background. Once the cluster has started, you can navigate to [http://localhost:8888](http://localhost:8888).
Review comment:
Also, maybe others would know, because the path is in a section above, but it wasn't obvious to me to run the command from druid_repo_home/distribution/docker. So optionally, could mention, i.e., "Run `docker-compose up` from the distribution/docker directory to launch...."
One more thing, do they need to do the docker commands that are in the druid_repo_home/distribution/docker/README.md ? I did, but wasn't sure I had to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] capistrant edited a comment on issue #9262: add docker
tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
capistrant edited a comment on issue #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#issuecomment-579426219
👍
Ran with the apache druid image using java 8 and it worked great out of the box. Ran an ingest job with local firehose and was able to query data (Only did this very basic testing). The document is straight-forward and the local mount shared by the nodes is awesome for convenience.
For the Java 11 container, I did the same simple test. Seemed to work just fine. Unfortunately, I didn't have a lot of time to poke around with this at the moment, but I think it is great that we are kicking the tires on Java11.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9262:
add docker tutorial, friendlier docker-compose.yml,
experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#discussion_r377962501
##########
File path: docs/tutorials/docker.md
##########
@@ -0,0 +1,81 @@
+---
+id: docker
+title: "Docker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In this quickstart, we will download the Druid docker image from [Apache Druid on Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data
+after completing this initial setup.
+
+Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the
+[ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended.
+
+## Prerequisites
+
+* Docker
+
+## Getting started
+
+The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/master/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments.
+
+### Compose file
+
+The example `docker-compose.yml` will create a container for each Druid service, as well as Zookeeper and a PostgreSQL container as the metadata store. Deep storage will be a local directory, by default configured as `./storage` relative to your `docker-compose.yml` file, and will be mounted as `/opt/data` and shared between Druid containers which require access to deep storage. The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/master/distribution/docker/environment).
+
+### Configuration
+
+Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md)
+
+Special environment variables:
+
+* `JAVA_OPTS` -- set java options
+* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim
+* `DRUID_LOG_LEVEL` -- override the default log level in default log4j
+* `DRUID_XMX` -- set Java `Xmx`
+* `DRUID_XMS` -- set Java `Xms`
+* `DRUID_MAXNEWSIZE` -- set Java max new size
+* `DRUID_NEWSIZE` -- set Java new size
+* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size
+* `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties
+* `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties
+
+In addition to the special environment variables, the script which launches Druid in the container will also attempt to use any environment variable starting with the `druid_` prefix as a command-line configuration. For example, an environment variable
+
+```druid_metadata_storage_type=postgresql```
+
+would be translated into
+
+```-Ddruid.metadata.storage.type=postgresql```
+
+for the Druid process in the container.
+
+The Druid `docker-compose.yml` example utilizes a single environment file to specify the complete Druid configuration, however in production use cases we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored service specific environment files.
+
+## Launching the cluster
+
+Run `docker-compose up` to launch the cluster will a shell attached, or `docker-compose up -d` to run the cluster in the background. Once the cluster has started, you can navigate to [http://localhost:8888](http://localhost:8888).
Review comment:
Fixed typo.
The steps in the readme are only necessary if building druid image from source, which I'm not sure needs to be mentioned here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] sthetland commented on a change in pull request #9262: add
docker tutorial, friendlier docker-compose.yml,
experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
sthetland commented on a change in pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#discussion_r372726820
##########
File path: docs/tutorials/docker.md
##########
@@ -0,0 +1,81 @@
+---
+id: docker
+title: "Docker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In this quickstart, we will download the Druid docker image from [Apache Druid on Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data
+after completing this initial setup.
+
+Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the
+[ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended.
+
+## Prerequisites
+
+* Docker
+
+## Getting started
+
+The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/master/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments.
+
+### Compose file
+
+The example `docker-compose.yml` will create a container for each Druid service, as well as Zookeeper and a PostgreSQL container as the metadata store. Deep storage will be a local directory, by default configured as `./storage` relative to your `docker-compose.yml` file, and will be mounted as `/opt/data` and shared between Druid containers which require access to deep storage. The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/master/distribution/docker/environment).
+
+### Configuration
+
+Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md)
+
+Special environment variables:
+
+* `JAVA_OPTS` -- set java options
+* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim
+* `DRUID_LOG_LEVEL` -- override the default log level in default log4j
+* `DRUID_XMX` -- set Java `Xmx`
+* `DRUID_XMS` -- set Java `Xms`
+* `DRUID_MAXNEWSIZE` -- set Java max new size
+* `DRUID_NEWSIZE` -- set Java new size
+* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size
+* `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties
+* `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties
+
+In addition to the special environment variables, the script which launches Druid in the container will also attempt to use any environment variable starting with the `druid_` prefix as a command-line configuration. For example, an environment variable
+
+```druid_metadata_storage_type=postgresql```
+
+would be translated into
+
+```-Ddruid.metadata.storage.type=postgresql```
+
+for the Druid process in the container.
+
+The Druid `docker-compose.yml` example utilizes a single environment file to specify the complete Druid configuration, however in production use cases we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored service specific environment files.
+
+## Launching the cluster
+
+Run `docker-compose up` to launch the cluster will a shell attached, or `docker-compose up -d` to run the cluster in the background. Once the cluster has started, you can navigate to [http://localhost:8888](http://localhost:8888).
Review comment:
"cluster will" -> " cluster with"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9262: add docker tutorial,
friendlier docker-compose.yml, experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#issuecomment-579427036
The CI failure looks legit.
```
../docs/tutorials/docker.md
25 | r image from [Apache Druid on Dockerhub](https://hub.docker.com/r/apa
37 | which can pull an image from dockerhub and is suited to be used as a
50 | RUID_LOG4J` -- set the entire log4j.xml verbatim
52 | * `DRUID_XMX` -- set Java Xmx
53 | * `DRUID_XMS` -- set Java Xms
54 | XNEWSIZE` -- set Java max new size`
56 | MAXDIRECTMEMORYSIZE` -- set Java max direct memory size
56 | -- set Java max direct memory size
74 | Run `docker-compose up` to launch the cluster will a shell attache
74 | ached, or `docker-compose up -d` to run the cluster in the background. Once
74 | r-compose up -d` to run the cluster in the background. Once the
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9262:
add docker tutorial, friendlier docker-compose.yml,
experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#discussion_r377962690
##########
File path: docs/tutorials/docker.md
##########
@@ -0,0 +1,81 @@
+---
+id: docker
+title: "Docker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In this quickstart, we will download the Druid docker image from [Apache Druid on Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data
Review comment:
changed to 'Druid image'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] capistrant commented on issue #9262: add docker tutorial,
friendlier docker-compose.yml, experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
capistrant commented on issue #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#issuecomment-579426219
👍
Ran with the dockerhub image using java 8 and it worked great out of the box. Ran an ingest job with local firehose and was able to query data (Only did this very basic testing). The document is straight-forward and the local mount shared by the nodes is awesome for convenience.
For the Java 11 container, I did the same simple test. Seemed to work just fine. Unfortunately, I didn't have a lot of time to poke around with this at the moment, but I think it is great that we are kicking the tires on Java11.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] sthetland commented on a change in pull request #9262: add
docker tutorial, friendlier docker-compose.yml,
experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
sthetland commented on a change in pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#discussion_r372725127
##########
File path: docs/tutorials/docker.md
##########
@@ -0,0 +1,81 @@
+---
+id: docker
+title: "Docker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In this quickstart, we will download the Druid docker image from [Apache Druid on Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data
Review comment:
Capitalize Docker? So "Druid docker image" > "Druid Docker image" ... or maybe just "Druid image"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] sthetland commented on a change in pull request #9262: add
docker tutorial, friendlier docker-compose.yml,
experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
sthetland commented on a change in pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#discussion_r372726666
##########
File path: docs/tutorials/docker.md
##########
@@ -0,0 +1,81 @@
+---
+id: docker
+title: "Docker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In this quickstart, we will download the Druid docker image from [Apache Druid on Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data
+after completing this initial setup.
+
+Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the
+[ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended.
+
+## Prerequisites
+
+* Docker
+
+## Getting started
+
+The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/master/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments.
+
+### Compose file
+
+The example `docker-compose.yml` will create a container for each Druid service, as well as Zookeeper and a PostgreSQL container as the metadata store. Deep storage will be a local directory, by default configured as `./storage` relative to your `docker-compose.yml` file, and will be mounted as `/opt/data` and shared between Druid containers which require access to deep storage. The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/master/distribution/docker/environment).
+
+### Configuration
+
+Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md)
+
+Special environment variables:
+
+* `JAVA_OPTS` -- set java options
+* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim
+* `DRUID_LOG_LEVEL` -- override the default log level in default log4j
+* `DRUID_XMX` -- set Java `Xmx`
+* `DRUID_XMS` -- set Java `Xms`
+* `DRUID_MAXNEWSIZE` -- set Java max new size
+* `DRUID_NEWSIZE` -- set Java new size
+* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size
+* `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties
+* `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties
+
+In addition to the special environment variables, the script which launches Druid in the container will also attempt to use any environment variable starting with the `druid_` prefix as a command-line configuration. For example, an environment variable
+
+```druid_metadata_storage_type=postgresql```
+
+would be translated into
+
+```-Ddruid.metadata.storage.type=postgresql```
+
+for the Druid process in the container.
+
+The Druid `docker-compose.yml` example utilizes a single environment file to specify the complete Druid configuration, however in production use cases we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored service specific environment files.
Review comment:
A nit, but i suggest splitting this long sentence, maybe using a semi-colon, with a few other punctuation adjustments:
The Druid `docker-compose.yml` example utilizes a single environment file to specify the complete Druid configuration; however, in production use cases we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored, service-specific environment files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #9262: add docker tutorial,
friendlier docker-compose.yml, experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
clintropolis commented on issue #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#issuecomment-579473564
>The CI failure looks legit.
Oh yeah, will fix it up, just haven't got back to it yet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9262:
add docker tutorial, friendlier docker-compose.yml,
experimental java 11 dockerfile
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9262: add docker tutorial, friendlier docker-compose.yml, experimental java 11 dockerfile
URL: https://github.com/apache/druid/pull/9262#discussion_r377962639
##########
File path: docs/tutorials/docker.md
##########
@@ -0,0 +1,81 @@
+---
+id: docker
+title: "Docker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In this quickstart, we will download the Druid docker image from [Apache Druid on Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data
+after completing this initial setup.
+
+Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the
+[ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended.
+
+## Prerequisites
+
+* Docker
+
+## Getting started
+
+The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/master/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments.
+
+### Compose file
+
+The example `docker-compose.yml` will create a container for each Druid service, as well as Zookeeper and a PostgreSQL container as the metadata store. Deep storage will be a local directory, by default configured as `./storage` relative to your `docker-compose.yml` file, and will be mounted as `/opt/data` and shared between Druid containers which require access to deep storage. The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/master/distribution/docker/environment).
+
+### Configuration
+
+Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md)
+
+Special environment variables:
+
+* `JAVA_OPTS` -- set java options
+* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim
+* `DRUID_LOG_LEVEL` -- override the default log level in default log4j
+* `DRUID_XMX` -- set Java `Xmx`
+* `DRUID_XMS` -- set Java `Xms`
+* `DRUID_MAXNEWSIZE` -- set Java max new size
+* `DRUID_NEWSIZE` -- set Java new size
+* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size
+* `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties
+* `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties
+
+In addition to the special environment variables, the script which launches Druid in the container will also attempt to use any environment variable starting with the `druid_` prefix as a command-line configuration. For example, an environment variable
+
+```druid_metadata_storage_type=postgresql```
+
+would be translated into
+
+```-Ddruid.metadata.storage.type=postgresql```
+
+for the Druid process in the container.
+
+The Druid `docker-compose.yml` example utilizes a single environment file to specify the complete Druid configuration, however in production use cases we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored service specific environment files.
Review comment:
changed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org