You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "techdocsmith (via GitHub)" <gi...@apache.org> on 2023/04/06 22:28:03 UTC
[GitHub] [druid] techdocsmith commented on a diff in pull request #13984: Docs: Tutorial for streaming ingestion using Kafka + Docker file to use with Jupyter tutorials
techdocsmith commented on code in PR #13984:
URL: https://github.com/apache/druid/pull/13984#discussion_r1160303834
##########
docs/tutorials/tutorial-jupyter-docker.md:
##########
@@ -0,0 +1,181 @@
+---
+id: tutorial-jupyter-docker
+title: "Docker for Jupyter Notebook tutorials"
+sidebar_label: "Docker for tutorials"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Apache Druid provides a custom Jupyter container that contains the prerequisites
+for all Jupyter-based Druid tutorials, as well as all of the tutorials themselves.
+You can run the Jupyter container, as well as containers for Druid and Apache Kafka,
+using the Docker Compose file provided in the Druid GitHub repository.
+
+You can run the following combination of applications:
+* [Jupyter only](#start-only-the-jupyter-container)
+* [Jupyter and Druid](#start-jupyter-and-druid)
+* [Jupyter, Druid, and Kafka](#start-jupyter-druid-and-kafka)
+
+## Prerequisites
+
+Jupyter in Docker requires that you have **Docker** and **Docker Compose**.
+We recommend installing these through [Docker Desktop](https://docs.docker.com/desktop/).
+
+## Launch the Docker containers
+
+You run Docker Compose to launch Jupyter and optionally Druid or Kafka.
+Docker Compose references the configuration in `docker-compose.yaml`.
+Running Druid in Docker also requires the `environment` file, which
+sets the configuration properties for the Druid services.
+To get started, download both `docker-compose.yaml` and `environment` from
+[`tutorial-jupyter-docker.zip`](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/docker-jupyter/tutorial-jupyter-docker.zip).
+
+Alternatively, you can clone the [Apache Druid repo](https://github.com/apache/druid) and
+access the files in `druid/examples/quickstart/jupyter-notebooks/docker-jupyter`.
+
+### Start only the Jupyter container
+
+If you already have Druid running locally, you can run only the Jupyter container to complete the tutorials.
+In the same directory as `docker-compose.yaml`, start the application:
+
+```bash
+docker-compose --profile jupyter up -d
+```
+
+The port assigned to Jupyter is `8889` by default.
+You can override the port number by setting the `JUPYTER_PORT` environment variable before starting the Docker application.
+
+### Start Jupyter and Druid
+
+Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable.
+`DRUID_VERSION` supplies the Docker tag that specifies which version of Druid to pull from the
Review Comment:
```suggestion
`DRUID_VERSION` references the Docker tag for the version of Druid to pull from the
```
##########
docs/tutorials/tutorial-jupyter-docker.md:
##########
@@ -0,0 +1,181 @@
+---
+id: tutorial-jupyter-docker
+title: "Docker for Jupyter Notebook tutorials"
+sidebar_label: "Docker for tutorials"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Apache Druid provides a custom Jupyter container that contains the prerequisites
+for all Jupyter-based Druid tutorials, as well as all of the tutorials themselves.
+You can run the Jupyter container, as well as containers for Druid and Apache Kafka,
+using the Docker Compose file provided in the Druid GitHub repository.
+
+You can run the following combination of applications:
+* [Jupyter only](#start-only-the-jupyter-container)
+* [Jupyter and Druid](#start-jupyter-and-druid)
+* [Jupyter, Druid, and Kafka](#start-jupyter-druid-and-kafka)
+
+## Prerequisites
+
+Jupyter in Docker requires that you have **Docker** and **Docker Compose**.
+We recommend installing these through [Docker Desktop](https://docs.docker.com/desktop/).
+
+## Launch the Docker containers
+
+You run Docker Compose to launch Jupyter and optionally Druid or Kafka.
+Docker Compose references the configuration in `docker-compose.yaml`.
+Running Druid in Docker also requires the `environment` file, which
+sets the configuration properties for the Druid services.
+To get started, download both `docker-compose.yaml` and `environment` from
+[`tutorial-jupyter-docker.zip`](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/docker-jupyter/tutorial-jupyter-docker.zip).
+
+Alternatively, you can clone the [Apache Druid repo](https://github.com/apache/druid) and
+access the files in `druid/examples/quickstart/jupyter-notebooks/docker-jupyter`.
+
+### Start only the Jupyter container
+
+If you already have Druid running locally, you can run only the Jupyter container to complete the tutorials.
+In the same directory as `docker-compose.yaml`, start the application:
+
+```bash
+docker-compose --profile jupyter up -d
+```
+
+The port assigned to Jupyter is `8889` by default.
+You can override the port number by setting the `JUPYTER_PORT` environment variable before starting the Docker application.
+
+### Start Jupyter and Druid
+
+Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable.
+`DRUID_VERSION` supplies the Docker tag that specifies which version of Druid to pull from the
+[Apache Druid Docker Hub](https://hub.docker.com/r/apache/druid/tags).
+
+In the same directory as `docker-compose.yaml` and `environment`, start the application:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile druid-jupyter up -d
+```
+
+### Start Jupyter, Druid, and Kafka
+
+Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable.
+
+In the same directory as `docker-compose.yaml` and `environment`, start the application:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile all-services up -d
+```
+
+### Use locally built image
+
+The default Docker Compose file pulls the custom Jupyter Notebook image from Imply's Docker Hub.
+If you would rather build the image locally, do the following:
Review Comment:
```suggestion
If you prefer to build the image locally, do the following:
```
avoid conditional
##########
docs/tutorials/tutorial-jupyter-docker.md:
##########
@@ -0,0 +1,181 @@
+---
+id: tutorial-jupyter-docker
+title: "Docker for Jupyter Notebook tutorials"
+sidebar_label: "Docker for tutorials"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Apache Druid provides a custom Jupyter container that contains the prerequisites
+for all Jupyter-based Druid tutorials, as well as all of the tutorials themselves.
+You can run the Jupyter container, as well as containers for Druid and Apache Kafka,
+using the Docker Compose file provided in the Druid GitHub repository.
+
+You can run the following combination of applications:
+* [Jupyter only](#start-only-the-jupyter-container)
+* [Jupyter and Druid](#start-jupyter-and-druid)
+* [Jupyter, Druid, and Kafka](#start-jupyter-druid-and-kafka)
+
+## Prerequisites
+
+Jupyter in Docker requires that you have **Docker** and **Docker Compose**.
+We recommend installing these through [Docker Desktop](https://docs.docker.com/desktop/).
+
+## Launch the Docker containers
+
+You run Docker Compose to launch Jupyter and optionally Druid or Kafka.
+Docker Compose references the configuration in `docker-compose.yaml`.
+Running Druid in Docker also requires the `environment` file, which
+sets the configuration properties for the Druid services.
+To get started, download both `docker-compose.yaml` and `environment` from
+[`tutorial-jupyter-docker.zip`](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/docker-jupyter/tutorial-jupyter-docker.zip).
+
+Alternatively, you can clone the [Apache Druid repo](https://github.com/apache/druid) and
+access the files in `druid/examples/quickstart/jupyter-notebooks/docker-jupyter`.
+
+### Start only the Jupyter container
+
+If you already have Druid running locally, you can run only the Jupyter container to complete the tutorials.
+In the same directory as `docker-compose.yaml`, start the application:
+
+```bash
+docker-compose --profile jupyter up -d
+```
+
+The port assigned to Jupyter is `8889` by default.
+You can override the port number by setting the `JUPYTER_PORT` environment variable before starting the Docker application.
+
+### Start Jupyter and Druid
+
+Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable.
+`DRUID_VERSION` supplies the Docker tag that specifies which version of Druid to pull from the
+[Apache Druid Docker Hub](https://hub.docker.com/r/apache/druid/tags).
+
+In the same directory as `docker-compose.yaml` and `environment`, start the application:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile druid-jupyter up -d
+```
+
+### Start Jupyter, Druid, and Kafka
+
+Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable.
+
+In the same directory as `docker-compose.yaml` and `environment`, start the application:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile all-services up -d
+```
+
+### Use locally built image
+
+The default Docker Compose file pulls the custom Jupyter Notebook image from Imply's Docker Hub.
+If you would rather build the image locally, do the following:
+1. Clone the Apache Druid repository.
+2. Navigate to `examples/quickstart/jupyter-notebooks/docker-jupyter`.
+3. Start the services using `-f docker-compose-local.yaml` in the `docker-compose` command. For example:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile all-services -f docker-compose-local.yaml up -d
+```
+
+## Access Jupyter-based tutorials
+
+At startup, Docker creates and mounts a volume to persist data from the container to your local machine.
+Access the files created in the Docker container in the `notebooks` folder in your local working directory.
+
+1. Navigate to available notebooks at http://localhost:8889.
+ > If you set `JUPYTER_PORT` to another port number, replace `8889` with the value of the Jupyter port.
+
+2. Select a tutorial.
+
+3. From the navigation menu, select **File > Save as...**.
Review Comment:
you only need to do this if you want to edit the notebook? If so, Maybe make an optional step?
##########
docs/tutorials/tutorial-jupyter-docker.md:
##########
@@ -0,0 +1,181 @@
+---
+id: tutorial-jupyter-docker
+title: "Docker for Jupyter Notebook tutorials"
+sidebar_label: "Docker for tutorials"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Apache Druid provides a custom Jupyter container that contains the prerequisites
+for all Jupyter-based Druid tutorials, as well as all of the tutorials themselves.
+You can run the Jupyter container, as well as containers for Druid and Apache Kafka,
+using the Docker Compose file provided in the Druid GitHub repository.
+
+You can run the following combination of applications:
+* [Jupyter only](#start-only-the-jupyter-container)
+* [Jupyter and Druid](#start-jupyter-and-druid)
+* [Jupyter, Druid, and Kafka](#start-jupyter-druid-and-kafka)
+
+## Prerequisites
+
+Jupyter in Docker requires that you have **Docker** and **Docker Compose**.
+We recommend installing these through [Docker Desktop](https://docs.docker.com/desktop/).
+
+## Launch the Docker containers
+
+You run Docker Compose to launch Jupyter and optionally Druid or Kafka.
+Docker Compose references the configuration in `docker-compose.yaml`.
+Running Druid in Docker also requires the `environment` file, which
+sets the configuration properties for the Druid services.
+To get started, download both `docker-compose.yaml` and `environment` from
+[`tutorial-jupyter-docker.zip`](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/docker-jupyter/tutorial-jupyter-docker.zip).
+
+Alternatively, you can clone the [Apache Druid repo](https://github.com/apache/druid) and
+access the files in `druid/examples/quickstart/jupyter-notebooks/docker-jupyter`.
+
+### Start only the Jupyter container
+
+If you already have Druid running locally, you can run only the Jupyter container to complete the tutorials.
+In the same directory as `docker-compose.yaml`, start the application:
+
+```bash
+docker-compose --profile jupyter up -d
+```
+
+The port assigned to Jupyter is `8889` by default.
+You can override the port number by setting the `JUPYTER_PORT` environment variable before starting the Docker application.
+
+### Start Jupyter and Druid
+
+Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable.
+`DRUID_VERSION` supplies the Docker tag that specifies which version of Druid to pull from the
+[Apache Druid Docker Hub](https://hub.docker.com/r/apache/druid/tags).
+
+In the same directory as `docker-compose.yaml` and `environment`, start the application:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile druid-jupyter up -d
+```
+
+### Start Jupyter, Druid, and Kafka
+
+Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable.
+
+In the same directory as `docker-compose.yaml` and `environment`, start the application:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile all-services up -d
+```
+
+### Use locally built image
+
+The default Docker Compose file pulls the custom Jupyter Notebook image from Imply's Docker Hub.
+If you would rather build the image locally, do the following:
+1. Clone the Apache Druid repository.
+2. Navigate to `examples/quickstart/jupyter-notebooks/docker-jupyter`.
+3. Start the services using `-f docker-compose-local.yaml` in the `docker-compose` command. For example:
+
+```bash
+DRUID_VERSION=25.0.0 docker-compose --profile all-services -f docker-compose-local.yaml up -d
+```
+
+## Access Jupyter-based tutorials
+
+At startup, Docker creates and mounts a volume to persist data from the container to your local machine.
+Access the files created in the Docker container in the `notebooks` folder in your local working directory.
+
+1. Navigate to available notebooks at http://localhost:8889.
+ > If you set `JUPYTER_PORT` to another port number, replace `8889` with the value of the Jupyter port.
+
+2. Select a tutorial.
+
+3. From the navigation menu, select **File > Save as...**.
+In the **Save As** dialog, enter `work/<notebook name>.ipynb`. This step allows you to retain a local copy of your work in the notebook. If the notebook still displays as read only, you may need to refresh the page in your browser.
+
+## View the Druid web console
+
+To access the Druid web console in Docker, go to http://localhost:8888/unified-console.html.
+Use the web console to view datasources and ingestion tasks that you create in the tutorials.
+
+## Stop Docker containers
+
+Shut down the Docker application using the following command:
+
+```bash
+docker-compose down -v
+```
+
+## Tutorial setup without using Docker
+
+To use the Jupyter Notebook-based tutorials without using Docker, do the following:
+
+1. Clone the Apache Druid repo, or download the [tutorials](tutorial-jupyter-index.md#tutorials)
+as well as the [Python client for Druid](tutorial-jupyter-index.md#python-api-for-druid).
+
+2. Install the prerequisite Python packages with the following commands:
Review Comment:
Add note that individual notebooks may require you to install additional packages and that they'll be listed w/in the notebook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org