You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "errose28 (via GitHub)" <gi...@apache.org> on 2024/11/06 00:43:54 UTC
Re: [PR] HDDS-11631. Add Run with Docker Compose guide [ozone-site]
errose28 commented on code in PR #101:
URL: https://github.com/apache/ozone-site/pull/101#discussion_r1830234822
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other pre-built image).
- Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+
Review Comment:
We can delete the TODO above this section since we are filling in the page in this PR.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other pre-built image).
- Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official pre-built Docker images.
Review Comment:
I think we should organize this a little differently. We can have one section on running docker-compose from a pre-build Ozone tarball, regardless of where the tarball came from (built from source or Ozone release). Then we can mention the pre-built docker images for each release separately if people want to use those for other things.
I don't see why someone would pull just the compose file from the tarball and manually pair it with a docker image built from the same version.
##########
package.json:
##########
@@ -1,5 +1,5 @@
{
- "packageManager": "pnpm@8.7.1",
+ "packageManager": "pnpm@9.12.3",
Review Comment:
Can you clarify why we need the pnpm version bump? Is mermaid incompatible with older versions?
##########
cspell.yaml:
##########
@@ -48,6 +48,7 @@ words:
- ratis
- OM
- SCM
+- DN
Review Comment:
Unlike OM and SCM we don't usually use `DN` as an abbreviation for datanode when talking about Ozone in an official context, so it was deliberately omitted here. It might be better to use an inline ignore to skip it in the mermaid code block.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other pre-built image).
- Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+ https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml \
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
+
+This image shows the containers that will be created when running the `docker-compose up -d` command.
+
+```mermaid
+graph TB
+ subgraph "Apache Ozone Architecture"
+ scm["SCM<br/>Storage Container Manager<br/>(Port: 9876, 9860)"]
+ om["OM<br/>Ozone Manager<br/>(Port: 9874, 9862)"]
+ s3g["S3G<br/>S3 Gateway<br/>(Port: 9878)"]
+ recon["Recon<br/>Monitoring Service<br/>(Port: 9888)"]
+ httpfs["HttpFS<br/>HTTP FileSystem<br/>(Port: 14000)"]
+
+ subgraph "Data Nodes"
+ dn1["DataNode 1<br/>(Port: 19864, 9882)"]
+ dnN["DataNode N<br/>(Port: 19864, 9882)"]
+ end
+
+ %% Connections
+ scm --> dn1
+ scm --> dnN
+ om --> dn1
+ om --> dnN
+ s3g --> om
+ s3g --> scm
+ httpfs --> om
+ httpfs --> scm
+ recon --> scm
+ recon --> om
+ end
+
+ classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
+ classDef datanode fill:#e1f5fe,stroke:#0288d1,stroke-width:2px;
+ classDef manager fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ class dn1,dnN datanode;
+ class scm,om manager;
+```
+
+## Cluster Configuration
+
+### Default Services
+
+The default Docker Compose configuration includes:
+
+- Storage Container Manager (SCM)
+- Ozone Manager (OM)
+- S3 Gateway
+- Recon (Monitoring Service)
+- Datanodes
+- HttpFS
+
+### Port Mappings
+
+The default setup exposes the following ports:
Review Comment:
I'm not sure the RPC connections will work with outside clients. the configuration and hostnames would need to be configured manually. HTTP endpoints should work out of the box though.
##########
package.json:
##########
@@ -16,22 +16,23 @@
},
"dependencies": {
"@docusaurus/core": "3.3.2",
- "@docusaurus/preset-classic": "3.3.2",
"@docusaurus/plugin-pwa": "3.3.2",
+ "@docusaurus/preset-classic": "3.3.2",
+ "@docusaurus/theme-mermaid": "3.3.2",
Review Comment:
I checked out this change and ran `docker compse up --build` to build an image with the new dependencies but got this error. Not sure exactly what the cause is:
```
ozone-site-site-1 |
ozone-site-site-1 | > doc-site@0.1.0 start /ozone-site
ozone-site-site-1 | > docusaurus start --port 3001 "--host" "0.0.0.0"
ozone-site-site-1 |
ozone-site-site-1 | [INFO] Starting the development server...
ozone-site-site-1 |
ozone-site-site-1 | [ERROR] Error: Docusaurus was unable to resolve the "@docusaurus/theme-mermaid" theme. Make sure one of the following packages are installed:
ozone-site-site-1 | - @docusaurus/theme-mermaid
ozone-site-site-1 | - @docusaurus/docusaurus-theme-theme-mermaid
ozone-site-site-1 | at resolveModuleName (/ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/server/plugins/moduleShorthand.js:41:15)
ozone-site-site-1 | at normalizeShorthand (/ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/server/plugins/configs.js:76:60)
ozone-site-site-1 | at /ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/server/plugins/configs.js:90:63
ozone-site-site-1 | at Array.map (<anonymous>)
ozone-site-site-1 | at loadPluginConfigs (/ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/server/plugins/configs.js:90:48)
ozone-site-site-1 | at async initPlugins (/ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/server/plugins/init.js:42:27)
ozone-site-site-1 | at async /ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:136:36
ozone-site-site-1 | at async loadSite (/ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/server/site.js:127:45)
ozone-site-site-1 | at async createReloadableSite (/ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/commands/start/utils.js:46:16)
ozone-site-site-1 | at async Command.start (/ozone-site/node_modules/.pnpm/@docusaurus+core@3.3.2_@docusaurus+types@3.3.2_react-dom@18.2.0_react@18.2.0_typescript@5.4.5/node_modules/@docusaurus/core/lib/commands/start/start.js:22:28)
ozone-site-site-1 | [INFO] Docusaurus version: 3.3.2
ozone-site-site-1 | Node version: v20.18.0
ozone-site-site-1 | ELIFECYCLE Command failed with exit code 1.
ozone-site-site-1 exited with code 1
```
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other pre-built image).
- Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
Review Comment:
Let's link to the relevant pages on the docker website for installing these components.
##########
docs/06-troubleshooting/15-docker.md:
##########
@@ -0,0 +1,7 @@
+---
+sidebar_label: Docker
+---
+
+# Troubleshooting Docker
Review Comment:
```suggestion
# Troubleshooting Ozone in Docker
```
Or some similar title to indicate we aren't troubleshooting docker in general.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other pre-built image).
- Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+ https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml \
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
+
+This image shows the containers that will be created when running the `docker-compose up -d` command.
+
+```mermaid
+graph TB
+ subgraph "Apache Ozone Architecture"
+ scm["SCM<br/>Storage Container Manager<br/>(Port: 9876, 9860)"]
+ om["OM<br/>Ozone Manager<br/>(Port: 9874, 9862)"]
+ s3g["S3G<br/>S3 Gateway<br/>(Port: 9878)"]
+ recon["Recon<br/>Monitoring Service<br/>(Port: 9888)"]
+ httpfs["HttpFS<br/>HTTP FileSystem<br/>(Port: 14000)"]
+
+ subgraph "Data Nodes"
+ dn1["DataNode 1<br/>(Port: 19864, 9882)"]
+ dnN["DataNode N<br/>(Port: 19864, 9882)"]
+ end
+
+ %% Connections
+ scm --> dn1
+ scm --> dnN
+ om --> dn1
+ om --> dnN
+ s3g --> om
+ s3g --> scm
+ httpfs --> om
+ httpfs --> scm
+ recon --> scm
+ recon --> om
+ end
+
+ classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
+ classDef datanode fill:#e1f5fe,stroke:#0288d1,stroke-width:2px;
+ classDef manager fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ class dn1,dnN datanode;
+ class scm,om manager;
+```
+
+## Cluster Configuration
+
+### Default Services
+
+The default Docker Compose configuration includes:
+
+- Storage Container Manager (SCM)
+- Ozone Manager (OM)
+- S3 Gateway
+- Recon (Monitoring Service)
+- Datanodes
+- HttpFS
+
+### Port Mappings
+
+The default setup exposes the following ports:
+
+- SCM: 9876 (RPC), 9860 (Client)
+- OM: 9874 (RPC), 9862 (Client)
+- S3 Gateway: 9878
+- Recon: 9888
+- HttpFS: 14000
+- Datanodes: 19864 (Container), 9882 (Client)
+
+## Cluster Management
+
+Common Docker Compose commands:
+
+```bash
+# Start the cluster
+docker-compose up -d
+
+# Stop the cluster
+docker-compose down
+
+# View service logs
+docker-compose logs -f [service_name]
+
+# Scale data nodes
+docker-compose up -d --scale datanode=3
Review Comment:
This only works on clusters that provide a service called `datanode`. Some of the compose clusters like `topology` and `upgrade` that need to use individual datanodes differently give each datanode a different service.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other pre-built image).
- Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+ https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml \
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
+
+This image shows the containers that will be created when running the `docker-compose up -d` command.
+
+```mermaid
+graph TB
+ subgraph "Apache Ozone Architecture"
+ scm["SCM<br/>Storage Container Manager<br/>(Port: 9876, 9860)"]
+ om["OM<br/>Ozone Manager<br/>(Port: 9874, 9862)"]
+ s3g["S3G<br/>S3 Gateway<br/>(Port: 9878)"]
+ recon["Recon<br/>Monitoring Service<br/>(Port: 9888)"]
+ httpfs["HttpFS<br/>HTTP FileSystem<br/>(Port: 14000)"]
+
+ subgraph "Data Nodes"
+ dn1["DataNode 1<br/>(Port: 19864, 9882)"]
+ dnN["DataNode N<br/>(Port: 19864, 9882)"]
+ end
+
+ %% Connections
+ scm --> dn1
+ scm --> dnN
+ om --> dn1
+ om --> dnN
+ s3g --> om
+ s3g --> scm
+ httpfs --> om
+ httpfs --> scm
+ recon --> scm
+ recon --> om
+ end
+
+ classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
+ classDef datanode fill:#e1f5fe,stroke:#0288d1,stroke-width:2px;
+ classDef manager fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ class dn1,dnN datanode;
+ class scm,om manager;
+```
+
+## Cluster Configuration
+
+### Default Services
+
+The default Docker Compose configuration includes:
+
+- Storage Container Manager (SCM)
+- Ozone Manager (OM)
+- S3 Gateway
+- Recon (Monitoring Service)
+- Datanodes
+- HttpFS
+
+### Port Mappings
+
+The default setup exposes the following ports:
+
+- SCM: 9876 (RPC), 9860 (Client)
+- OM: 9874 (RPC), 9862 (Client)
+- S3 Gateway: 9878
+- Recon: 9888
+- HttpFS: 14000
+- Datanodes: 19864 (Container), 9882 (Client)
+
+## Cluster Management
+
+Common Docker Compose commands:
+
+```bash
+# Start the cluster
+docker-compose up -d
Review Comment:
Lets use `docker compose` since v2 is the standard going forward.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other pre-built image).
- Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+ https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml \
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
Review Comment:
To avoid duplicate content I think we should leave this information for the page dedicated to Ozone architecture. Default ports will be covered in `docs/administrator-guide/configuration/basic/network/`. This may mean we don't need to add mermaid in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org