You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/08/04 23:35:18 UTC
[GitHub] [druid] sthetland commented on a change in pull request #11420: Add more guidelines on the use of aliyun-oss-extensions
sthetland commented on a change in pull request #11420:
URL: https://github.com/apache/druid/pull/11420#discussion_r683022529
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
Review comment:
I might explicitly give the name of the extension here, since the link takes you to the section with all extensions.
```suggestion
Use the [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install the `aliyun-oss-extensions` extension, as described [here](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.|
-|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | |
+to use OSS as deep storage, add following configurations:
Review comment:
```suggestion
To use OSS as deep storage, add the following configurations:
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
Review comment:
```suggestion
First add the following OSS configurations to common.runtime.properties
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
Review comment:
```suggestion
|`druid.oss.accessKey`|The `AccessKey ID` of the account to be used to access the OSS bucket|yes|
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
Review comment:
```suggestion
|`druid.oss.endpoint`|The endpoint URL of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.|
-|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | |
+to use OSS as deep storage, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. |yes|
+|`druid.storage.oss.bucket`|storage bucket name.| yes |
+|`druid.storage.oss.prefix`| folder where segments will be published to. `druid/segments` is recommended. | No |
+
+If OSS is used as deep storage for segment files, it's also recommended saving index logs in the OSS too.
+To do this, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | yes |
+|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs. It could be the same as `druid.storage.oss.bucket`| yes |
+|`druid.indexer.logs.oss.prefix`|folder where log files will be published to. `druid/logs` is recommended. | no |
+
+
+## Reading data from OSS
+
+Currently, Web Console does not support ingestion from OSS, but it could be done by submitting an ingestion task with OSS's input source configuration.
+
+Below shows the configurations of OSS's input source.
+
+### OSS Input Source
+
+|property|description|Required|
+|--------|-----------|-------|
+|type|This should be `oss`.|yes|
+|uris|JSON array of URIs where OSS objects to be ingested are located.<br/>For example, `oss://{your_bucket}/{source_file_path}`|`uris` or `prefixes` or `objects` must be set|
+|prefixes|JSON array of URI prefixes for the locations of OSS objects to be ingested. Empty objects starting with one of the given prefixes will be skipped.|`uris` or `prefixes` or `objects` must be set|
+|objects|JSON array of [OSS Objects](#oss-object) to be ingested. |`uris` or `prefixes` or `objects` must be set|
+|properties|[Properties Object](#properties-object) for overriding the default OSS configuration. See below for more information.|no (defaults will be used if not given)
+
+#### OSS Object
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|---------|
+|bucket|Name of the OSS bucket|None|yes|
+|path|The path where data is located.|None|yes|
+
+#### Properties Object
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|---------|
+|accessKey|The [Password Provider](../../operations/password-provider.md) or plain text string of this OSS InputSource's access key|None|yes|
+|secretKey|The [Password Provider](../../operations/password-provider.md) or plain text string of this OSS InputSource's secret key|None|yes|
+|endpoint|The endpoint of this OSS InputSource|None|no|
+
+### Reading from a file
+
+Say that the file `rollup-data.json`, which could be found under Druid's `quickstart/tutorial` directory, has been uploaded to a folder `druid` in your OSS bucket, which is the same bucket your Druid is configured ,
Review comment:
Splitting into two sentences. i think this is what's meant, but double check please:
```suggestion
Say that the file `rollup-data.json`, which can be found under Druid's `quickstart/tutorial` directory, has been uploaded to a folder `druid` in your OSS bucket, the bucket for which your Druid is configured.
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.|
-|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | |
+to use OSS as deep storage, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. |yes|
+|`druid.storage.oss.bucket`|storage bucket name.| yes |
+|`druid.storage.oss.prefix`| folder where segments will be published to. `druid/segments` is recommended. | No |
+
+If OSS is used as deep storage for segment files, it's also recommended saving index logs in the OSS too.
+To do this, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | yes |
+|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs. It could be the same as `druid.storage.oss.bucket`| yes |
Review comment:
```suggestion
|`druid.indexer.logs.oss.bucket`|The bucket used to keep logs. It could be the same as `druid.storage.oss.bucket`| yes |
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.|
-|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | |
+to use OSS as deep storage, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. |yes|
+|`druid.storage.oss.bucket`|storage bucket name.| yes |
Review comment:
```suggestion
|`druid.storage.oss.bucket`|Storage bucket name.| yes |
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
Review comment:
```suggestion
|`druid.oss.secretKey`|The `AccessKey Secret` of the account to be used to access the OSS bucket| yes|
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.|
-|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | |
+to use OSS as deep storage, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. |yes|
+|`druid.storage.oss.bucket`|storage bucket name.| yes |
+|`druid.storage.oss.prefix`| folder where segments will be published to. `druid/segments` is recommended. | No |
Review comment:
```suggestion
|`druid.storage.oss.prefix`| Folder where segments will be published to. `druid/segments` is recommended. | No |
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.|
-|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | |
+to use OSS as deep storage, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. |yes|
+|`druid.storage.oss.bucket`|storage bucket name.| yes |
+|`druid.storage.oss.prefix`| folder where segments will be published to. `druid/segments` is recommended. | No |
+
+If OSS is used as deep storage for segment files, it's also recommended saving index logs in the OSS too.
+To do this, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | yes |
+|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs. It could be the same as `druid.storage.oss.bucket`| yes |
+|`druid.indexer.logs.oss.prefix`|folder where log files will be published to. `druid/logs` is recommended. | no |
Review comment:
```suggestion
|`druid.indexer.logs.oss.prefix`|Folder where log files will be published to. `druid/logs` is recommended. | no |
```
##########
File path: docs/development/extensions-contrib/aliyun-oss-extensions.md
##########
@@ -22,33 +22,215 @@ title: "Aliyun OSS"
~ under the License.
-->
+[Alibaba Cloud](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+This document describes how to use OSS as Druid deep storage.
-To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension.
+## Installation
-## Deep Storage
+Use [pull-deps](../../operations/pull-deps.md) tool shipped with Druid to install this [extension](../../development/extensions.md#community-extensions) on middle manager and historical nodes.
-[Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss).
+```bash
+java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION}
+```
-To use aliyun OSS as deep storage, first config as below
+## Enabling
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.|
-|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. |
-|`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.|
+After installation, add this `aliyun-oss-extensions` extension to `druid.extensions.loadList` in common.runtime.properties and then restart middle manager and historical nodes.
-if you want to use OSS as deep storage, use the configurations below
+## Configuration
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.storage.oss.bucket`|storage bucket name.| | Must be set.|
-|`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | |
+First add following OSS configurations to common.runtime.properties
-To save index logs to OSS, apply the configurations below:
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the OSS bucket|yes|
+|`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the OSS bucket| yes|
+|`druid.oss.endpoint`|the endpoint url of your OSS storage. <br/>If your Druid cluster is also hosted in the same region on Alibaba Cloud as the region of your OSS bucket, it's recommended to use the internal network endpoint url, so that any inbound and outbound traffic to the OSS bucket is free of charge. | yes|
-|Property|Description|Possible Values|Default|
-|--------|---------------|-----------|-------|
-|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.|
-|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.|
-|`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | |
+to use OSS as deep storage, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. |yes|
+|`druid.storage.oss.bucket`|storage bucket name.| yes |
+|`druid.storage.oss.prefix`| folder where segments will be published to. `druid/segments` is recommended. | No |
+
+If OSS is used as deep storage for segment files, it's also recommended saving index logs in the OSS too.
+To do this, add following configurations:
+
+|Property|Description|Required|
+|--------|---------------|-----------|
+|`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | yes |
+|`druid.indexer.logs.oss.bucket`|the bucket used to keep logs. It could be the same as `druid.storage.oss.bucket`| yes |
+|`druid.indexer.logs.oss.prefix`|folder where log files will be published to. `druid/logs` is recommended. | no |
+
+
+## Reading data from OSS
+
+Currently, Web Console does not support ingestion from OSS, but it could be done by submitting an ingestion task with OSS's input source configuration.
+
+Below shows the configurations of OSS's input source.
+
+### OSS Input Source
+
+|property|description|Required|
+|--------|-----------|-------|
+|type|This should be `oss`.|yes|
+|uris|JSON array of URIs where OSS objects to be ingested are located.<br/>For example, `oss://{your_bucket}/{source_file_path}`|`uris` or `prefixes` or `objects` must be set|
+|prefixes|JSON array of URI prefixes for the locations of OSS objects to be ingested. Empty objects starting with one of the given prefixes will be skipped.|`uris` or `prefixes` or `objects` must be set|
+|objects|JSON array of [OSS Objects](#oss-object) to be ingested. |`uris` or `prefixes` or `objects` must be set|
+|properties|[Properties Object](#properties-object) for overriding the default OSS configuration. See below for more information.|no (defaults will be used if not given)
+
+#### OSS Object
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|---------|
+|bucket|Name of the OSS bucket|None|yes|
+|path|The path where data is located.|None|yes|
+
+#### Properties Object
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|---------|
+|accessKey|The [Password Provider](../../operations/password-provider.md) or plain text string of this OSS InputSource's access key|None|yes|
+|secretKey|The [Password Provider](../../operations/password-provider.md) or plain text string of this OSS InputSource's secret key|None|yes|
+|endpoint|The endpoint of this OSS InputSource|None|no|
+
+### Reading from a file
+
+Say that the file `rollup-data.json`, which could be found under Druid's `quickstart/tutorial` directory, has been uploaded to a folder `druid` in your OSS bucket, which is the same bucket your Druid is configured ,
+`uri` property of OSS's input source could be used for the reading.
Review comment:
Should it be `uris` property, rather than `uri`?
```suggestion
In this case, the `uri` property of the OSS's input source can be used for reading, as shown:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org