You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by vo...@apache.org on 2022/09/16 09:17:45 UTC
[druid] branch 24.0.0 updated: Doc fixes around msq (#13090)
This is an automated email from the ASF dual-hosted git repository.
vogievetsky pushed a commit to branch 24.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/24.0.0 by this push:
new 7c8cfb890f Doc fixes around msq (#13090)
7c8cfb890f is described below
commit 7c8cfb890f467a2a452134c763e50aaab3194646
Author: Vadim Ogievetsky <va...@ogievetsky.com>
AuthorDate: Fri Sep 16 02:15:26 2022 -0700
Doc fixes around msq (#13090)
* remove things that do not apply
* fix more things
* pin node to a working version
* fix
* fixes
* known issues tidy up
* revert auto formatting changes
* remove management-uis page which is 100% lies
* don't mention the Coordinator console (that no longer exits)
* goodies
* fix typo
---
docs/configuration/index.md | 5 +-
docs/design/coordinator.md | 4 -
docs/design/overlord.md | 4 -
docs/development/modules.md | 4 +-
docs/ingestion/data-formats.md | 4 +-
docs/ingestion/faq.md | 4 +-
docs/multi-stage-query/index.md | 12 +-
docs/multi-stage-query/msq-api.md | 1669 ++++----------------
docs/multi-stage-query/msq-concepts.md | 2 +-
docs/multi-stage-query/msq-example-queries.md | 48 +-
docs/multi-stage-query/msq-known-issues.md | 104 +-
docs/multi-stage-query/msq-reference.md | 6 +-
.../msq-tutorial-connect-external-data.md | 23 +-
.../msq-tutorial-convert-ingest-spec.md | 6 +-
docs/operations/management-uis.md | 64 -
docs/operations/rule-configuration.md | 6 +-
docs/tutorials/tutorial-msq-connect-extern.md | 4 +-
docs/tutorials/tutorial-msq-convert-json.md | 4 +-
website/package.json | 6 +
website/redirects.json | 1 +
website/sidebars.json | 3 +-
21 files changed, 402 insertions(+), 1581 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index e327249eca..25064e107d 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -881,9 +881,10 @@ These Coordinator static configurations can be defined in the `coordinator/runti
#### Dynamic Configuration
-The Coordinator has dynamic configuration to change certain behavior on the fly. The Coordinator uses a JSON spec object from the Druid [metadata storage](../dependencies/metadata-storage.md) config table. This object is detailed below:
+The Coordinator has dynamic configuration to change certain behavior on the fly.
-It is recommended that you use the Coordinator Console to configure these parameters. However, if you need to do it via HTTP, the JSON object can be submitted to the Coordinator via a POST request at:
+It is recommended that you use the [web console](../operations/druid-console.md) to configure these parameters.
+However, if you need to do it via HTTP, the JSON object can be submitted to the Coordinator via a POST request at:
```
http://<COORDINATOR_IP>:<PORT>/druid/coordinator/v1/config
diff --git a/docs/design/coordinator.md b/docs/design/coordinator.md
index 4fd27dc248..c2e2f26f7e 100644
--- a/docs/design/coordinator.md
+++ b/docs/design/coordinator.md
@@ -145,10 +145,6 @@ For more information, see [Avoid conflicts with ingestion](../ingestion/automati
> and their total size exceeds [`inputSegmentSizeBytes`](../configuration/index.md#automatic-compaction-dynamic-configuration).
> If it finds such segments, it simply skips them.
-### The Coordinator console
-
-The Druid Coordinator exposes a web GUI for displaying cluster information and rule configuration. For more details, see [Coordinator console](../operations/management-uis.md#coordinator-consoles).
-
### FAQ
1. **Do clients ever contact the Coordinator process?**
diff --git a/docs/design/overlord.md b/docs/design/overlord.md
index 1530e5dc86..74c09dd590 100644
--- a/docs/design/overlord.md
+++ b/docs/design/overlord.md
@@ -40,10 +40,6 @@ In local mode Overlord is also responsible for creating Peons for executing task
Local mode is typically used for simple workflows. In remote mode, the Overlord and MiddleManager are run in separate processes and you can run each on a different server.
This mode is recommended if you intend to use the indexing service as the single endpoint for all Druid indexing.
-### Overlord console
-
-The Overlord provides a UI for managing tasks and workers. For more details, please see [overlord console](../operations/management-uis.md#overlord-console).
-
### Blacklisted workers
If a MiddleManager has task failures above a threshold, the Overlord will blacklist these MiddleManagers. No more than 20% of the MiddleManagers can be blacklisted. Blacklisted MiddleManagers will be periodically whitelisted.
diff --git a/docs/development/modules.md b/docs/development/modules.md
index 2d5f6a8238..bf2f178314 100644
--- a/docs/development/modules.md
+++ b/docs/development/modules.md
@@ -144,11 +144,11 @@ T00:00:00.000Z/2015-04-14T02:41:09.484Z/0/index.zip] to [/opt/druid/zk_druid/dde
* DataSegmentKiller
-The easiest way of testing the segment killing is marking a segment as not used and then starting a killing task through the old Coordinator console.
+The easiest way of testing the segment killing is marking a segment as not used and then starting a killing task in the [web console](../operations/druid-console.md).
To mark a segment as not used, you need to connect to your metadata storage and update the `used` column to `false` on the segment table rows.
-To start a segment killing task, you need to access the old Coordinator console `http://<COODRINATOR_IP>:<COORDINATOR_PORT/old-console/kill.html` then select the appropriate datasource and then input a time range (e.g. `2000/3000`).
+To start a segment killing task, you need to access the web console then select `issue kill task` for the appropriate datasource.
After the killing task ends, `index.zip` (`partitionNum_index.zip` for HDFS data storage) file should be deleted from the data storage.
diff --git a/docs/ingestion/data-formats.md b/docs/ingestion/data-formats.md
index bb1b659c49..ee2b822671 100644
--- a/docs/ingestion/data-formats.md
+++ b/docs/ingestion/data-formats.md
@@ -75,7 +75,9 @@ parsing data is less efficient than writing a native Java parser or using an ext
## Input format
You can use the `inputFormat` field to specify the data format for your input data.
-> `inputFormat` doesn't support all data formats or ingestion methods supported by Druid yet.
+
+> `inputFormat` doesn't support all data formats or ingestion methods supported by Druid.
+
Especially if you want to use the Hadoop ingestion, you still need to use the [Parser](#parser).
If your data is formatted in some format not listed in this section, please consider using the Parser instead.
diff --git a/docs/ingestion/faq.md b/docs/ingestion/faq.md
index 736ef0c6bb..38e99aa0d1 100644
--- a/docs/ingestion/faq.md
+++ b/docs/ingestion/faq.md
@@ -47,7 +47,7 @@ Other common reasons that hand-off fails are as follows:
1) Druid is unable to write to the metadata storage. Make sure your configurations are correct.
-2) Historical processes are out of capacity and cannot download any more segments. You'll see exceptions in the Coordinator logs if this occurs and the Coordinator console will show the Historicals are near capacity.
+2) Historical processes are out of capacity and cannot download any more segments. You'll see exceptions in the Coordinator logs if this occurs and the web console will show the Historicals are near capacity.
3) Segments are corrupt and cannot be downloaded. You'll see exceptions in your Historical processes if this occurs.
@@ -73,7 +73,7 @@ Note that this workflow only guarantees that the segments are available at the t
## I don't see my Druid segments on my Historical processes
-You can check the Coordinator console located at `<COORDINATOR_IP>:<PORT>`. Make sure that your segments have actually loaded on [Historical processes](../design/historical.md). If your segments are not present, check the Coordinator logs for messages about capacity of replication errors. One reason that segments are not downloaded is because Historical processes have maxSizes that are too small, making them incapable of downloading more data. You can change that with (for example):
+You can check the [web console](../operations/druid-console.md) to make sure that your segments have actually loaded on [Historical processes](../design/historical.md). If your segments are not present, check the Coordinator logs for messages about capacity of replication errors. One reason that segments are not downloaded is because Historical processes have maxSizes that are too small, making them incapable of downloading more data. You can change that with (for example):
```
-Ddruid.segmentCache.locations=[{"path":"/tmp/druid/storageLocation","maxSize":"500000000000"}]
diff --git a/docs/multi-stage-query/index.md b/docs/multi-stage-query/index.md
index b82408607f..07c943fbe7 100644
--- a/docs/multi-stage-query/index.md
+++ b/docs/multi-stage-query/index.md
@@ -81,7 +81,7 @@ SELECT
*
FROM TABLE(
EXTERN(
- '{"type": "http", "uris": ["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type": "http", "uris": ["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type": "json"}',
'[{"name": "timestamp", "type": "string"}, {"name": "page", "type": "string"}, {"name": "user", "type": "string"}]'
)
@@ -107,7 +107,7 @@ SELECT
"user"
FROM TABLE(
EXTERN(
- '{"type": "http", "uris": ["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type": "http", "uris": ["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type": "json"}',
'[{"name": "timestamp", "type": "string"}, {"name": "page", "type": "string"}, {"name": "user", "type": "string"}]'
)
@@ -145,7 +145,7 @@ SELECT
"user"
FROM TABLE(
EXTERN(
- '{"type": "http", "uris": ["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type": "http", "uris": ["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type": "json"}',
'[{"name": "timestamp", "type": "string"}, {"name": "page", "type": "string"}, {"name": "user", "type": "string"}]'
)
@@ -166,7 +166,7 @@ SELECT
"user"
FROM TABLE(
EXTERN(
- '{"type": "http", "uris": ["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type": "http", "uris": ["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type": "json"}',
'[{"name": "timestamp", "type": "string"}, {"name": "page", "type": "string"}, {"name": "user", "type": "string"}]'
)
@@ -197,7 +197,7 @@ For more information, see [Primary timestamp](../ingestion/data-model.md#primary
### PARTITIONED BY
-INSERT and REPLACE queries require the PARTITIONED BY clause, which determines how time-based partitioning is done. In Druid, data is split into segments, one or more per time chunk defined by the PARTITIONED BY granularity. A good general rule is to adjust the granularity so that each segment contains about five million rows. Choose a granularity based on your ingestion rate. For example, if you ingest a million rows per day, PARTITION BY DAY is good. If you ingest a million rows an hou [...]
+INSERT and REPLACE queries require the PARTITIONED BY clause, which determines how time-based partitioning is done. In Druid, data is split into segments, one or more per time chunk defined by the PARTITIONED BY granularity. A good general rule is to adjust the granularity so that each segment contains about five million rows. Choose a granularity based on your ingestion rate. For example, if you ingest a million rows per day, PARTITIONED BY DAY is good. If you ingest a million rows an h [...]
Using the clause provides the following benefits:
@@ -235,7 +235,7 @@ You can use the following ISO 8601 periods for `TIME_FLOOR`:
### CLUSTERED BY
-Data is first divided by the PARTITION BY clause. Data can be further split by the CLUSTERED BY clause. For example, suppose you ingest 100 M rows per hour and use `PARTITIONED BY HOUR` as your time partition. You then divide up the data further by adding `CLUSTERED BY hostName`. The result is segments of about 5 million rows, with like `hostNames` grouped within the same segment.
+Data is first divided by the PARTITIONED BY clause. Data can be further split by the CLUSTERED BY clause. For example, suppose you ingest 100 M rows per hour and use `PARTITIONED BY HOUR` as your time partition. You then divide up the data further by adding `CLUSTERED BY hostName`. The result is segments of about 5 million rows, with like `hostName`s grouped within the same segment.
Using CLUSTERED BY has the following benefits:
diff --git a/docs/multi-stage-query/msq-api.md b/docs/multi-stage-query/msq-api.md
index b9c2f165ac..89e568063a 100644
--- a/docs/multi-stage-query/msq-api.md
+++ b/docs/multi-stage-query/msq-api.md
@@ -57,38 +57,38 @@ POST /druid/v2/sql/task
```json
{
- "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://static.imply.io/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, {\"name\": \"cou [...]
+ "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, {\"name\": \"co [...]
"context": {
- "maxNumTasks": 3
+ "maxNumTasks": 3
}
}
```
<!--curl-->
-Make sure you replace `username`, `password`, `your-instance`, and `port` with the values for your deployment.
-
```bash
+# Make sure you replace `username`, `password`, `your-instance`, and `port` with the values for your deployment.
curl --location --request POST 'https://<username>:<password>@<your-instance>:<port>/druid/v2/sql/task/' \
---header 'Content-Type: application/json' \
---data-raw '{
- "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '\''{\"type\": \"http\", \"uris\": [\"https://static.imply.io/data/wikipedia.json.gz\"]}'\'',\n '\''{\"type\": \"json\"}'\'',\n '\''[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, [...]
+ --header 'Content-Type: application/json' \
+ --data-raw '{
+ "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '\''{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}'\'',\n '\''{\"type\": \"json\"}'\'',\n '\''[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"} [...]
"context": {
"maxNumTasks": 3
}
+ }'
```
<!--Python-->
-Make sure you replace `username`, `password`, `your-instance`, and `port` with the values for your deployment.
```python
import json
import requests
+# Make sure you replace `username`, `password`, `your-instance`, and `port` with the values for your deployment.
url = "https://<username>:<password>@<your-instance>:<port>/druid/v2/sql/task/"
payload = json.dumps({
- "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://static.imply.io/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, {\"name\": \"cou [...]
+ "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, {\"name\": \"co [...]
"context": {
"maxNumTasks": 3
}
@@ -123,492 +123,6 @@ print(response.text)
| state | Initial state for the query, which is "RUNNING".|
-## Get the payload for a query task
-
-You can retrieve basic information about a query task, such as the SQL query and context parameters that were submitted.
-
-### Request
-
-<!--DOCUSAURUS_CODE_TABS-->
-
-<!--HTTP-->
-
-```
-GET /druid/indexer/v1/task/<taskId>
-```
-
-<!--curl-->
-
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
-
-```bash
-curl --location --request GET 'https://<username>:<password>@<your-instance>:<port>/druid/indexer/v1/task/<taskId>'
-```
-
-<!--Python-->
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
-
-```python
-import requests
-
-url = "<username>:<password>@<your-instance>:<port>/druid/indexer/v1/task/<taskId>"
-
-payload={}
-headers = {}
-
-response = requests.request("GET", url, headers=headers, data=payload)
-
-print(response.text)
-
-```
-
-<!--END_DOCUSAURUS_CODE_TABS-->
-
-### Response
-
-<details><summary>Show the response</summary>
-
-```
-{
- "task": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "payload": {
- "type": "query_controller",
- "id": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "spec": {
- "query": {
- "queryType": "scan",
- "dataSource": {
- "type": "external",
- "inputSource": {
- "type": "http",
- "uris": [
- "https://static.imply.io/data/kttm/kttm-v2-2019-08-25.json.gz"
- ],
- "httpAuthenticationUsername": null,
- "httpAuthenticationPassword": null
- },
- "inputFormat": {
- "type": "json",
- "flattenSpec": null,
- "featureSpec": {},
- "keepNullColumns": false
- },
- "signature": [
- {
- "name": "timestamp",
- "type": "STRING"
- },
- {
- "name": "agent_category",
- "type": "STRING"
- },
- {
- "name": "agent_type",
- "type": "STRING"
- },
- {
- "name": "browser",
- "type": "STRING"
- },
- {
- "name": "browser_version",
- "type": "STRING"
- },
- {
- "name": "city",
- "type": "STRING"
- },
- {
- "name": "continent",
- "type": "STRING"
- },
- {
- "name": "country",
- "type": "STRING"
- },
- {
- "name": "version",
- "type": "STRING"
- },
- {
- "name": "event_type",
- "type": "STRING"
- },
- {
- "name": "event_subtype",
- "type": "STRING"
- },
- {
- "name": "loaded_image",
- "type": "STRING"
- },
- {
- "name": "adblock_list",
- "type": "STRING"
- },
- {
- "name": "forwarded_for",
- "type": "STRING"
- },
- {
- "name": "language",
- "type": "STRING"
- },
- {
- "name": "number",
- "type": "LONG"
- },
- {
- "name": "os",
- "type": "STRING"
- },
- {
- "name": "path",
- "type": "STRING"
- },
- {
- "name": "platform",
- "type": "STRING"
- },
- {
- "name": "referrer",
- "type": "STRING"
- },
- {
- "name": "referrer_host",
- "type": "STRING"
- },
- {
- "name": "region",
- "type": "STRING"
- },
- {
- "name": "remote_address",
- "type": "STRING"
- },
- {
- "name": "screen",
- "type": "STRING"
- },
- {
- "name": "session",
- "type": "STRING"
- },
- {
- "name": "session_length",
- "type": "LONG"
- },
- {
- "name": "timezone",
- "type": "STRING"
- },
- {
- "name": "timezone_offset",
- "type": "LONG"
- },
- {
- "name": "window",
- "type": "STRING"
- }
- ]
- },
- "intervals": {
- "type": "intervals",
- "intervals": [
- "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
- ]
- },
- "resultFormat": "compactedList",
- "columns": [
- "adblock_list",
- "agent_category",
- "agent_type",
- "browser",
- "browser_version",
- "city",
- "continent",
- "country",
- "event_subtype",
- "event_type",
- "forwarded_for",
- "language",
- "loaded_image",
- "number",
- "os",
- "path",
- "platform",
- "referrer",
- "referrer_host",
- "region",
- "remote_address",
- "screen",
- "session",
- "session_length",
- "timestamp",
- "timezone",
- "timezone_offset",
- "version",
- "window"
- ],
- "legacy": false,
- "context": {
- "finalize": true,
- "msqMaxNumTasks": 3,
- "msqSignature": "[{\"name\":\"adblock_list\",\"type\":\"STRING\"},{\"name\":\"agent_category\",\"type\":\"STRING\"},{\"name\":\"agent_type\",\"type\":\"STRING\"},{\"name\":\"browser\",\"type\":\"STRING\"},{\"name\":\"browser_version\",\"type\":\"STRING\"},{\"name\":\"city\",\"type\":\"STRING\"},{\"name\":\"continent\",\"type\":\"STRING\"},{\"name\":\"country\",\"type\":\"STRING\"},{\"name\":\"event_subtype\",\"type\":\"STRING\"},{\"name\":\"event_type\",\"type\":\"STR [...]
- "multiStageQuery": true,
- "sqlInsertSegmentGranularity": "{\"type\":\"all\"}",
- "sqlQueryId": "a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "sqlReplaceTimeChunks": "all"
- },
- "granularity": {
- "type": "all"
- }
- },
- "columnMappings": [
- {
- "queryColumn": "timestamp",
- "outputColumn": "timestamp"
- },
- {
- "queryColumn": "agent_category",
- "outputColumn": "agent_category"
- },
- {
- "queryColumn": "agent_type",
- "outputColumn": "agent_type"
- },
- {
- "queryColumn": "browser",
- "outputColumn": "browser"
- },
- {
- "queryColumn": "browser_version",
- "outputColumn": "browser_version"
- },
- {
- "queryColumn": "city",
- "outputColumn": "city"
- },
- {
- "queryColumn": "continent",
- "outputColumn": "continent"
- },
- {
- "queryColumn": "country",
- "outputColumn": "country"
- },
- {
- "queryColumn": "version",
- "outputColumn": "version"
- },
- {
- "queryColumn": "event_type",
- "outputColumn": "event_type"
- },
- {
- "queryColumn": "event_subtype",
- "outputColumn": "event_subtype"
- },
- {
- "queryColumn": "loaded_image",
- "outputColumn": "loaded_image"
- },
- {
- "queryColumn": "adblock_list",
- "outputColumn": "adblock_list"
- },
- {
- "queryColumn": "forwarded_for",
- "outputColumn": "forwarded_for"
- },
- {
- "queryColumn": "language",
- "outputColumn": "language"
- },
- {
- "queryColumn": "number",
- "outputColumn": "number"
- },
- {
- "queryColumn": "os",
- "outputColumn": "os"
- },
- {
- "queryColumn": "path",
- "outputColumn": "path"
- },
- {
- "queryColumn": "platform",
- "outputColumn": "platform"
- },
- {
- "queryColumn": "referrer",
- "outputColumn": "referrer"
- },
- {
- "queryColumn": "referrer_host",
- "outputColumn": "referrer_host"
- },
- {
- "queryColumn": "region",
- "outputColumn": "region"
- },
- {
- "queryColumn": "remote_address",
- "outputColumn": "remote_address"
- },
- {
- "queryColumn": "screen",
- "outputColumn": "screen"
- },
- {
- "queryColumn": "session",
- "outputColumn": "session"
- },
- {
- "queryColumn": "session_length",
- "outputColumn": "session_length"
- },
- {
- "queryColumn": "timezone",
- "outputColumn": "timezone"
- },
- {
- "queryColumn": "timezone_offset",
- "outputColumn": "timezone_offset"
- },
- {
- "queryColumn": "window",
- "outputColumn": "window"
- }
- ],
- "destination": {
- "type": "dataSource",
- "dataSource": "kttm_simple",
- "segmentGranularity": {
- "type": "all"
- },
- "replaceTimeChunks": [
- "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
- ]
- },
- "assignmentStrategy": "max",
- "tuningConfig": {
- "type": "index_parallel",
- "maxRowsPerSegment": 3000000,
- "appendableIndexSpec": {
- "type": "onheap",
- "preserveExistingMetrics": false
- },
- "maxRowsInMemory": 100000,
- "maxBytesInMemory": 0,
- "skipBytesInMemoryOverheadCheck": false,
- "maxTotalRows": null,
- "numShards": null,
- "splitHintSpec": null,
- "partitionsSpec": {
- "type": "dynamic",
- "maxRowsPerSegment": 3000000,
- "maxTotalRows": null
- },
- "indexSpec": {
- "bitmap": {
- "type": "roaring",
- "compressRunOnSerialization": true
- },
- "dimensionCompression": "lz4",
- "metricCompression": "lz4",
- "longEncoding": "longs",
- "segmentLoader": null
- },
- "indexSpecForIntermediatePersists": {
- "bitmap": {
- "type": "roaring",
- "compressRunOnSerialization": true
- },
- "dimensionCompression": "lz4",
- "metricCompression": "lz4",
- "longEncoding": "longs",
- "segmentLoader": null
- },
- "maxPendingPersists": 0,
- "forceGuaranteedRollup": false,
- "reportParseExceptions": false,
- "pushTimeout": 0,
- "segmentWriteOutMediumFactory": null,
- "maxNumConcurrentSubTasks": 2,
- "maxRetry": 1,
- "taskStatusCheckPeriodMs": 1000,
- "chatHandlerTimeout": "PT10S",
- "chatHandlerNumRetries": 5,
- "maxNumSegmentsToMerge": 100,
- "totalNumMergeTasks": 10,
- "logParseExceptions": false,
- "maxParseExceptions": 2147483647,
- "maxSavedParseExceptions": 0,
- "maxColumnsToMerge": -1,
- "awaitSegmentAvailabilityTimeoutMillis": 0,
- "maxAllowedLockCount": -1,
- "partitionDimensions": []
- }
- },
- "sqlQuery": "REPLACE INTO \"kttm_simple\" OVERWRITE ALL\nSELECT *\nFROM TABLE(\n EXTERN(\n '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/data/kttm/kttm-v2-2019-08-25.json.gz\"]}',\n '{\"type\":\"json\"}',\n '[{\"name\":\"timestamp\",\"type\":\"string\"},{\"name\":\"agent_category\",\"type\":\"string\"},{\"name\":\"agent_type\",\"type\":\"string\"},{\"name\":\"browser\",\"type\":\"string\"},{\"name\":\"browser_version\",\"type\":\"string\"},{\"name\":\"city\",\ [...]
- "sqlQueryContext": {
- "parseExceptions": 0,
- "maxNumTasks": 3,
- "signature": "[{\"name\":\"adblock_list\",\"type\":\"STRING\"},{\"name\":\"agent_category\",\"type\":\"STRING\"},{\"name\":\"agent_type\",\"type\":\"STRING\"},{\"name\":\"browser\",\"type\":\"STRING\"},{\"name\":\"browser_version\",\"type\":\"STRING\"},{\"name\":\"city\",\"type\":\"STRING\"},{\"name\":\"continent\",\"type\":\"STRING\"},{\"name\":\"country\",\"type\":\"STRING\"},{\"name\":\"event_subtype\",\"type\":\"STRING\"},{\"name\":\"event_type\",\"type\":\"STRING\"},{\"n [...]
- "multiStageQuery": true,
- "sqlInsertSegmentGranularity": "{\"type\":\"all\"}",
- "sqlQueryId": "a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "sqlReplaceTimeChunks": "all"
- },
- "sqlTypeNames": [
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "BIGINT",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "VARCHAR",
- "BIGINT",
- "VARCHAR",
- "BIGINT",
- "VARCHAR"
- ],
- "context": {
- "forceTimeChunkLock": true,
- "useLineageBasedSegmentAllocation": true
- },
- "groupId": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "dataSource": "kttm_simple",
- "resource": {
- "availabilityGroup": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "requiredCapacity": 1
- }
- }
-}
-```
-
-</details>
-
## Get the status for a query task
You can retrieve status of a query to see if it is still running, completed successfully, failed, or got canceled.
@@ -624,18 +138,18 @@ GET /druid/indexer/v1/task/<taskId>
```
<!--curl-->
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
```bash
+# Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
curl --location --request GET 'https://<username>:<password>@<hostname>:<port>/druid/indexer/v1/task/<taskId>/status'
```
<!--Python-->
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
```python
import requests
+# Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
url = "https://<username>:<password>@<hostname>:<port>/druid/indexer/v1/task/<taskId>/status"
payload={}
@@ -652,25 +166,25 @@ print(response.text)
```
{
- "task": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "status": {
- "id": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "groupId": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "type": "query_controller",
- "createdTime": "2022-07-27T20:09:23.551Z",
- "queueInsertionTime": "1970-01-01T00:00:00.000Z",
- "statusCode": "SUCCESS",
- "status": "SUCCESS",
- "runnerStatusCode": "WAITING",
- "duration": 136636,
- "location": {
- "host": "ip-10-201-5-81.ec2.internal",
- "port": -1,
- "tlsPort": 8100
- },
- "dataSource": "kttm_simple",
- "errorMsg": null
- }
+ "task": "query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e",
+ "status": {
+ "id": "query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e",
+ "groupId": "query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e",
+ "type": "query_controller",
+ "createdTime": "2022-09-14T22:12:00.183Z",
+ "queueInsertionTime": "1970-01-01T00:00:00.000Z",
+ "statusCode": "RUNNING",
+ "status": "RUNNING",
+ "runnerStatusCode": "RUNNING",
+ "duration": -1,
+ "location": {
+ "host": "localhost",
+ "port": 8100,
+ "tlsPort": -1
+ },
+ "dataSource": "kttm_simple",
+ "errorMsg": null
+ }
}
```
@@ -693,23 +207,22 @@ For an explanation of the fields in a report, see [Report response fields](#repo
<!--HTTP-->
```
-GET /druid/indexer/v1/task/<taskId>/report
+GET /druid/indexer/v1/task/<taskId>/reports
```
<!--curl-->
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
```bash
-curl --location --request GET 'https://<username>:<password>@<hostname>:<port>/druid/indexer/v1/task/<taskId>/report'
+# Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
+curl --location --request GET 'https://<username>:<password>@<hostname>:<port>/druid/indexer/v1/task/<taskId>/reports'
```
<!--Python-->
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
-
```python
import requests
+# Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
url = "https://<username>:<password>@<hostname>:<port>/druid/indexer/v1/task/<taskId>/reports"
payload={}
@@ -731,848 +244,300 @@ The response shows an example report for a query.
```json
{
- "multiStageQuery": {
- "taskId": "query-a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "payload": {
- "status": {
- "status": "SUCCESS",
- "startTime": "2022-07-27T20:09:39.915Z",
- "durationMs": 116516,
- "warningReports": []
- },
- "stages": [
- {
- "stageNumber": 0,
- "definition": {
- "id": "f224410f-1cad-4ee7-b10d-f10ddf8bb517_0",
- "input": [
- {
- "type": "external",
- "inputSource": {
- "type": "http",
- "uris": [
- "https://static.imply.io/data/kttm/kttm-v2-2019-08-25.json.gz"
- ],
- "httpAuthenticationUsername": null,
- "httpAuthenticationPassword": null
- },
- "inputFormat": {
- "type": "json",
- "flattenSpec": null,
- "featureSpec": {},
- "keepNullColumns": false
- },
- "signature": [
- {
- "name": "timestamp",
- "type": "STRING"
- },
- {
- "name": "agent_category",
- "type": "STRING"
- },
- {
- "name": "agent_type",
- "type": "STRING"
- },
- {
- "name": "browser",
- "type": "STRING"
- },
- {
- "name": "browser_version",
- "type": "STRING"
- },
- {
- "name": "city",
- "type": "STRING"
- },
- {
- "name": "continent",
- "type": "STRING"
- },
- {
- "name": "country",
- "type": "STRING"
- },
- {
- "name": "version",
- "type": "STRING"
- },
- {
- "name": "event_type",
- "type": "STRING"
- },
- {
- "name": "event_subtype",
- "type": "STRING"
- },
- {
- "name": "loaded_image",
- "type": "STRING"
- },
- {
- "name": "adblock_list",
- "type": "STRING"
- },
- {
- "name": "forwarded_for",
- "type": "STRING"
- },
- {
- "name": "language",
- "type": "STRING"
- },
- {
- "name": "number",
- "type": "LONG"
- },
- {
- "name": "os",
- "type": "STRING"
- },
- {
- "name": "path",
- "type": "STRING"
- },
- {
- "name": "platform",
- "type": "STRING"
- },
- {
- "name": "referrer",
- "type": "STRING"
- },
- {
- "name": "referrer_host",
- "type": "STRING"
- },
- {
- "name": "region",
- "type": "STRING"
- },
- {
- "name": "remote_address",
- "type": "STRING"
- },
- {
- "name": "screen",
- "type": "STRING"
- },
- {
- "name": "session",
- "type": "STRING"
- },
- {
- "name": "session_length",
- "type": "LONG"
- },
- {
- "name": "timezone",
- "type": "STRING"
- },
- {
- "name": "timezone_offset",
- "type": "LONG"
- },
- {
- "name": "window",
- "type": "STRING"
- }
- ]
- }
- ],
- "processor": {
- "type": "scan",
- "query": {
- "queryType": "scan",
- "dataSource": {
- "type": "inputNumber",
- "inputNumber": 0
- },
- "intervals": {
- "type": "intervals",
- "intervals": [
- "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
- ]
- },
- "resultFormat": "compactedList",
- "columns": [
- "adblock_list",
- "agent_category",
- "agent_type",
- "browser",
- "browser_version",
- "city",
- "continent",
- "country",
- "event_subtype",
- "event_type",
- "forwarded_for",
- "language",
- "loaded_image",
- "number",
- "os",
- "path",
- "platform",
- "referrer",
- "referrer_host",
- "region",
- "remote_address",
- "screen",
- "session",
- "session_length",
- "timestamp",
- "timezone",
- "timezone_offset",
- "version",
- "window"
- ],
- "legacy": false,
- "context": {
- "finalize": true,
- "msqMaxNumTasks": 3,
- "msqSignature": "[{\"name\":\"adblock_list\",\"type\":\"STRING\"},{\"name\":\"agent_category\",\"type\":\"STRING\"},{\"name\":\"agent_type\",\"type\":\"STRING\"},{\"name\":\"browser\",\"type\":\"STRING\"},{\"name\":\"browser_version\",\"type\":\"STRING\"},{\"name\":\"city\",\"type\":\"STRING\"},{\"name\":\"continent\",\"type\":\"STRING\"},{\"name\":\"country\",\"type\":\"STRING\"},{\"name\":\"event_subtype\",\"type\":\"STRING\"},{\"name\":\"event_type\ [...]
- "multiStageQuery": true,
- "sqlInsertSegmentGranularity": "{\"type\":\"all\"}",
- "sqlQueryId": "a6b65442-f77e-44e4-af28-ab3b711a27ac",
- "sqlReplaceTimeChunks": "all"
- },
- "granularity": {
- "type": "all"
- }
- }
- },
- "signature": [
- {
- "name": "__boost",
- "type": "LONG"
- },
- {
- "name": "adblock_list",
- "type": "STRING"
- },
- {
- "name": "agent_category",
- "type": "STRING"
- },
- {
- "name": "agent_type",
- "type": "STRING"
- },
- {
- "name": "browser",
- "type": "STRING"
- },
- {
- "name": "browser_version",
- "type": "STRING"
- },
- {
- "name": "city",
- "type": "STRING"
- },
- {
- "name": "continent",
- "type": "STRING"
- },
- {
- "name": "country",
- "type": "STRING"
- },
- {
- "name": "event_subtype",
- "type": "STRING"
- },
- {
- "name": "event_type",
- "type": "STRING"
- },
- {
- "name": "forwarded_for",
- "type": "STRING"
- },
- {
- "name": "language",
- "type": "STRING"
- },
- {
- "name": "loaded_image",
- "type": "STRING"
- },
- {
- "name": "number",
- "type": "LONG"
- },
- {
- "name": "os",
- "type": "STRING"
- },
- {
- "name": "path",
- "type": "STRING"
- },
- {
- "name": "platform",
- "type": "STRING"
- },
- {
- "name": "referrer",
- "type": "STRING"
- },
- {
- "name": "referrer_host",
- "type": "STRING"
- },
- {
- "name": "region",
- "type": "STRING"
- },
- {
- "name": "remote_address",
- "type": "STRING"
- },
- {
- "name": "screen",
- "type": "STRING"
- },
- {
- "name": "session",
- "type": "STRING"
- },
- {
- "name": "session_length",
- "type": "LONG"
- },
- {
- "name": "timestamp",
- "type": "STRING"
- },
- {
- "name": "timezone",
- "type": "STRING"
- },
- {
- "name": "timezone_offset",
- "type": "LONG"
- },
- {
- "name": "version",
- "type": "STRING"
- },
- {
- "name": "window",
- "type": "STRING"
- }
- ],
- "shuffleSpec": {
- "type": "targetSize",
- "clusterBy": {
- "columns": [
- {
- "columnName": "__boost"
- }
- ]
- },
- "targetSize": 3000000,
- "aggregate": false
- },
- "maxWorkerCount": 2,
- "shuffleCheckHasMultipleValues": true
- },
- "phase": "FINISHED",
- "workerCount": 1,
- "partitionCount": 1,
- "startTime": "2022-07-27T20:09:43.168Z",
- "duration": 62837,
- "sort": true
+ "multiStageQuery": {
+ "type": "multiStageQuery",
+ "taskId": "query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e",
+ "payload": {
+ "status": {
+ "status": "SUCCESS",
+ "startTime": "2022-09-14T22:12:09.266Z",
+ "durationMs": 28227
+ },
+ "stages": [
+ {
+ "stageNumber": 0,
+ "definition": {
+ "id": "71ecb11e-09d7-42f8-9225-1662c8e7e121_0",
+ "input": [
+ {
+ "type": "external",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/example-data/kttm-v2/kttm-v2-2019-08-25.json.gz"
+ ],
+ "httpAuthenticationUsername": null,
+ "httpAuthenticationPassword": null
+ },
+ "inputFormat": {
+ "type": "json",
+ "flattenSpec": null,
+ "featureSpec": {},
+ "keepNullColumns": false
+ },
+ "signature": [
+ {
+ "name": "timestamp",
+ "type": "STRING"
+ },
+ {
+ "name": "agent_category",
+ "type": "STRING"
+ },
+ {
+ "name": "agent_type",
+ "type": "STRING"
+ }
+ ]
+ }
+ ],
+ "processor": {
+ "type": "scan",
+ "query": {
+ "queryType": "scan",
+ "dataSource": {
+ "type": "inputNumber",
+ "inputNumber": 0
},
- {
- "stageNumber": 1,
- "definition": {
- "id": "f224410f-1cad-4ee7-b10d-f10ddf8bb517_1",
- "input": [
- {
- "type": "stage",
- "stage": 0
- }
- ],
- "processor": {
- "type": "segmentGenerator",
- "dataSchema": {
- "dataSource": "kttm_simple",
- "timestampSpec": {
- "column": "__time",
- "format": "millis",
- "missingValue": null
- },
- "dimensionsSpec": {
- "dimensions": [
- {
- "type": "string",
- "name": "timestamp",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "agent_category",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "agent_type",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "browser",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "browser_version",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "city",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "continent",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "country",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "version",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "event_type",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "event_subtype",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "loaded_image",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "adblock_list",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "forwarded_for",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "language",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "long",
- "name": "number",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": false
- },
- {
- "type": "string",
- "name": "os",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "path",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "platform",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "referrer",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "referrer_host",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "region",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "remote_address",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "screen",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "string",
- "name": "session",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "long",
- "name": "session_length",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": false
- },
- {
- "type": "string",
- "name": "timezone",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- },
- {
- "type": "long",
- "name": "timezone_offset",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": false
- },
- {
- "type": "string",
- "name": "window",
- "multiValueHandling": "SORTED_ARRAY",
- "createBitmapIndex": true
- }
- ],
- "dimensionExclusions": [
- "__time"
- ],
- "includeAllDimensions": false
- },
- "metricsSpec": [],
- "granularitySpec": {
- "type": "arbitrary",
- "queryGranularity": {
- "type": "none"
- },
- "rollup": false,
- "intervals": [
- "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
- ]
- },
- "transformSpec": {
- "filter": null,
- "transforms": []
- }
- },
- "columnMappings": [
- {
- "queryColumn": "timestamp",
- "outputColumn": "timestamp"
- },
- {
- "queryColumn": "agent_category",
- "outputColumn": "agent_category"
- },
- {
- "queryColumn": "agent_type",
- "outputColumn": "agent_type"
- },
- {
- "queryColumn": "browser",
- "outputColumn": "browser"
- },
- {
- "queryColumn": "browser_version",
- "outputColumn": "browser_version"
- },
- {
- "queryColumn": "city",
- "outputColumn": "city"
- },
- {
- "queryColumn": "continent",
- "outputColumn": "continent"
- },
- {
- "queryColumn": "country",
- "outputColumn": "country"
- },
- {
- "queryColumn": "version",
- "outputColumn": "version"
- },
- {
- "queryColumn": "event_type",
- "outputColumn": "event_type"
- },
- {
- "queryColumn": "event_subtype",
- "outputColumn": "event_subtype"
- },
- {
- "queryColumn": "loaded_image",
- "outputColumn": "loaded_image"
- },
- {
- "queryColumn": "adblock_list",
- "outputColumn": "adblock_list"
- },
- {
- "queryColumn": "forwarded_for",
- "outputColumn": "forwarded_for"
- },
- {
- "queryColumn": "language",
- "outputColumn": "language"
- },
- {
- "queryColumn": "number",
- "outputColumn": "number"
- },
- {
- "queryColumn": "os",
- "outputColumn": "os"
- },
- {
- "queryColumn": "path",
- "outputColumn": "path"
- },
- {
- "queryColumn": "platform",
- "outputColumn": "platform"
- },
- {
- "queryColumn": "referrer",
- "outputColumn": "referrer"
- },
- {
- "queryColumn": "referrer_host",
- "outputColumn": "referrer_host"
- },
- {
- "queryColumn": "region",
- "outputColumn": "region"
- },
- {
- "queryColumn": "remote_address",
- "outputColumn": "remote_address"
- },
- {
- "queryColumn": "screen",
- "outputColumn": "screen"
- },
- {
- "queryColumn": "session",
- "outputColumn": "session"
- },
- {
- "queryColumn": "session_length",
- "outputColumn": "session_length"
- },
- {
- "queryColumn": "timezone",
- "outputColumn": "timezone"
- },
- {
- "queryColumn": "timezone_offset",
- "outputColumn": "timezone_offset"
- },
- {
- "queryColumn": "window",
- "outputColumn": "window"
- }
- ],
- "tuningConfig": {
- "type": "index_parallel",
- "maxRowsPerSegment": 3000000,
- "appendableIndexSpec": {
- "type": "onheap",
- "preserveExistingMetrics": false
- },
- "maxRowsInMemory": 100000,
- "maxBytesInMemory": 0,
- "skipBytesInMemoryOverheadCheck": false,
- "maxTotalRows": null,
- "numShards": null,
- "splitHintSpec": null,
- "partitionsSpec": {
- "type": "dynamic",
- "maxRowsPerSegment": 3000000,
- "maxTotalRows": null
- },
- "indexSpec": {
- "bitmap": {
- "type": "roaring",
- "compressRunOnSerialization": true
- },
- "dimensionCompression": "lz4",
- "metricCompression": "lz4",
- "longEncoding": "longs",
- "segmentLoader": null
- },
- "indexSpecForIntermediatePersists": {
- "bitmap": {
- "type": "roaring",
- "compressRunOnSerialization": true
- },
- "dimensionCompression": "lz4",
- "metricCompression": "lz4",
- "longEncoding": "longs",
- "segmentLoader": null
- },
- "maxPendingPersists": 0,
- "forceGuaranteedRollup": false,
- "reportParseExceptions": false,
- "pushTimeout": 0,
- "segmentWriteOutMediumFactory": null,
- "maxNumConcurrentSubTasks": 2,
- "maxRetry": 1,
- "taskStatusCheckPeriodMs": 1000,
- "chatHandlerTimeout": "PT10S",
- "chatHandlerNumRetries": 5,
- "maxNumSegmentsToMerge": 100,
- "totalNumMergeTasks": 10,
- "logParseExceptions": false,
- "maxParseExceptions": 2147483647,
- "maxSavedParseExceptions": 0,
- "maxColumnsToMerge": -1,
- "awaitSegmentAvailabilityTimeoutMillis": 0,
- "maxAllowedLockCount": -1,
- "partitionDimensions": []
- }
- },
- "signature": [],
- "maxWorkerCount": 2
- },
- "phase": "FINISHED",
- "workerCount": 1,
- "partitionCount": 1,
- "startTime": "2022-07-27T20:10:45.840Z",
- "duration": 50590
+ "intervals": {
+ "type": "intervals",
+ "intervals": [
+ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
+ ]
+ },
+ "resultFormat": "compactedList",
+ "columns": [
+ "agent_category",
+ "agent_type",
+ "timestamp"
+ ],
+ "legacy": false,
+ "context": {
+ "finalize": false,
+ "finalizeAggregations": false,
+ "groupByEnableMultiValueUnnesting": false,
+ "scanSignature": "[{\"name\":\"agent_category\",\"type\":\"STRING\"},{\"name\":\"agent_type\",\"type\":\"STRING\"},{\"name\":\"timestamp\",\"type\":\"STRING\"}]",
+ "sqlInsertSegmentGranularity": "{\"type\":\"all\"}",
+ "sqlQueryId": "3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e",
+ "sqlReplaceTimeChunks": "all"
+ },
+ "granularity": {
+ "type": "all"
}
+ }
+ },
+ "signature": [
+ {
+ "name": "__boost",
+ "type": "LONG"
+ },
+ {
+ "name": "agent_category",
+ "type": "STRING"
+ },
+ {
+ "name": "agent_type",
+ "type": "STRING"
+ },
+ {
+ "name": "timestamp",
+ "type": "STRING"
+ }
+ ],
+ "shuffleSpec": {
+ "type": "targetSize",
+ "clusterBy": {
+ "columns": [
+ {
+ "columnName": "__boost"
+ }
+ ]
+ },
+ "targetSize": 3000000
+ },
+ "maxWorkerCount": 1,
+ "shuffleCheckHasMultipleValues": true
+ },
+ "phase": "FINISHED",
+ "workerCount": 1,
+ "partitionCount": 1,
+ "startTime": "2022-09-14T22:12:11.663Z",
+ "duration": 19965,
+ "sort": true
+ },
+ {
+ "stageNumber": 1,
+ "definition": {
+ "id": "71ecb11e-09d7-42f8-9225-1662c8e7e121_1",
+ "input": [
+ {
+ "type": "stage",
+ "stage": 0
+ }
],
- "counters": {
- "0": {
- "0": {
- "input0": {
- "type": "channel",
- "rows": [
- 465346
- ],
- "files": [
- 1
- ],
- "totalFiles": [
- 1
- ]
- },
- "output": {
- "type": "channel",
- "rows": [
- 465346
- ],
- "bytes": [
- 267146161
- ],
- "frames": [
- 42
- ]
- },
- "sort": {
- "type": "channel",
- "rows": [
- 465346
- ],
- "bytes": [
- 265300383
- ],
- "frames": [
- 501
- ]
- },
- "sortProgress": {
- "type": "sortProgress",
- "totalMergingLevels": 3,
- "levelToTotalBatches": {
- "0": 9,
- "1": 2,
- "2": 1
- },
- "levelToMergedBatches": {
- "0": 9,
- "1": 2,
- "2": 1
- },
- "totalMergersForUltimateLevel": 1,
- "progressDigest": 1.0
- }
+ "processor": {
+ "type": "segmentGenerator",
+ "dataSchema": {
+ "dataSource": "kttm_simple",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "millis",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "timestamp",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "agent_category",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "agent_type",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
}
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "arbitrary",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": [
+ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
+ ]
},
- "1": {
- "0": {
- "input0": {
- "type": "channel",
- "rows": [
- 465346
- ],
- "bytes": [
- 265300383
- ],
- "frames": [
- 501
- ]
- }
- }
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "columnMappings": [
+ {
+ "queryColumn": "timestamp",
+ "outputColumn": "timestamp"
+ },
+ {
+ "queryColumn": "agent_category",
+ "outputColumn": "agent_category"
+ },
+ {
+ "queryColumn": "agent_type",
+ "outputColumn": "agent_type"
}
+ ],
+ "tuningConfig": {
+ "maxNumWorkers": 1,
+ "maxRowsInMemory": 100000,
+ "rowsPerSegment": 3000000
+ }
+ },
+ "signature": [],
+ "maxWorkerCount": 1
+ },
+ "phase": "FINISHED",
+ "workerCount": 1,
+ "partitionCount": 1,
+ "startTime": "2022-09-14T22:12:31.602Z",
+ "duration": 5891
+ }
+ ],
+ "counters": {
+ "0": {
+ "0": {
+ "input0": {
+ "type": "channel",
+ "rows": [
+ 465346
+ ],
+ "files": [
+ 1
+ ],
+ "totalFiles": [
+ 1
+ ]
+ },
+ "output": {
+ "type": "channel",
+ "rows": [
+ 465346
+ ],
+ "bytes": [
+ 43694447
+ ],
+ "frames": [
+ 7
+ ]
+ },
+ "shuffle": {
+ "type": "channel",
+ "rows": [
+ 465346
+ ],
+ "bytes": [
+ 41835307
+ ],
+ "frames": [
+ 73
+ ]
+ },
+ "sortProgress": {
+ "type": "sortProgress",
+ "totalMergingLevels": 3,
+ "levelToTotalBatches": {
+ "0": 1,
+ "1": 1,
+ "2": 1
+ },
+ "levelToMergedBatches": {
+ "0": 1,
+ "1": 1,
+ "2": 1
+ },
+ "totalMergersForUltimateLevel": 1,
+ "progressDigest": 1
+ }
+ }
+ },
+ "1": {
+ "0": {
+ "input0": {
+ "type": "channel",
+ "rows": [
+ 465346
+ ],
+ "bytes": [
+ 41835307
+ ],
+ "frames": [
+ 73
+ ]
}
+ }
}
+ }
}
+ }
}
```
### Report response fields
-The following table describes the response fields when you retrieve a report for a MSQ task engine using the `/druid/indexer/v1/task/<taskId>/report` endpoint:
+The following table describes the response fields when you retrieve a report for a MSQ task engine using the `/druid/indexer/v1/task/<taskId>/reports` endpoint:
|Field|Description|
|-----|-----------|
@@ -1618,19 +583,17 @@ POST /druid/indexer/v1/task/<taskId>/shutdown
<!--curl-->
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
-
```bash
+# Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
curl --location --request POST 'https://<username>:<password>@<your-instance>:<port>/druid/indexer/v1/task/<taskId>/shutdown'
```
<!--Python-->
-Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
-
```
import requests
+# Make sure you replace `username`, `password`, `your-instance`, `port`, and `taskId` with the values for your deployment.
url = "https://<username>:<password>@<your-instance>:<port>/druid/indexer/v1/task/<taskId>/shutdown"
payload={}
diff --git a/docs/multi-stage-query/msq-concepts.md b/docs/multi-stage-query/msq-concepts.md
index 4f1cf7ce1d..a7d9cf893a 100644
--- a/docs/multi-stage-query/msq-concepts.md
+++ b/docs/multi-stage-query/msq-concepts.md
@@ -115,7 +115,7 @@ memory available (`-XX:MaxDirectMemorySize`) to at least
`(druid.processing.numThreads + 1) * druid.processing.buffer.sizeBytes`. Increasing the
amount of direct memory available beyond the minimum does not speed up processing.
-It may be necessary to override one or more memory-related parameters if you run into one of the [known issues around memory usage](./msq-known-issues.md#memory-usage).
+It may be necessary to override one or more memory-related parameters if you run into one of the [known issues](./msq-known-issues.md) around memory usage.
## Limits
diff --git a/docs/multi-stage-query/msq-example-queries.md b/docs/multi-stage-query/msq-example-queries.md
index f1b09e332f..d810d536cc 100644
--- a/docs/multi-stage-query/msq-example-queries.md
+++ b/docs/multi-stage-query/msq-example-queries.md
@@ -34,9 +34,6 @@ This example inserts data into a table named `w000` without performing any data
<details><summary>Show the query</summary>
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
INSERT INTO w000
SELECT
TIME_PARSE("timestamp") AS __time,
@@ -65,7 +62,7 @@ SELECT
regionName
FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type":"http","uris":["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type":"json"}',
'[{"name":"isRobot","type":"string"},{"name":"channel","type":"string"},{"name":"timestamp","type":"string"},{"name":"flags","type":"string"},{"name":"isUnpatrolled","type":"string"},{"name":"page","type":"string"},{"name":"diffUrl","type":"string"},{"name":"added","type":"long"},{"name":"comment","type":"string"},{"name":"commentLength","type":"long"},{"name":"isNew","type":"string"},{"name":"isMinor","type":"string"},{"name":"delta","type":"long"},{"name":"isAnonymous","type":"st [...]
)
@@ -83,15 +80,12 @@ This example inserts data into a table named `kttm_data` and performs data rollu
<details><summary>Show the query</summary>
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
INSERT INTO "kttm_rollup"
WITH kttm_data AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/kttm/kttm-v2-2019-08-25.json.gz"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/kttm-v2/kttm-v2-2019-08-25.json.gz"]}',
'{"type":"json"}',
'[{"name":"timestamp","type":"string"},{"name":"agent_category","type":"string"},{"name":"agent_type","type":"string"},{"name":"browser","type":"string"},{"name":"browser_version","type":"string"},{"name":"city","type":"string"},{"name":"continent","type":"string"},{"name":"country","type":"string"},{"name":"version","type":"string"},{"name":"event_type","type":"string"},{"name":"event_subtype","type":"string"},{"name":"loaded_image","type":"string"},{"name":"adblock_list","type":"st [...]
)
@@ -129,9 +123,6 @@ This example aggregates data from a table named `w000` and inserts the result in
<details><summary>Show the query</summary>
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
INSERT INTO w002
SELECT
FLOOR(__time TO MINUTE) AS __time,
@@ -160,21 +151,18 @@ This example inserts data into a table named `w003` and joins data from two sour
<details><summary>Show the query</summary>
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
INSERT INTO w003
WITH
wikidata AS (SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type":"http","uris":["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type":"json"}',
'[{"name":"isRobot","type":"string"},{"name":"channel","type":"string"},{"name":"timestamp","type":"string"},{"name":"flags","type":"string"},{"name":"isUnpatrolled","type":"string"},{"name":"page","type":"string"},{"name":"diffUrl","type":"string"},{"name":"added","type":"long"},{"name":"comment","type":"string"},{"name":"commentLength","type":"long"},{"name":"isNew","type":"string"},{"name":"isMinor","type":"string"},{"name":"delta","type":"long"},{"name":"isAnonymous","type":"stri [...]
)
)),
countries AS (SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/lookup/country.tsv"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/lookup/countries.tsv"]}',
'{"type":"tsv","findColumnsFromHeader":true}',
'[{"name":"Country","type":"string"},{"name":"Capital","type":"string"},{"name":"ISO3","type":"string"},{"name":"ISO2","type":"string"}]'
)
@@ -219,9 +207,6 @@ This example replaces the entire datasource used in the table `w007` with the ne
<details><summary>Show the query</summary>
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
REPLACE INTO w007
OVERWRITE ALL
SELECT
@@ -251,7 +236,7 @@ SELECT
regionName
FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type":"http","uris":["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type":"json"}',
'[{"name":"isRobot","type":"string"},{"name":"channel","type":"string"},{"name":"timestamp","type":"string"},{"name":"flags","type":"string"},{"name":"isUnpatrolled","type":"string"},{"name":"page","type":"string"},{"name":"diffUrl","type":"string"},{"name":"added","type":"long"},{"name":"comment","type":"string"},{"name":"commentLength","type":"long"},{"name":"isNew","type":"string"},{"name":"isMinor","type":"string"},{"name":"delta","type":"long"},{"name":"isAnonymous","type":"st [...]
)
@@ -269,9 +254,6 @@ This example replaces certain segments in a datasource with the new query data w
<details><summary>Show the query</summary>
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
REPLACE INTO w007
OVERWRITE WHERE __time >= TIMESTAMP '2019-08-25 02:00:00' AND __time < TIMESTAMP '2019-08-25 03:00:00'
SELECT
@@ -295,9 +277,6 @@ CLUSTERED BY page
<details><summary>Show the query</summary>
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
REPLACE INTO w000
OVERWRITE ALL
SELECT
@@ -326,13 +305,10 @@ CLUSTERED BY page
```sql
---:context finalizeAggregations: false
---:context groupByEnableMultiValueUnnesting: false
-
WITH flights AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/FlightCarrierOnTime/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip"]}',
'{"type":"csv","findColumnsFromHeader":true}',
'[{"name":"depaturetime","type":"string"},{"name":"arrivalime","type":"string"},{"name":"Year","type":"long"},{"name":"Quarter","type":"long"},{"name":"Month","type":"long"},{"name":"DayofMonth","type":"long"},{"name":"DayOfWeek","type":"long"},{"name":"FlightDate","type":"string"},{"name":"Reporting_Airline","type":"string"},{"name":"DOT_ID_Reporting_Airline","type":"long"},{"name":"IATA_CODE_Reporting_Airline","type":"string"},{"name":"Tail_Number","type":"string"},{"name":"Flight_ [...]
)
@@ -340,7 +316,7 @@ WITH flights AS (
L_AIRPORT AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/FlightCarrierOnTime/dimensions/L_AIRPORT.csv"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/flight_on_time/dimensions/L_AIRPORT.csv"]}',
'{"type":"csv","findColumnsFromHeader":true}',
'[{"name":"Code","type":"string"},{"name":"Description","type":"string"}]'
)
@@ -348,7 +324,7 @@ L_AIRPORT AS (
L_AIRPORT_ID AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/FlightCarrierOnTime/dimensions/L_AIRPORT_ID.csv"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/flight_on_time/dimensions/L_AIRPORT_ID.csv"]}',
'{"type":"csv","findColumnsFromHeader":true}',
'[{"name":"Code","type":"long"},{"name":"Description","type":"string"}]'
)
@@ -356,7 +332,7 @@ L_AIRPORT_ID AS (
L_AIRLINE_ID AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/FlightCarrierOnTime/dimensions/L_AIRLINE_ID.csv"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/flight_on_time/dimensions/L_AIRLINE_ID.csv"]}',
'{"type":"csv","findColumnsFromHeader":true}',
'[{"name":"Code","type":"long"},{"name":"Description","type":"string"}]'
)
@@ -364,7 +340,7 @@ L_AIRLINE_ID AS (
L_CITY_MARKET_ID AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/FlightCarrierOnTime/dimensions/L_CITY_MARKET_ID.csv"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/flight_on_time/dimensions/L_CITY_MARKET_ID.csv"]}',
'{"type":"csv","findColumnsFromHeader":true}',
'[{"name":"Code","type":"long"},{"name":"Description","type":"string"}]'
)
@@ -372,7 +348,7 @@ L_CITY_MARKET_ID AS (
L_CANCELLATION AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/FlightCarrierOnTime/dimensions/L_CANCELLATION.csv"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/flight_on_time/dimensions/L_CANCELLATION.csv"]}',
'{"type":"csv","findColumnsFromHeader":true}',
'[{"name":"Code","type":"string"},{"name":"Description","type":"string"}]'
)
@@ -380,7 +356,7 @@ L_CANCELLATION AS (
L_STATE_FIPS AS (
SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/FlightCarrierOnTime/dimensions/L_STATE_FIPS.csv"]}',
+ '{"type":"http","uris":["https://static.imply.io/example-data/flight_on_time/dimensions/L_STATE_FIPS.csv"]}',
'{"type":"csv","findColumnsFromHeader":true}',
'[{"name":"Code","type":"long"},{"name":"Description","type":"string"}]'
)
diff --git a/docs/multi-stage-query/msq-known-issues.md b/docs/multi-stage-query/msq-known-issues.md
index ac4b222fe6..b8b5747ac1 100644
--- a/docs/multi-stage-query/msq-known-issues.md
+++ b/docs/multi-stage-query/msq-known-issues.md
@@ -23,96 +23,42 @@ sidebar_label: Known issues
~ under the License.
-->
-> SQL-based ingestion using the multi-stage query task engine is our recommended solution starting in Druid 24.0. Alternative ingestion solutions, such as native batch and Hadoop-based ingestion systems, will still be supported. We recommend you read all [known issues](./msq-known-issues.md) and test the feature in a development environment before rolling it out in production. Using the multi-stage query task engine with `SELECT` statements that do not write to a datasource is experimental.
+> SQL-based ingestion using the multi-stage query task engine is our recommended solution starting in Druid 24.0.
+> Alternative ingestion solutions, such as native batch and Hadoop-based ingestion systems, will still be supported.
+> We recommend you read all [known issues](./msq-known-issues.md) and test the feature in a development environment
+> before rolling it out in production. Using the multi-stage query task engine with `SELECT` statements that do not
+> write to a datasource is experimental.
-## General query execution
+## Multi-stage query task runtime
-- There's no fault tolerance. If any task fails, the entire query fails.
+- Fault tolerance is not implemented. If any task fails, the entire query fails.
-- Only one local file system per server is used for stage output data during multi-stage query
- execution. If your servers have multiple local file systems, this causes queries to exhaust
- available disk space earlier than expected.
+- SELECT from a Druid datasource does not include unpublished real-time data.
-- When `msqMaxNumTasks` is higher than the total
- capacity of the cluster, more tasks may be launched than can run at once. This leads to a
- [TaskStartTimeout](./msq-reference.md#context-parameters) error code, as there is never enough capacity to run the query.
- To avoid this, set `msqMaxNumTasks` to a number of tasks that can run simultaneously on your cluster.
+- GROUPING SETS is not implemented. Queries that use GROUPING SETS fail.
-- When `msqTaskAssignment` is set to `auto`, the system generates one task per input file for certain splittable
- input sources where file sizes are not known ahead of time. This includes the `http` input source, where the system
- generates one task per URI.
+- Worker task stage outputs are stored in the working directory given by `druid.indexer.task.baseDir`. Stages that
+generate a large amount of output data may exhaust all available disk space. In this case, the query fails with
+an [UnknownError](./msq-reference.md#error-codes) with a message including "No space left on device".
-## Memory usage
+- The numeric varieties of the EARLIEST and LATEST aggregators do not work properly. Attempting to use the numeric
+ varieties of these aggregators lead to an error like
+ `java.lang.ClassCastException: class java.lang.Double cannot be cast to class org.apache.druid.collections.SerializablePair`.
+ The string varieties, however, do work properly.
-- INSERT queries can consume excessive memory when using complex types due to inaccurate footprint
- estimation. This can appear as an OutOfMemoryError during the SegmentGenerator stage when using
- sketches. If you run into this issue, try manually lowering the value of the
- [`msqRowsInMemory`](./msq-reference.md#context-parameters) parameter.
+## INSERT and REPLACE
-- EXTERN loads an entire row group into memory at once when reading from Parquet files. Row groups
- can be up to 1 GB in size, which can lead to excessive heap usage when reading many files in
- parallel. This can appear as an OutOfMemoryError during stages that read Parquet input files. If
- you run into this issue, try using a smaller number of worker tasks or you can increase the heap
- size of your Indexers or of your Middle Manager-launched indexing tasks.
+- INSERT with column lists, like `INSERT INTO tbl (a, b, c) SELECT ...`, is not implemented.
-- Ingesting a very long row may consume excessive memory and result in an OutOfMemoryError. If a row is read
- which requires more memory than is available, the service might throw OutOfMemoryError. If you run into this
- issue, allocate enough memory to be able to store the largest row to the indexer.
+- `INSERT ... SELECT` inserts columns from the SELECT statement based on column name. This differs from SQL standard
+behavior, where columns are inserted based on position.
-## SELECT queries
-
-- SELECT query results do not include real-time data until it has been published.
-
-- TIMESTAMP types are formatted as numbers rather than ISO8601 timestamp
- strings, which differs from Druid's standard result format.
-
-- BOOLEAN types are formatted as numbers like `1` and `0` rather
- than `true` or `false`, which differs from Druid's standard result
- format.
-
-- TopN is not implemented. The context parameter
- `useApproximateTopN` is ignored and always treated as if it
- were `false`. Therefore, topN-shaped queries will
- always run using the groupBy engine. There is no loss of
- functionality, but there may be a performance impact, since
- these queries will run using an exact algorithm instead of an
- approximate one.
-- GROUPING SETS is not implemented. Queries that use GROUPING SETS
- will fail.
-- The numeric flavors of the EARLIEST and LATEST aggregators do not work properly. Attempting to use the numeric flavors of these aggregators will lead to an error like `java.lang.ClassCastException: class java.lang.Double cannot be cast to class org.apache.druid.collections.SerializablePair`. The string flavors, however, do work properly.
-
-## INSERT queries
+## EXTERN
- The [schemaless dimensions](../ingestion/ingestion-spec.md#inclusions-and-exclusions)
-feature is not available. All columns and their types must be specified explicitly.
-
-- [Segment metadata queries](../querying/segmentmetadataquery.md)
- on datasources ingested with the Multi-Stage Query Engine will return values for`timestampSpec` that are not usable
- for introspection.
-
-- When INSERT with GROUP BY does the match the criteria mentioned in [GROUP BY](./index.md#group-by), the multi-stage engine generates segments that Druid's compaction
- functionality is not able to further roll up. This applies to automatic compaction as well as manually
- issued `compact` tasks. Individual queries executed with the multi-stage engine always guarantee
- perfect rollup for their output, so this only matters if you are performing a sequence of INSERT
- queries that each append data to the same time chunk. If necessary, you can compact such data
- using another SQL query instead of a `compact` task.
-
-- When using INSERT with GROUP BY, splitting of large partitions is not currently
- implemented. If a single partition key appears in a
- very large number of rows, an oversized segment will be created.
- You can mitigate this by adding additional columns to your
- partition key. Note that partition splitting _does_ work properly
- when performing INSERT without GROUP BY.
-
-- INSERT with column lists, like
- `INSERT INTO tbl (a, b, c) SELECT ...`, is not implemented.
-
-## EXTERN queries
-
-- EXTERN does not accept `druid` input sources.
-
-## Missing guardrails
+ feature is not available. All columns and their types must be specified explicitly using the `signature` parameter
+ of the [EXTERN function](msq-reference.md#extern).
-- Maximum number of input files. Since there's no limit, the controller can potentially run out of memory tracking all input files
+- EXTERN with input sources that match large numbers of files may exhaust available memory on the controller task.
-- Maximum amount of local disk space to use for temporary data. No guardrail today means worker tasks may exhaust all available disk space. In this case, you will receive an [UnknownError](./msq-reference.md#error-codes)) with a message including "No space left on device".
\ No newline at end of file
+- EXTERN does not accept `druid` input sources. Use FROM instead.
diff --git a/docs/multi-stage-query/msq-reference.md b/docs/multi-stage-query/msq-reference.md
index f5ef07124c..10af042271 100644
--- a/docs/multi-stage-query/msq-reference.md
+++ b/docs/multi-stage-query/msq-reference.md
@@ -50,9 +50,9 @@ The following table lists the context parameters for the MSQ task engine:
|Parameter|Description|Default value|
|---------|-----------|-------------|
| maxNumTasks | SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.<br /><br />May also be provided as `numTasks`. If both are present, `maxNumTasks` takes priority.| 2 |
-| taskAssignment | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Use as many tasks as possible, up to the maximum `maxNumTasks`.</li><li>`auto`: Use as few tasks as possible without exceeding 10 GiB or 10,000 files per task. Review the [limitations](./msq-known-issues.md#general-query-execution) of `auto` mode before using it.</li></ui>| `max` |
+| taskAssignment | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 10 GiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When file sizes cannot be determined th [...]
| finalizeAggregations | SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggreg [...]
-| rowsInMemory | INSERT or REPLACE<br /><br />Maximum number of rows to store in memory at once before flushing to disk during the segment generation process. Ignored for non-INSERT queries. In most cases, use the default value. You may need to override the default if you run into one of the [known issues around memory usage](./msq-known-issues.md#memory-usage)</a>. | 100,000 |
+| rowsInMemory | INSERT or REPLACE<br /><br />Maximum number of rows to store in memory at once before flushing to disk during the segment generation process. Ignored for non-INSERT queries. In most cases, use the default value. You may need to override the default if you run into one of the [known issues](./msq-known-issues.md) around memory usage. | 100,000 |
| segmentSortOrder | INSERT or REPLACE<br /><br />Normally, Druid sorts rows in individual segments using `__time` first, followed by the [CLUSTERED BY](./index.md#clustered-by) clause. When you set `segmentSortOrder`, Druid sorts rows in segments using this column list first, followed by the CLUSTERED BY order.<br /><br />You provide the column list as comma-separated values or as a JSON array in string form. If your query includes `__time`, then this list must begin with `__time`. For [...]
| maxParseExceptions| SELECT, INSERT, REPLACE<br /><br />Maximum number of parse exceptions that are ignored while executing the query before it stops with `TooManyWarningsFault`. To ignore all the parse exceptions, set the value to -1.| 0 |
| rowsPerSegment | INSERT or REPLACE<br /><br />The number of rows per segment to target. The actual number of rows per segment may be somewhat higher or lower than this number. In most cases, use the default. For general information about sizing rows per segment, see [Segment Size Optimization](../operations/segment-optimization.md). | 3,000,000 |
@@ -166,4 +166,4 @@ REPLACE consists of the following parts:
- OVERWRITE WHERE drops the time segments that match the condition you set. Conditions are based on the `__time` column and use the format `__time [< > = <= >=] TIMESTAMP`. Use them with AND, OR, and NOT between them, inclusive of the timestamps specified. For example, see [REPLACE INTO ... OVERWRITE WHERE ... SELECT](./index.md#replace-some-data).
4. A clause for the actual data you want to use for the replacement.
5. A [PARTITIONED BY](./index.md#partitioned-by) clause to your REPLACE statement. For example, use PARTITIONED BY DAY for daily partitioning, or PARTITIONED BY ALL TIME to skip time partitioning completely.
-6. An optional [CLUSTERED BY](./index.md#clustered-by) clause.
+6. An optional [CLUSTERED BY](./index.md#clustered-by) clause.
\ No newline at end of file
diff --git a/docs/multi-stage-query/msq-tutorial-connect-external-data.md b/docs/multi-stage-query/msq-tutorial-connect-external-data.md
index 26f2c8f62a..e442bbbdf6 100644
--- a/docs/multi-stage-query/msq-tutorial-connect-external-data.md
+++ b/docs/multi-stage-query/msq-tutorial-connect-external-data.md
@@ -1,6 +1,6 @@
---
id: connect-external-data
-title: Tutorial - Connect external data for SQL-based ingestion
+title: Tutorial - Load files with SQL-based ingestion
description: How to generate a query that references externally hosted data
---
@@ -27,22 +27,19 @@ description: How to generate a query that references externally hosted data
This tutorial demonstrates how to generate a query that references externally hosted data using the **Connect external data** wizard.
-The following example uses EXTERN to query a JSON file located at https://static.imply.io/data/wikipedia.json.gz.
+The following example uses EXTERN to query a JSON file located at https://druid.apache.org/data/wikipedia.json.gz.
Although you can manually create a query in the UI, you can use Druid to generate a base query for you that you can modify to meet your requirements.
To generate a query from external data, do the following:
1. In the **Query** view of the Druid console, click **Connect external data**.
-2. On the **Select input type** screen, choose **HTTP(s)** and enter the following value in the **URIs** field: `https://static.imply.io/data/wikipedia.json.gz`. Leave the HTTP auth username and password blank.
+2. On the **Select input type** screen, choose **HTTP(s)** and enter the following value in the **URIs** field: `https://druid.apache.org/data/wikipedia.json.gz`. Leave the HTTP auth username and password blank.
3. Click **Connect data**.
4. On the **Parse** screen, you can perform additional actions before you load the data into Druid:
- Expand a row to see what data it corresponds to from the source.
- Customize how Druid handles the data by selecting the **Input format** and its related options, such as adding **JSON parser features** for JSON files.
-5. When you're ready, click **Done**. You're returned to the **Query** view where you can see the newly generated query:
-
- - The query inserts the data from the external source into a table named `wikipedia`.
- - Context parameters appear before the query in the syntax unique to the Druid console: `--: context {key}: {value}`. When submitting queries to Druid directly, set the `context` parameters in the context section of the SQL query object. For more information about context parameters, see [Context parameters](./msq-reference.md#context-parameters).
+5. When you're ready, click **Done**. You're returned to the **Query** view where you can see the starter query that will insert the data from the external source into a table named `wikipedia`.
<details><summary>Show the query</summary>
@@ -51,7 +48,7 @@ To generate a query from external data, do the following:
WITH ext AS (SELECT *
FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type":"http","uris":["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type":"json"}',
'[{"name":"isRobot","type":"string"},{"name":"channel","type":"string"},{"name":"timestamp","type":"string"},{"name":"flags","type":"string"},{"name":"isUnpatrolled","type":"string"},{"name":"page","type":"string"},{"name":"diffUrl","type":"string"},{"name":"added","type":"long"},{"name":"comment","type":"string"},{"name":"commentLength","type":"long"},{"name":"isNew","type":"string"},{"name":"isMinor","type":"string"},{"name":"delta","type":"long"},{"name":"isAnonymous","type":"s [...]
)
@@ -91,15 +88,17 @@ To generate a query from external data, do the following:
For example, to specify day-based segment granularity, change the partitioning to `PARTITIONED BY DAY`:
```sql
- ...
+ INSERT INTO ...
SELECT
TIME_PARSE("timestamp") AS __time,
...
...
- PARTITIONED BY DAY
+ PARTITIONED BY DAY
```
-1. Optionally, select **Preview** to review the data before you ingest it. A preview runs the query without the INSERT INTO clause and with an added LIMIT to the main query and to all helper queries. You can see the general shape of the data before you commit to inserting it. The LIMITs make the query run faster but can cause incomplete results.
+1. Optionally, select **Preview** to review the data before you ingest it. A preview runs the query without the REPLACE INTO clause and with an added LIMIT.
+ You can see the general shape of the data before you commit to inserting it.
+ The LIMITs make the query run faster but can cause incomplete results.
2. Click **Run** to launch your query. The query returns information including its duration and the number of rows inserted into the table.
## Query the data
@@ -126,7 +125,7 @@ SELECT
COUNT(*)
FROM TABLE(
EXTERN(
- '{"type": "http", "uris": ["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type": "http", "uris": ["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type": "json"}',
'[{"name": "added", "type": "long"}, {"name": "channel", "type": "string"}, {"name": "cityName", "type": "string"}, {"name": "comment", "type": "string"}, {"name": "commentLength", "type": "long"}, {"name": "countryIsoCode", "type": "string"}, {"name": "countryName", "type": "string"}, {"name": "deleted", "type": "long"}, {"name": "delta", "type": "long"}, {"name": "deltaBucket", "type": "string"}, {"name": "diffUrl", "type": "string"}, {"name": "flags", "type": "string"}, {"name": " [...]
)
diff --git a/docs/multi-stage-query/msq-tutorial-convert-ingest-spec.md b/docs/multi-stage-query/msq-tutorial-convert-ingest-spec.md
index 55038e5cfa..cddf887418 100644
--- a/docs/multi-stage-query/msq-tutorial-convert-ingest-spec.md
+++ b/docs/multi-stage-query/msq-tutorial-convert-ingest-spec.md
@@ -34,7 +34,7 @@ To convert the ingestion spec to a query task, do the following:
1. In the **Query** view of the Druid console, navigate to the menu bar that includes **Run**.
2. Click the ellipsis icon and select **Convert ingestion spec to SQL**.
-3. In the **Ingestion spec to covert** window, insert your ingestion spec. You can use your own spec or the sample ingestion spec provided in the tutorial. The sample spec uses data hosted at `https://static.imply.io/data/wikipedia.json.gz` and loads it into a table named `wikipedia`:
+3. In the **Ingestion spec to covert** window, insert your ingestion spec. You can use your own spec or the sample ingestion spec provided in the tutorial. The sample spec uses data hosted at `https://druid.apache.org/data/wikipedia.json.gz` and loads it into a table named `wikipedia`:
<details><summary>Show the spec</summary>
@@ -47,7 +47,7 @@ To convert the ingestion spec to a query task, do the following:
"inputSource": {
"type": "http",
"uris": [
- "https://static.imply.io/data/wikipedia.json.gz"
+ "https://druid.apache.org/data/wikipedia.json.gz"
]
},
"inputFormat": {
@@ -129,7 +129,7 @@ To convert the ingestion spec to a query task, do the following:
REPLACE INTO wikipedia OVERWRITE ALL
WITH source AS (SELECT * FROM TABLE(
EXTERN(
- '{"type":"http","uris":["https://static.imply.io/data/wikipedia.json.gz"]}',
+ '{"type":"http","uris":["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type":"json"}',
'[{"name":"timestamp","type":"string"},{"name":"isRobot","type":"string"},{"name":"channel","type":"string"},{"name":"flags","type":"string"},{"name":"isUnpatrolled","type":"string"},{"name":"page","type":"string"},{"name":"diffUrl","type":"string"},{"name":"added","type":"long"},{"name":"comment","type":"string"},{"name":"commentLength","type":"long"},{"name":"isNew","type":"string"},{"name":"isMinor","type":"string"},{"name":"delta","type":"long"},{"name":"isAnonymous","type":"s [...]
)
diff --git a/docs/operations/management-uis.md b/docs/operations/management-uis.md
deleted file mode 100644
index 2b1d234703..0000000000
--- a/docs/operations/management-uis.md
+++ /dev/null
@@ -1,64 +0,0 @@
----
-id: management-uis
-title: "Legacy Management UIs"
----
-
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one
- ~ or more contributor license agreements. See the NOTICE file
- ~ distributed with this work for additional information
- ~ regarding copyright ownership. The ASF licenses this file
- ~ to you under the Apache License, Version 2.0 (the
- ~ "License"); you may not use this file except in compliance
- ~ with the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing,
- ~ software distributed under the License is distributed on an
- ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- ~ KIND, either express or implied. See the License for the
- ~ specific language governing permissions and limitations
- ~ under the License.
- -->
-
-
-## Legacy consoles
-
-Druid provides a console for managing datasources, segments, tasks, data processes (Historicals and MiddleManagers), and coordinator dynamic configuration. The user can also run SQL and native Druid queries within the console.
-
-For more information on the Druid Console, have a look at the [Druid Console overview](./druid-console.md)
-
-The Druid Console contains all of the functionality provided by the older consoles described below, which are still available if needed. The legacy consoles may be replaced by the Druid Console in the future.
-
-These older consoles provide a subset of the functionality of the Druid Console. We recommend using the Druid Console if possible.
-
-### Coordinator consoles
-
-#### Version 2
-
-The Druid Coordinator exposes a web console for displaying cluster information and rule configuration. After the Coordinator starts, the console can be accessed at:
-
-```
-http://<COORDINATOR_IP>:<COORDINATOR_PORT>
-```
-
-There exists a full cluster view (which shows indexing tasks and Historical processes), as well as views for individual Historical processes, datasources and segments themselves. Segment information can be displayed in raw JSON form or as part of a sortable and filterable table.
-
-The Coordinator console also exposes an interface to creating and editing rules. All valid datasources configured in the segment database, along with a default datasource, are available for configuration. Rules of different types can be added, deleted or edited.
-
-#### Version 1
-
-The oldest version of Druid's Coordinator console is still available for backwards compatibility at:
-
-```
-http://<COORDINATOR_IP>:<COORDINATOR_PORT>/old-console
-```
-
-### Overlord console
-
-The Overlord console can be used to view pending tasks, running tasks, available workers, and recent worker creation and termination. The console can be accessed at:
-
-```
-http://<OVERLORD_IP>:<OVERLORD_PORT>/console.html
-```
diff --git a/docs/operations/rule-configuration.md b/docs/operations/rule-configuration.md
index 1ea074f723..6ad72ebacb 100644
--- a/docs/operations/rule-configuration.md
+++ b/docs/operations/rule-configuration.md
@@ -23,7 +23,7 @@ title: "Retaining or automatically dropping data"
-->
-In Apache Druid, Coordinator processes use rules to determine what data should be loaded to or dropped from the cluster. Rules are used for data retention and query execution, and are set on the Coordinator console (http://coordinator_ip:port).
+In Apache Druid, Coordinator processes use rules to determine what data should be loaded to or dropped from the cluster. Rules are used for data retention and query execution, and are set via the [web console](./druid-console.md).
There are three types of rules, i.e., load rules, drop rules, and broadcast rules. Load rules indicate how segments should be assigned to different historical process tiers and how many replicas of a segment should exist in each tier.
Drop rules indicate when segments should be dropped entirely from the cluster. Finally, broadcast rules indicate how segments of different datasources should be co-located in Historical processes.
@@ -33,7 +33,7 @@ default set of rules can be configured. Rules are read in order and hence the or
Coordinator will cycle through all used segments and match each segment with the first rule that applies. Each segment
may only match a single rule.
-Note: It is recommended that the Coordinator console is used to configure rules. However, the Coordinator process does have HTTP endpoints to programmatically configure rules.
+Note: It is recommended that the web console is used to configure rules. However, the Coordinator process does have HTTP endpoints to programmatically configure rules.
## Load rules
@@ -230,5 +230,5 @@ submit a [kill task](../ingestion/tasks.md) to the [Overlord](../design/overlord
Data that has been dropped from a Druid cluster cannot be reloaded using only rules. To reload dropped data in Druid,
you must first set your retention period (i.e. changing the retention period from 1 month to 2 months), and then mark as
-used all segments belonging to the datasource in the Druid Coordinator console, or through the Druid Coordinator
+used all segments belonging to the datasource in the web console, or through the Druid Coordinator
endpoints.
diff --git a/docs/tutorials/tutorial-msq-connect-extern.md b/docs/tutorials/tutorial-msq-connect-extern.md
index 7d548b8874..379601b90e 100644
--- a/docs/tutorials/tutorial-msq-connect-extern.md
+++ b/docs/tutorials/tutorial-msq-connect-extern.md
@@ -1,7 +1,7 @@
---
id: tutorial-msq-external-data
-title: "Connect external data"
-sidebar_label: "Connect external data"
+title: "Loading files with SQL"
+sidebar_label: "Loading files with SQL"
---
<!--
diff --git a/docs/tutorials/tutorial-msq-convert-json.md b/docs/tutorials/tutorial-msq-convert-json.md
index 9353edb24f..f4ff853d25 100644
--- a/docs/tutorials/tutorial-msq-convert-json.md
+++ b/docs/tutorials/tutorial-msq-convert-json.md
@@ -1,7 +1,7 @@
---
id: tutorial-msq-convert-json
-title: "Convert JSON ingestion spec to SQL"
-sidebar_label: "Convert JSON ingestion spec"
+title: "Convert ingestion spec to SQL"
+sidebar_label: "Convert ingestion spec to SQL"
---
<!--
diff --git a/website/package.json b/website/package.json
index 379f3d11a0..4fa43a77d6 100644
--- a/website/package.json
+++ b/website/package.json
@@ -13,6 +13,12 @@
"lint": "npm run link-lint",
"spellcheck": "mdspell --en-us --ignore-numbers --report '../docs/**/*.md'"
},
+ "engines": {
+ "node": ">=12"
+ },
+ "volta": {
+ "node": "12.22.12"
+ },
"devDependencies": {
"docusaurus": "^1.14.4",
"markdown-spellcheck": "^1.3.1",
diff --git a/website/redirects.json b/website/redirects.json
index d1f4bd94a4..aa7a0fcdec 100644
--- a/website/redirects.json
+++ b/website/redirects.json
@@ -198,3 +198,4 @@
{"source": "development/extensions-contrib/google.html", "target": "../extensions-core/google.html"}
{"source": "development/integrating-druid-with-other-technologies.html", "target": "../ingestion/index.html"}
{"source": "operations/recommendations.html", "target": "basic-cluster-tuning.html"}
+{"source": "operations/management-uis.html", "target": "operations/druid-console.html"}
diff --git a/website/sidebars.json b/website/sidebars.json
index e946a6f3ff..35c0283ec4 100644
--- a/website/sidebars.json
+++ b/website/sidebars.json
@@ -9,6 +9,7 @@
],
"Tutorials": [
"tutorials/tutorial-batch",
+ "tutorials/tutorial-msq-external-data",
"tutorials/tutorial-kafka",
"tutorials/tutorial-batch-hadoop",
"tutorials/tutorial-query",
@@ -21,7 +22,6 @@
"tutorials/tutorial-ingestion-spec",
"tutorials/tutorial-transform-spec",
"tutorials/tutorial-kerberos-hadoop",
- "tutorials/tutorial-msq-external-data",
"tutorials/tutorial-msq-convert-json"
],
"Design": [
@@ -203,7 +203,6 @@
"type": "subcategory",
"label": "Misc",
"ids": [
- "operations/management-uis",
"operations/dump-segment",
"operations/reset-cluster",
"operations/insert-segment-to-db",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org