You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/10/25 10:18:31 UTC
[GitHub] [druid] writer-jill opened a new pull request, #13261: Update Kafka ingestion tutorial
writer-jill opened a new pull request, #13261:
URL: https://github.com/apache/druid/pull/13261
Update the Kafka ingestion tutorial to use a nested data source.
This PR has:
- [x] been self-reviewed
- [x] been tested and reviewed by @ektravel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1020142676
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.tgz
+ tar -xzf kttm-nested-data.json.tgz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017694285
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith merged pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
techdocsmith merged PR #13261:
URL: https://github.com/apache/druid/pull/13261
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017714971
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
Review Comment:
I think it's worthwhile to orient the user. Less possibility of confusion if you instruct them where to put the file and then reference that location in the command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on PR #13261:
URL: https://github.com/apache/druid/pull/13261#issuecomment-1298929116
@writer-jill , can we make sure to include an explicit mention on `querying/nested-columns.md` that streaming ingestion is supported and link out to the tutorial for an example.
FYI @gianm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
gianm commented on PR #13261:
URL: https://github.com/apache/druid/pull/13261#issuecomment-1298929514
Would it be possible to address #13261 as well, by updating `querying/nested-columns.md`? i.e. adding an example to this page: https://druid.apache.org/docs/latest/querying/nested-columns.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017263814
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
Review Comment:
The figures don't all seem to align with the corresponding text in the step (multiple steps)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017251702
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
Review Comment:
Is there a reason this needs to be done from the Druid root directory? It also requires the next step to have `{PATH_TO_DRUID}` but you can post events regardless of the data location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017085680
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
Review Comment:
```suggestion
9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to fine-tune how Druid ingests data.
```
Raised by Travis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017712282
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
Review Comment:
I don't like including the ellipsis and brackets in the instruction. I think it's clear what it's referring to in the UI. I'll add it to our style guide discussion list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1016919446
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
Review Comment:
Replaced the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017694285
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
Review Comment:
I always use gunzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017704898
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
Review Comment:
I don't think that's true. The action is the step, in each case. Then you are presented with the next screen. The next action is the next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017695024
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
Review Comment:
Google style guide says to avoid hyphenating words unless it's essential for their meaning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017708317
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
-Since this is a small dataset, you can simply run a `SELECT * FROM "wikipedia-kafka"` query to see your results.
+### Submit a supervisor spec
-
+As an alternative to using the data loader, you can submit a supervisor spec to Druid. You can do this in the console or using the Druid API.
+
+#### Use the console
+
+To submit a supervisor spec using the Druid console:
-Check out the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-
-### Submit a supervisor via the console
-
-In the console, click `Submit supervisor` to open the submit supervisor dialog.
-
-
-
-Paste in this spec and click `Submit`.
-
-```json
-{
- "type": "kafka",
- "spec" : {
- "dataSchema": {
- "dataSource": "wikipedia",
- "timestampSpec": {
- "column": "time",
- "format": "auto"
- },
- "dimensionsSpec": {
- "dimensions": [
- "channel",
- "cityName",
- "comment",
- "countryIsoCode",
- "countryName",
- "isAnonymous",
- "isMinor",
- "isNew",
- "isRobot",
- "isUnpatrolled",
- "metroCode",
- "namespace",
- "page",
- "regionIsoCode",
- "regionName",
- "user",
- { "name": "added", "type": "long" },
- { "name": "deleted", "type": "long" },
- { "name": "delta", "type": "long" }
- ]
- },
- "metricsSpec" : [],
- "granularitySpec": {
- "type": "uniform",
- "segmentGranularity": "DAY",
- "queryGranularity": "NONE",
- "rollup": false
- }
- },
- "tuningConfig": {
- "type": "kafka",
- "reportParseExceptions": false
- },
- "ioConfig": {
- "topic": "wikipedia",
- "inputFormat": {
- "type": "json"
- },
- "replicas": 2,
- "taskDuration": "PT10M",
- "completionTimeout": "PT20M",
- "consumerProperties": {
- "bootstrap.servers": "localhost:9092"
- }
- }
- }
-}
-```
-
-This will start the supervisor that will in turn spawn some tasks that will start listening for incoming data.
-
-### Submit a supervisor directly
-
-To start the service directly, we will need to submit a supervisor spec to the Druid overlord by running the following from the Druid package root:
-
-```bash
-curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/wikipedia-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
-```
-
-
-If the supervisor was successfully created, you will get a response containing the ID of the supervisor; in our case we should see `{"id":"wikipedia"}`.
-
-For more details about what's going on here, check out the
-[Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md).
-
-You can view the current supervisors and tasks in the web console: [http://localhost:8888/unified-console.md#tasks](http://localhost:8888/unified-console.html#tasks).
-
-## Querying your data
-
-After data is sent to the Kafka stream, it is immediately available for querying.
-
-Please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-## Cleanup
-
-To go through any of the other ingestion tutorials, you will need to shut down the cluster and reset the cluster state by removing the contents of the `var` directory in the Druid home, as the other tutorials will write to the same "wikipedia" datasource.
-
-You should additionally clear out any Kafka state. Do so by shutting down the Kafka broker with CTRL-C before stopping ZooKeeper and the Druid services, and then deleting the Kafka log directory at `/tmp/kafka-logs`:
-
-```bash
-rm -rf /tmp/kafka-logs
-```
+1. Click **Ingestion** in the console, then click the ellipses next to the refresh button and select **Submit JSON supervisor**.
+2. Paste this spec into the JSON window and click **Submit**.
+ ```json
+ {
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092"
+ },
+ "topic": "kttm",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "kttm-kafka-supervisor-console",
+ "timestampSpec": {
+ "column": "timestamp",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "session",
+ "number",
+ "client_ip",
+ "language",
+ "adblock_list",
+ "app_version",
+ "path",
+ "loaded_image",
+ "referrer",
+ "referrer_host",
+ "server_ip",
+ "screen",
+ "window",
+ {
+ "type": "long",
+ "name": "session_length"
+ },
+ "timezone",
+ "timezone_offset",
+ {
+ "type": "json",
+ "name": "event"
+ },
+ {
+ "type": "json",
+ "name": "agent"
+ },
+ {
+ "type": "json",
+ "name": "geo_ip"
+ }
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+ }
+ }
+ }
+ }
+ ```
+
+
+ This starts the supervisor—the supervisor spawns tasks that start listening for incoming data.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-console`.
+
+#### Use the API
+
+You can also use the Druid API to submit a supervisor spec:
+
+1. Run the following commands from your Druid root directory to download the sample spec:
+
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-kafka-supervisor.json
+ ```
+
+2. Run the following command to submit the spec in the `kttm-kafka-supervisor.json` file:
+
+ ```bash
+ curl -XPOST -H 'Content-Type: application/json' kttm-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
+ ```
+
+ After Druid successfully creates the supervisor, you get a response containing the supervisor ID: `{"id":"kttm-kafka-supervisor-api"}`.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-api`.
+
+## Query your data
+
+After Druid sends data to the Kafka stream, it is immediately available for querying. Click **Query** in the Druid console to run SQL queries against the datasource.
+
+Since this tutorial ingests a small dataset, you can run the query `SELECT * FROM "kttm-kafka"` to return all of the data in the dataset you created with the [console data loader](#load-data-with-the-console-data-loader).
Review Comment:
This is related to the screenshot below but agree that the text can be general. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1020142321
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1016041742
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
Review Comment:
Is it just a gz file? Does it work to call `gunzip kttm-nested-data.json.gz`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017255714
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
Review Comment:
```suggestion
In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `None` to use a placeholder value.
```
Match UI casing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1015961992
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
Review Comment:
there's something wrong with this archive.
```
tar -xzf kttm-nested-data.json.gz
```
returns
```
tar: Error opening archive: Unrecognized archive format
```
I did some troubleshooting, but wasn't able to un-tar the file
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
-Since this is a small dataset, you can simply run a `SELECT * FROM "wikipedia-kafka"` query to see your results.
+### Submit a supervisor spec
-
+As an alternative to using the data loader, you can submit a supervisor spec to Druid. You can do this in the console or using the Druid API.
+
+#### Use the console
+
+To submit a supervisor spec using the Druid console:
-Check out the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-
-### Submit a supervisor via the console
-
-In the console, click `Submit supervisor` to open the submit supervisor dialog.
-
-
-
-Paste in this spec and click `Submit`.
-
-```json
-{
- "type": "kafka",
- "spec" : {
- "dataSchema": {
- "dataSource": "wikipedia",
- "timestampSpec": {
- "column": "time",
- "format": "auto"
- },
- "dimensionsSpec": {
- "dimensions": [
- "channel",
- "cityName",
- "comment",
- "countryIsoCode",
- "countryName",
- "isAnonymous",
- "isMinor",
- "isNew",
- "isRobot",
- "isUnpatrolled",
- "metroCode",
- "namespace",
- "page",
- "regionIsoCode",
- "regionName",
- "user",
- { "name": "added", "type": "long" },
- { "name": "deleted", "type": "long" },
- { "name": "delta", "type": "long" }
- ]
- },
- "metricsSpec" : [],
- "granularitySpec": {
- "type": "uniform",
- "segmentGranularity": "DAY",
- "queryGranularity": "NONE",
- "rollup": false
- }
- },
- "tuningConfig": {
- "type": "kafka",
- "reportParseExceptions": false
- },
- "ioConfig": {
- "topic": "wikipedia",
- "inputFormat": {
- "type": "json"
- },
- "replicas": 2,
- "taskDuration": "PT10M",
- "completionTimeout": "PT20M",
- "consumerProperties": {
- "bootstrap.servers": "localhost:9092"
- }
- }
- }
-}
-```
-
-This will start the supervisor that will in turn spawn some tasks that will start listening for incoming data.
-
-### Submit a supervisor directly
-
-To start the service directly, we will need to submit a supervisor spec to the Druid overlord by running the following from the Druid package root:
-
-```bash
-curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/wikipedia-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
-```
-
-
-If the supervisor was successfully created, you will get a response containing the ID of the supervisor; in our case we should see `{"id":"wikipedia"}`.
-
-For more details about what's going on here, check out the
-[Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md).
-
-You can view the current supervisors and tasks in the web console: [http://localhost:8888/unified-console.md#tasks](http://localhost:8888/unified-console.html#tasks).
-
-## Querying your data
-
-After data is sent to the Kafka stream, it is immediately available for querying.
-
-Please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-## Cleanup
-
-To go through any of the other ingestion tutorials, you will need to shut down the cluster and reset the cluster state by removing the contents of the `var` directory in the Druid home, as the other tutorials will write to the same "wikipedia" datasource.
-
-You should additionally clear out any Kafka state. Do so by shutting down the Kafka broker with CTRL-C before stopping ZooKeeper and the Druid services, and then deleting the Kafka log directory at `/tmp/kafka-logs`:
-
-```bash
-rm -rf /tmp/kafka-logs
-```
+1. Click **Ingestion** in the console, then click the ellipses next to the refresh button and select **Submit JSON supervisor**.
+2. Paste this spec into the JSON window and click **Submit**.
+ ```json
+ {
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092"
+ },
+ "topic": "kttm",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "kttm-kafka-supervisor-console",
+ "timestampSpec": {
+ "column": "timestamp",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "session",
+ "number",
+ "client_ip",
+ "language",
+ "adblock_list",
+ "app_version",
+ "path",
+ "loaded_image",
+ "referrer",
+ "referrer_host",
+ "server_ip",
+ "screen",
+ "window",
+ {
+ "type": "long",
+ "name": "session_length"
+ },
+ "timezone",
+ "timezone_offset",
+ {
+ "type": "json",
+ "name": "event"
+ },
+ {
+ "type": "json",
+ "name": "agent"
+ },
+ {
+ "type": "json",
+ "name": "geo_ip"
+ }
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+ }
+ }
+ }
+ }
+ ```
+
+
+ This starts the supervisor—the supervisor spawns tasks that start listening for incoming data.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-console`.
+
+#### Use the API
+
+You can also use the Druid API to submit a supervisor spec:
+
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
+
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-kafka-supervisor.json.gz
+ tar -xzf kttm-kafka-supervisor.json.gz
Review Comment:
same problem with this archive as line 73. The supervisor spec shouldn't be that large, so could we just use the `.json` for his one instead of bothering with zip. so replacing the zip with the json and then it's one command
```bash
curl -O https://druid.apache.org/docs/latest/assets/files/kttm-kafka-supervisor.json
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on PR #13261:
URL: https://github.com/apache/druid/pull/13261#issuecomment-1290322884
One outstanding item is to decide where to host the kttm-kafka-supervisor.json file and then update the reference to it. It's not currently in the Druid distribution along with the other sample data files (quickstart/tutorial).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1020143752
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.tgz
+ tar -xzf kttm-nested-data.json.tgz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time` column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `None` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The console does most of this for you, but you need to create JSON-type dimensions for the three nested columns in the data.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to fine tune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions(...)** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
-Since this is a small dataset, you can simply run a `SELECT * FROM "wikipedia-kafka"` query to see your results.
+### Submit a supervisor spec
-
+As an alternative to using the data loader, you can submit a supervisor spec to Druid. You can do this in the console or using the Druid API.
+
+#### Use the console
+
+To submit a supervisor spec using the Druid console:
-Check out the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-
-### Submit a supervisor via the console
-
-In the console, click `Submit supervisor` to open the submit supervisor dialog.
-
-
-
-Paste in this spec and click `Submit`.
-
-```json
-{
- "type": "kafka",
- "spec" : {
- "dataSchema": {
- "dataSource": "wikipedia",
- "timestampSpec": {
- "column": "time",
- "format": "auto"
- },
- "dimensionsSpec": {
- "dimensions": [
- "channel",
- "cityName",
- "comment",
- "countryIsoCode",
- "countryName",
- "isAnonymous",
- "isMinor",
- "isNew",
- "isRobot",
- "isUnpatrolled",
- "metroCode",
- "namespace",
- "page",
- "regionIsoCode",
- "regionName",
- "user",
- { "name": "added", "type": "long" },
- { "name": "deleted", "type": "long" },
- { "name": "delta", "type": "long" }
- ]
- },
- "metricsSpec" : [],
- "granularitySpec": {
- "type": "uniform",
- "segmentGranularity": "DAY",
- "queryGranularity": "NONE",
- "rollup": false
- }
- },
- "tuningConfig": {
- "type": "kafka",
- "reportParseExceptions": false
- },
- "ioConfig": {
- "topic": "wikipedia",
- "inputFormat": {
- "type": "json"
- },
- "replicas": 2,
- "taskDuration": "PT10M",
- "completionTimeout": "PT20M",
- "consumerProperties": {
- "bootstrap.servers": "localhost:9092"
- }
- }
- }
-}
-```
-
-This will start the supervisor that will in turn spawn some tasks that will start listening for incoming data.
-
-### Submit a supervisor directly
-
-To start the service directly, we will need to submit a supervisor spec to the Druid overlord by running the following from the Druid package root:
-
-```bash
-curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/wikipedia-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
-```
-
-
-If the supervisor was successfully created, you will get a response containing the ID of the supervisor; in our case we should see `{"id":"wikipedia"}`.
-
-For more details about what's going on here, check out the
-[Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md).
-
-You can view the current supervisors and tasks in the web console: [http://localhost:8888/unified-console.md#tasks](http://localhost:8888/unified-console.html#tasks).
-
-## Querying your data
-
-After data is sent to the Kafka stream, it is immediately available for querying.
-
-Please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-## Cleanup
-
-To go through any of the other ingestion tutorials, you will need to shut down the cluster and reset the cluster state by removing the contents of the `var` directory in the Druid home, as the other tutorials will write to the same "wikipedia" datasource.
-
-You should additionally clear out any Kafka state. Do so by shutting down the Kafka broker with CTRL-C before stopping ZooKeeper and the Druid services, and then deleting the Kafka log directory at `/tmp/kafka-logs`:
-
-```bash
-rm -rf /tmp/kafka-logs
-```
+1. Click **Ingestion** in the console, then click the ellipses next to the refresh button and select **Submit JSON supervisor**.
+2. Paste this spec into the JSON window and click **Submit**.
+ ```json
+ {
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092"
+ },
+ "topic": "kttm",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "kttm-kafka-supervisor-console",
+ "timestampSpec": {
+ "column": "timestamp",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "session",
+ "number",
+ "client_ip",
+ "language",
+ "adblock_list",
+ "app_version",
+ "path",
+ "loaded_image",
+ "referrer",
+ "referrer_host",
+ "server_ip",
+ "screen",
+ "window",
+ {
+ "type": "long",
+ "name": "session_length"
+ },
+ "timezone",
+ "timezone_offset",
+ {
+ "type": "json",
+ "name": "event"
+ },
+ {
+ "type": "json",
+ "name": "agent"
+ },
+ {
+ "type": "json",
+ "name": "geo_ip"
+ }
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+ }
+ }
+ }
+ }
+ ```
+
+
+ This starts the supervisor—the supervisor spawns tasks that start listening for incoming data.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-console`.
+
+#### Use the API
+
+You can also use the Druid API to submit a supervisor spec:
+
+1. Run the following commands from your Druid root directory to download the sample spec:
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1016919842
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
-Since this is a small dataset, you can simply run a `SELECT * FROM "wikipedia-kafka"` query to see your results.
+### Submit a supervisor spec
-
+As an alternative to using the data loader, you can submit a supervisor spec to Druid. You can do this in the console or using the Druid API.
+
+#### Use the console
+
+To submit a supervisor spec using the Druid console:
-Check out the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-
-### Submit a supervisor via the console
-
-In the console, click `Submit supervisor` to open the submit supervisor dialog.
-
-
-
-Paste in this spec and click `Submit`.
-
-```json
-{
- "type": "kafka",
- "spec" : {
- "dataSchema": {
- "dataSource": "wikipedia",
- "timestampSpec": {
- "column": "time",
- "format": "auto"
- },
- "dimensionsSpec": {
- "dimensions": [
- "channel",
- "cityName",
- "comment",
- "countryIsoCode",
- "countryName",
- "isAnonymous",
- "isMinor",
- "isNew",
- "isRobot",
- "isUnpatrolled",
- "metroCode",
- "namespace",
- "page",
- "regionIsoCode",
- "regionName",
- "user",
- { "name": "added", "type": "long" },
- { "name": "deleted", "type": "long" },
- { "name": "delta", "type": "long" }
- ]
- },
- "metricsSpec" : [],
- "granularitySpec": {
- "type": "uniform",
- "segmentGranularity": "DAY",
- "queryGranularity": "NONE",
- "rollup": false
- }
- },
- "tuningConfig": {
- "type": "kafka",
- "reportParseExceptions": false
- },
- "ioConfig": {
- "topic": "wikipedia",
- "inputFormat": {
- "type": "json"
- },
- "replicas": 2,
- "taskDuration": "PT10M",
- "completionTimeout": "PT20M",
- "consumerProperties": {
- "bootstrap.servers": "localhost:9092"
- }
- }
- }
-}
-```
-
-This will start the supervisor that will in turn spawn some tasks that will start listening for incoming data.
-
-### Submit a supervisor directly
-
-To start the service directly, we will need to submit a supervisor spec to the Druid overlord by running the following from the Druid package root:
-
-```bash
-curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/wikipedia-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
-```
-
-
-If the supervisor was successfully created, you will get a response containing the ID of the supervisor; in our case we should see `{"id":"wikipedia"}`.
-
-For more details about what's going on here, check out the
-[Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md).
-
-You can view the current supervisors and tasks in the web console: [http://localhost:8888/unified-console.md#tasks](http://localhost:8888/unified-console.html#tasks).
-
-## Querying your data
-
-After data is sent to the Kafka stream, it is immediately available for querying.
-
-Please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-## Cleanup
-
-To go through any of the other ingestion tutorials, you will need to shut down the cluster and reset the cluster state by removing the contents of the `var` directory in the Druid home, as the other tutorials will write to the same "wikipedia" datasource.
-
-You should additionally clear out any Kafka state. Do so by shutting down the Kafka broker with CTRL-C before stopping ZooKeeper and the Druid services, and then deleting the Kafka log directory at `/tmp/kafka-logs`:
-
-```bash
-rm -rf /tmp/kafka-logs
-```
+1. Click **Ingestion** in the console, then click the ellipses next to the refresh button and select **Submit JSON supervisor**.
+2. Paste this spec into the JSON window and click **Submit**.
+ ```json
+ {
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092"
+ },
+ "topic": "kttm",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "kttm-kafka-supervisor-console",
+ "timestampSpec": {
+ "column": "timestamp",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "session",
+ "number",
+ "client_ip",
+ "language",
+ "adblock_list",
+ "app_version",
+ "path",
+ "loaded_image",
+ "referrer",
+ "referrer_host",
+ "server_ip",
+ "screen",
+ "window",
+ {
+ "type": "long",
+ "name": "session_length"
+ },
+ "timezone",
+ "timezone_offset",
+ {
+ "type": "json",
+ "name": "event"
+ },
+ {
+ "type": "json",
+ "name": "agent"
+ },
+ {
+ "type": "json",
+ "name": "geo_ip"
+ }
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+ }
+ }
+ }
+ }
+ ```
+
+
+ This starts the supervisor—the supervisor spawns tasks that start listening for incoming data.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-console`.
+
+#### Use the API
+
+You can also use the Druid API to submit a supervisor spec:
+
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
+
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-kafka-supervisor.json.gz
+ tar -xzf kttm-kafka-supervisor.json.gz
Review Comment:
Replaced with the .json file and updated command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017706303
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
Review Comment:
Yes - updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017231257
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
Review Comment:
```suggestion
The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) website into a Kafka topic, then ingest the data into Druid.
```
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
Review Comment:
This statement almost makes it seem that you have to configure all the dimensions and metrics, but the web console actually does most of this for you -- it's just the JSON dimensions that need to be added. Worth clarifying?
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
Review Comment:
```suggestion
Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time` column in your Druid datasource.
```
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
Review Comment:
Is there a reason this needs to be done from the Druid root directory? It also requires the next step to have `{PATH_TO_DRUID}` but I think you should be able to post events regardless of the data location.
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
Review Comment:
I think this figure belongs in the next step (and they all should get shifted down a step)
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
Review Comment:
```suggestion
In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `None` to use a placeholder value.
```
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
Review Comment:
The tar command did not work for me, but gunzip did.
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
Review Comment:
```suggestion
> If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions (...)** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
```
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time`) column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `none` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The sample data contains three nested columns, so you need to create JSON-type dimensions for them.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to finetune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
-Since this is a small dataset, you can simply run a `SELECT * FROM "wikipedia-kafka"` query to see your results.
+### Submit a supervisor spec
-
+As an alternative to using the data loader, you can submit a supervisor spec to Druid. You can do this in the console or using the Druid API.
+
+#### Use the console
+
+To submit a supervisor spec using the Druid console:
-Check out the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-
-### Submit a supervisor via the console
-
-In the console, click `Submit supervisor` to open the submit supervisor dialog.
-
-
-
-Paste in this spec and click `Submit`.
-
-```json
-{
- "type": "kafka",
- "spec" : {
- "dataSchema": {
- "dataSource": "wikipedia",
- "timestampSpec": {
- "column": "time",
- "format": "auto"
- },
- "dimensionsSpec": {
- "dimensions": [
- "channel",
- "cityName",
- "comment",
- "countryIsoCode",
- "countryName",
- "isAnonymous",
- "isMinor",
- "isNew",
- "isRobot",
- "isUnpatrolled",
- "metroCode",
- "namespace",
- "page",
- "regionIsoCode",
- "regionName",
- "user",
- { "name": "added", "type": "long" },
- { "name": "deleted", "type": "long" },
- { "name": "delta", "type": "long" }
- ]
- },
- "metricsSpec" : [],
- "granularitySpec": {
- "type": "uniform",
- "segmentGranularity": "DAY",
- "queryGranularity": "NONE",
- "rollup": false
- }
- },
- "tuningConfig": {
- "type": "kafka",
- "reportParseExceptions": false
- },
- "ioConfig": {
- "topic": "wikipedia",
- "inputFormat": {
- "type": "json"
- },
- "replicas": 2,
- "taskDuration": "PT10M",
- "completionTimeout": "PT20M",
- "consumerProperties": {
- "bootstrap.servers": "localhost:9092"
- }
- }
- }
-}
-```
-
-This will start the supervisor that will in turn spawn some tasks that will start listening for incoming data.
-
-### Submit a supervisor directly
-
-To start the service directly, we will need to submit a supervisor spec to the Druid overlord by running the following from the Druid package root:
-
-```bash
-curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/wikipedia-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
-```
-
-
-If the supervisor was successfully created, you will get a response containing the ID of the supervisor; in our case we should see `{"id":"wikipedia"}`.
-
-For more details about what's going on here, check out the
-[Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md).
-
-You can view the current supervisors and tasks in the web console: [http://localhost:8888/unified-console.md#tasks](http://localhost:8888/unified-console.html#tasks).
-
-## Querying your data
-
-After data is sent to the Kafka stream, it is immediately available for querying.
-
-Please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-## Cleanup
-
-To go through any of the other ingestion tutorials, you will need to shut down the cluster and reset the cluster state by removing the contents of the `var` directory in the Druid home, as the other tutorials will write to the same "wikipedia" datasource.
-
-You should additionally clear out any Kafka state. Do so by shutting down the Kafka broker with CTRL-C before stopping ZooKeeper and the Druid services, and then deleting the Kafka log directory at `/tmp/kafka-logs`:
-
-```bash
-rm -rf /tmp/kafka-logs
-```
+1. Click **Ingestion** in the console, then click the ellipses next to the refresh button and select **Submit JSON supervisor**.
+2. Paste this spec into the JSON window and click **Submit**.
+ ```json
+ {
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092"
+ },
+ "topic": "kttm",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "kttm-kafka-supervisor-console",
+ "timestampSpec": {
+ "column": "timestamp",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "session",
+ "number",
+ "client_ip",
+ "language",
+ "adblock_list",
+ "app_version",
+ "path",
+ "loaded_image",
+ "referrer",
+ "referrer_host",
+ "server_ip",
+ "screen",
+ "window",
+ {
+ "type": "long",
+ "name": "session_length"
+ },
+ "timezone",
+ "timezone_offset",
+ {
+ "type": "json",
+ "name": "event"
+ },
+ {
+ "type": "json",
+ "name": "agent"
+ },
+ {
+ "type": "json",
+ "name": "geo_ip"
+ }
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+ }
+ }
+ }
+ }
+ ```
+
+
+ This starts the supervisor—the supervisor spawns tasks that start listening for incoming data.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-console`.
+
+#### Use the API
+
+You can also use the Druid API to submit a supervisor spec:
+
+1. Run the following commands from your Druid root directory to download the sample spec:
+
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-kafka-supervisor.json
+ ```
+
+2. Run the following command to submit the spec in the `kttm-kafka-supervisor.json` file:
+
+ ```bash
+ curl -XPOST -H 'Content-Type: application/json' kttm-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
+ ```
+
+ After Druid successfully creates the supervisor, you get a response containing the supervisor ID: `{"id":"kttm-kafka-supervisor-api"}`.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-api`.
+
+## Query your data
+
+After Druid sends data to the Kafka stream, it is immediately available for querying. Click **Query** in the Druid console to run SQL queries against the datasource.
+
+Since this tutorial ingests a small dataset, you can run the query `SELECT * FROM "kttm-kafka"` to return all of the data in the dataset you created with the [console data loader](#load-data-with-the-console-data-loader).
Review Comment:
This query isn't specific to using the data loader, is it? I think it should also hold if you submit a supervisor spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1019569782
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.tgz
+ tar -xzf kttm-nested-data.json.tgz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
Review Comment:
file name is wrong here. In the new archive the file name is `kttm-nested-v2-2019-08-25.json`
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.tgz
+ tar -xzf kttm-nested-data.json.tgz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
Review Comment:
If you accept my change from the comment above, you can get rid of the "replace path to druid" bit since it will all be contained w/in the Kafka directory.
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
Review Comment:
Change `sample spec` to `sample data`
My point here is that the file doesn't need to be in the Druid directory and it makes you have to do the "path to druid" later, if we download the file into the Kafka directory then it can all be scripted w/o effort on the user's part.
for example
From within your kafka directory, create a directory for the sample data. Download the data, and extract it:
```
mkdir sample-data
cd sample-data
curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.tgz
tar -xzf kttm-nested-data.json.tgz
```
Then in line 80, the command is simply:
```
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < ./sample-data/kttm-nested-v2-2019-08-25.json
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.tgz
+ tar -xzf kttm-nested-data.json.tgz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
-```bash
-export KAFKA_OPTS="-Dfile.encoding=UTF-8"
-./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ export KAFKA_OPTS="-Dfile.encoding=UTF-8"
+ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kttm < {PATH_TO_DRUID}/kttm-nested-data.json
+ ```
-The previous command posted sample events to the *wikipedia* Kafka topic.
-Now we will use Druid's Kafka indexing service to ingest messages from our newly created topic.
+## Load data into Druid
-## Loading data with the data loader
+Now that you have data in your Kafka topic, you can use Druid's Kafka indexing service to ingest the data into Druid.
-Navigate to [localhost:8888](http://localhost:8888) and click `Load data` in the console header.
+To do this, you can use the Druid console data loader or you can submit a supervisor spec. Follow the steps below to try each method.
-
+### Load data with the console data loader
-Select `Apache Kafka` and click `Connect data`.
+The Druid console data loader presents you with several screens to configure each section of the supervisor spec, then creates an ingestion task to ingest the Kafka data.
-
+To use the console data loader:
-Enter `localhost:9092` as the bootstrap server and `wikipedia` as the topic.
+1. Navigate to [localhost:8888](http://localhost:8888) and click **Load data > Streaming**.
-Click `Apply` and make sure that the data you are seeing is correct.
+ 
-Once the data is located, you can click "Next: Parse data" to go to the next step.
+2. Click **Apache Kafka** and then **Connect data**.
-
+3. Enter `localhost:9092` as the bootstrap server and `kttm` as the topic, then click **Apply** and make sure you see data similar to the following:
-The data loader will try to automatically determine the correct parser for the data.
-In this case it will successfully determine `json`.
-Feel free to play around with different parser options to get a preview of how Druid will parse your data.
+ 
-With the `json` parser selected, click `Next: Parse time` to get to the step centered around determining your primary timestamp column.
+4. Click **Next: Parse data**.
-
+ 
-Druid's architecture requires a primary timestamp column (internally stored in a column called `__time`).
-If you do not have a timestamp in your data, select `Constant value`.
-In our example, the data loader will determine that the `time` column in our raw data is the only candidate that can be used as the primary time column.
+ The data loader automatically tries to determine the correct parser for the data. For the sample data, it selects input format `json`. You can play around with the different options to get a preview of how Druid parses your data.
-Click `Next: ...` twice to go past the `Transform` and `Filter` steps.
-You do not need to enter anything in these steps as applying ingestion time transforms and filters are out of scope for this tutorial.
+5. With the `json` input format selected, click **Next: Parse time**. You may need to click **Apply** first.
-
+ 
-In the `Configure schema` step, you can configure which [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) will be ingested into Druid.
-This is exactly what the data will appear like in Druid once it is ingested.
-Since our dataset is very small, go ahead and turn off [`Rollup`](../ingestion/rollup.md) by clicking on the switch and confirming the change.
+ Druid's architecture requires that you specify a primary timestamp column. Druid stores the timestamp in the `__time` column in your Druid datasource.
+ In a production environment, if you don't have a timestamp in your data, you can select **Parse timestamp from:** `None` to use a placeholder value.
-Once you are satisfied with the schema, click `Next` to go to the `Partition` step where you can fine tune how the data will be partitioned into segments.
+ For the sample data, the data loader selects the `timestamp` column in the raw data as the primary time column.
-
+6. Click **Next: ...** three times to go past the **Transform** and **Filter** steps to **Configure schema**. You don't need to enter anything in these two steps because applying transforms and filters is out of scope for this tutorial.
-Here, you can adjust how the data will be split up into segments in Druid.
-Since this is a small dataset, there are no adjustments that need to be made in this step.
+ 
-Click `Next: Tune` to go to the tuning step.
+7. In the **Configure schema** step, you can select data types for the columns and configure [dimensions](../ingestion/data-model.md#dimensions) and [metrics](../ingestion/data-model.md#metrics) to ingest into Druid. The console does most of this for you, but you need to create JSON-type dimensions for the three nested columns in the data.
-
+ Click **Add dimension** and enter the following information. You can only add one dimension at a time.
+ - Name: `event`, Type: `json`
+ - Name: `agent`, Type: `json`
+ - Name: `geo_ip`, Type: `json`
+
+ After you create the dimensions, you can scroll to the right in the preview window to see the nested columns:
-In the `Tune` step is it *very important* to set `Use earliest offset` to `True` since we want to consume the data from the start of the stream.
-There are no other changes that need to be made here, so click `Next: Publish` to go to the `Publish` step.
+ 
-
+8. Click **Next: Partition** to configure how Druid partitions the data into segments.
-Let's name this datasource `wikipedia-kafka`.
+ 
-Finally, click `Next` to review your spec.
+9. Select `day` as the **Segment granularity**. Since this is a small dataset, you don't need to make any further adjustments. Click **Next: Tune** to fine tune how Druid ingests data.
+
+ 
-
+10. In **Input tuning**, set **Use earliest offset** to `True`—this is very important because you want to consume the data from the start of the stream. There are no other changes to make here, so click **Next: Publish**.
-This is the spec you have constructed.
-Feel free to go back and make changes in previous steps to see how changes will update the spec.
-Similarly, you can also edit the spec directly and see it reflected in the previous steps.
+ 
-Once you are satisfied with the spec, click `Submit` and an ingestion task will be created.
+11. Name the datasource `kttm-kafka` and click **Next: Edit spec** to review your spec.
-
+ 
-You will be taken to the task view with the focus on the newly created supervisor.
+ The console presents the spec you've constructed. You can click the buttons above the spec to make changes in previous steps and see how the changes update the spec. You can also edit the spec directly and see it reflected in the previous steps.
+
+12. Click **Submit** to create an ingestion task.
-The task view is set to auto refresh, wait until your supervisor launches a task.
+ Druid displays the task view with the focus on the newly created supervisor.
-When a tasks starts running, it will also start serving the data that it is ingesting.
+ The task view auto-refreshes, so wait until the supervisor launches a task. The status changes from **Pending** to **Running** as Druid starts to ingest data.
-Navigate to the `Datasources` view from the header.
+ 
-
+13. Navigate to the **Datasources** view from the header.
-When the `wikipedia-kafka` datasource appears here it can be queried.
+ 
-*Note:* if the datasource does not appear after a minute you might have not set the supervisor to read from the start of the stream (in the `Tune` step).
+ When the `kttm-kafka` datasource appears here, you can query it. See [Query your data](#query-your-data) for details.
-At this point, you can go to the `Query` view to run SQL queries against the datasource.
+ > If the datasource doesn't appear after a minute you might not have set the supervisor to read data from the start of the stream—the `Use earliest offset` setting in the **Tune** step. Go to the **Ingestion** page and terminate the supervisor using the **Actions(...)** menu. [Load the sample data](#load-data-with-the-console-data-loader) again and apply the correct setting when you get to the **Tune** step.
-Since this is a small dataset, you can simply run a `SELECT * FROM "wikipedia-kafka"` query to see your results.
+### Submit a supervisor spec
-
+As an alternative to using the data loader, you can submit a supervisor spec to Druid. You can do this in the console or using the Druid API.
+
+#### Use the console
+
+To submit a supervisor spec using the Druid console:
-Check out the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-
-### Submit a supervisor via the console
-
-In the console, click `Submit supervisor` to open the submit supervisor dialog.
-
-
-
-Paste in this spec and click `Submit`.
-
-```json
-{
- "type": "kafka",
- "spec" : {
- "dataSchema": {
- "dataSource": "wikipedia",
- "timestampSpec": {
- "column": "time",
- "format": "auto"
- },
- "dimensionsSpec": {
- "dimensions": [
- "channel",
- "cityName",
- "comment",
- "countryIsoCode",
- "countryName",
- "isAnonymous",
- "isMinor",
- "isNew",
- "isRobot",
- "isUnpatrolled",
- "metroCode",
- "namespace",
- "page",
- "regionIsoCode",
- "regionName",
- "user",
- { "name": "added", "type": "long" },
- { "name": "deleted", "type": "long" },
- { "name": "delta", "type": "long" }
- ]
- },
- "metricsSpec" : [],
- "granularitySpec": {
- "type": "uniform",
- "segmentGranularity": "DAY",
- "queryGranularity": "NONE",
- "rollup": false
- }
- },
- "tuningConfig": {
- "type": "kafka",
- "reportParseExceptions": false
- },
- "ioConfig": {
- "topic": "wikipedia",
- "inputFormat": {
- "type": "json"
- },
- "replicas": 2,
- "taskDuration": "PT10M",
- "completionTimeout": "PT20M",
- "consumerProperties": {
- "bootstrap.servers": "localhost:9092"
- }
- }
- }
-}
-```
-
-This will start the supervisor that will in turn spawn some tasks that will start listening for incoming data.
-
-### Submit a supervisor directly
-
-To start the service directly, we will need to submit a supervisor spec to the Druid overlord by running the following from the Druid package root:
-
-```bash
-curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/wikipedia-kafka-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor
-```
-
-
-If the supervisor was successfully created, you will get a response containing the ID of the supervisor; in our case we should see `{"id":"wikipedia"}`.
-
-For more details about what's going on here, check out the
-[Druid Kafka indexing service documentation](../development/extensions-core/kafka-ingestion.md).
-
-You can view the current supervisors and tasks in the web console: [http://localhost:8888/unified-console.md#tasks](http://localhost:8888/unified-console.html#tasks).
-
-## Querying your data
-
-After data is sent to the Kafka stream, it is immediately available for querying.
-
-Please follow the [query tutorial](../tutorials/tutorial-query.md) to run some example queries on the newly loaded data.
-
-## Cleanup
-
-To go through any of the other ingestion tutorials, you will need to shut down the cluster and reset the cluster state by removing the contents of the `var` directory in the Druid home, as the other tutorials will write to the same "wikipedia" datasource.
-
-You should additionally clear out any Kafka state. Do so by shutting down the Kafka broker with CTRL-C before stopping ZooKeeper and the Druid services, and then deleting the Kafka log directory at `/tmp/kafka-logs`:
-
-```bash
-rm -rf /tmp/kafka-logs
-```
+1. Click **Ingestion** in the console, then click the ellipses next to the refresh button and select **Submit JSON supervisor**.
+2. Paste this spec into the JSON window and click **Submit**.
+ ```json
+ {
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092"
+ },
+ "topic": "kttm",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "kttm-kafka-supervisor-console",
+ "timestampSpec": {
+ "column": "timestamp",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "session",
+ "number",
+ "client_ip",
+ "language",
+ "adblock_list",
+ "app_version",
+ "path",
+ "loaded_image",
+ "referrer",
+ "referrer_host",
+ "server_ip",
+ "screen",
+ "window",
+ {
+ "type": "long",
+ "name": "session_length"
+ },
+ "timezone",
+ "timezone_offset",
+ {
+ "type": "json",
+ "name": "event"
+ },
+ {
+ "type": "json",
+ "name": "agent"
+ },
+ {
+ "type": "json",
+ "name": "geo_ip"
+ }
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+ }
+ }
+ }
+ }
+ ```
+
+
+ This starts the supervisor—the supervisor spawns tasks that start listening for incoming data.
+
+3. Click **Tasks** on the console home page to monitor the status of the job. This spec writes the data in the `kttm` topic to a datasource named `kttm-kafka-supervisor-console`.
+
+#### Use the API
+
+You can also use the Druid API to submit a supervisor spec:
+
+1. Run the following commands from your Druid root directory to download the sample spec:
Review Comment:
nit: this doesn't necessarily need to be done from the Druid root directory. The commands use curl.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1020142908
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.tgz
+ tar -xzf kttm-nested-data.json.tgz
+ ```
-In your Kafka directory, run the following command, where {PATH_TO_DRUID} is replaced by the path to the Druid directory:
+2. In your Kafka root directory, run the following commands to post sample events to the `kttm` Kafka topic. Replace `{PATH_TO_DRUID}` with the path to your Druid root directory:
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017695798
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,267 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
Review Comment:
It is a game, though. It's clear from the link that it's a website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #13261: Update Kafka ingestion tutorial
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #13261:
URL: https://github.com/apache/druid/pull/13261#discussion_r1017694285
##########
docs/tutorials/tutorial-kafka.md:
##########
@@ -24,260 +24,268 @@ sidebar_label: "Load from Apache Kafka"
-->
-## Getting started
+This tutorial shows you how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
-This tutorial demonstrates how to load data into Apache Druid from a Kafka stream, using Druid's Kafka indexing service.
+The tutorial guides you through the steps to load sample nested clickstream data from the [Koalas to the Max](https://www.koalastothemax.com/) game into a Kafka topic, then ingest the data into Druid.
-For this tutorial, we'll assume you've already downloaded Druid as described in
-the [quickstart](index.md) using the `micro-quickstart` single-machine configuration and have it
-running on your local machine. You don't need to have loaded any data yet.
+## Prerequisites
+
+Before you follow the steps in this tutorial, download Druid as described in the [quickstart](index.md) using the [micro-quickstart](../operations/single-server.md#micro-quickstart-4-cpu-16gib-ram) single-machine configuration and have it running on your local machine. You don't need to have loaded any data.
## Download and start Kafka
-[Apache Kafka](http://kafka.apache.org/) is a high throughput message bus that works well with
-Druid. For this tutorial, we will use Kafka 2.7.0. To download Kafka, issue the following
-commands in your terminal:
+[Apache Kafka](http://kafka.apache.org/) is a high-throughput message bus that works well with Druid. For this tutorial, use Kafka 2.7.0.
+
+1. To download Kafka, run the following commands in your terminal:
-```bash
-curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
-tar -xzf kafka_2.13-2.7.0.tgz
-cd kafka_2.13-2.7.0
-```
-Start zookeeper first with the following command:
+ ```bash
+ curl -O https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
+ tar -xzf kafka_2.13-2.7.0.tgz
+ cd kafka_2.13-2.7.0
+ ```
+2. If you're already running Kafka on the machine you're using for this tutorial, delete or rename the `kafka-logs` directory in `/tmp`.
+
+ > Druid and Kafka both rely on [Apache ZooKeeper](https://zookeeper.apache.org/) to coordinate and manage services. Because Druid is already running, Kafka attaches to the Druid ZooKeeper instance when it starts up.<br>
+ In a production environment where you're running Druid and Kafka on different machines, [start the Kafka ZooKeeper](https://kafka.apache.org/quickstart) before you start the Kafka broker.
-```bash
-./bin/zookeeper-server-start.sh config/zookeeper.properties
-```
+3. In the Kafka root directory, run this command to start a Kafka broker:
-Start a Kafka broker by running the following command in a new terminal:
+ ```bash
+ ./bin/kafka-server-start.sh config/server.properties
+ ```
-```bash
-./bin/kafka-server-start.sh config/server.properties
-```
+4. In a new terminal window, navigate to the Kafka root directory and run the following command to create a Kafka topic called `kttm`:
-Run this command to create a Kafka topic called *wikipedia*, to which we'll send data:
+ ```bash
+ ./bin/kafka-topics.sh --create --topic kttm --bootstrap-server localhost:9092
+ ```
-```bash
-./bin/kafka-topics.sh --create --topic wikipedia --bootstrap-server localhost:9092
-```
+ Kafka returns a message when it successfully adds the topic: `Created topic kttm`.
## Load data into Kafka
-Let's launch a producer for our topic and send some data!
+In this section, you download sample data to the tutorial's directory and send the data to your Kafka topic.
-In your Druid directory, run the following command:
+1. Run the following commands from your Druid root directory to download and extract the sample spec:
-```bash
-cd quickstart/tutorial
-gunzip -c wikiticker-2015-09-12-sampled.json.gz > wikiticker-2015-09-12-sampled.json
-```
+ ```bash
+ curl -O https://druid.apache.org/docs/latest/assets/files/kttm-nested-data.json.gz
+ tar -xzf kttm-nested-data.json.gz
Review Comment:
There was still a problem with the file - replaced with the original one and tested. Looks ok now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org