You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by ur...@apache.org on 2022/05/10 12:09:07 UTC
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#ca614e0)
This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 33881a59715 Docs sync done from apache/pulsar(#ca614e0)
33881a59715 is described below
commit 33881a59715de56cc8cd6b4f01defc4ef89b7b72
Author: Pulsar Site Updater <de...@pulsar.apache.org>
AuthorDate: Tue May 10 12:09:03 2022 +0000
Docs sync done from apache/pulsar(#ca614e0)
---
site2/docs/getting-started-docker.md | 4 +-
site2/docs/io-kafka-source.md | 152 +++++++++--------
site2/website-next/docs/getting-started-docker.md | 5 +-
site2/website-next/docs/io-kafka-source.md | 180 +++++++++++----------
.../getting-started-docker.md | 5 +-
.../version-2.10.0/getting-started-docker.md | 5 +-
.../version-2.10.0/io-kafka-source.md | 180 +++++++++++----------
.../version-2.2.0/getting-started-docker.md | 5 +-
.../version-2.3.0/getting-started-docker.md | 5 +-
.../version-2.3.1/getting-started-docker.md | 4 +-
.../version-2.3.2/getting-started-docker.md | 5 +-
.../version-2.4.0/getting-started-docker.md | 5 +-
.../version-2.4.1/getting-started-docker.md | 5 +-
.../version-2.4.2/getting-started-docker.md | 5 +-
.../version-2.5.0/getting-started-docker.md | 5 +-
.../version-2.5.1/getting-started-docker.md | 5 +-
.../version-2.5.2/getting-started-docker.md | 5 +-
.../version-2.6.0/getting-started-docker.md | 5 +-
.../version-2.6.1/getting-started-docker.md | 5 +-
.../version-2.6.2/getting-started-docker.md | 5 +-
.../version-2.6.3/getting-started-docker.md | 5 +-
.../version-2.6.4/getting-started-docker.md | 5 +-
.../version-2.7.0/getting-started-docker.md | 5 +-
.../version-2.7.1/getting-started-docker.md | 5 +-
.../version-2.7.2/getting-started-docker.md | 5 +-
.../version-2.7.3/getting-started-docker.md | 5 +-
.../version-2.7.4/getting-started-docker.md | 5 +-
.../version-2.8.0/getting-started-docker.md | 5 +-
.../version-2.8.1/getting-started-docker.md | 5 +-
.../version-2.8.2/getting-started-docker.md | 5 +-
.../version-2.8.3/getting-started-docker.md | 5 +-
.../version-2.9.0/getting-started-docker.md | 5 +-
.../version-2.9.1/getting-started-docker.md | 5 +-
.../version-2.9.2/getting-started-docker.md | 5 +-
.../version-2.9.2/io-kafka-source.md | 178 ++++++++++----------
.../version-2.10.0/getting-started-docker.md | 4 +-
.../version-2.10.0/io-kafka-source.md | 152 +++++++++--------
.../version-2.3.1/getting-started-docker.md | 4 +-
.../version-2.3.2/getting-started-docker.md | 4 +-
.../version-2.5.0/getting-started-docker.md | 4 +-
.../version-2.5.1/standalone-docker.md | 4 +-
.../version-2.6.0/getting-started-docker.md | 4 +-
.../version-2.6.1/getting-started-docker.md | 4 +-
.../version-2.6.2/getting-started-docker.md | 4 +-
.../version-2.6.3/getting-started-docker.md | 4 +-
.../version-2.6.4/getting-started-docker.md | 4 +-
.../version-2.7.0/getting-started-docker.md | 4 +-
.../version-2.7.1/getting-started-docker.md | 4 +-
.../version-2.7.2/getting-started-docker.md | 4 +-
.../version-2.7.3/getting-started-docker.md | 4 +-
.../version-2.7.4/getting-started-docker.md | 4 +-
.../version-2.8.0/getting-started-docker.md | 4 +-
.../version-2.8.1/getting-started-docker.md | 4 +-
.../version-2.8.2/getting-started-docker.md | 4 +-
.../version-2.8.3/getting-started-docker.md | 4 +-
.../version-2.9.0/getting-started-docker.md | 4 +-
.../version-2.9.1/getting-started-docker.md | 4 +-
.../version-2.9.2/getting-started-docker.md | 4 +-
.../version-2.9.2/io-kafka-source.md | 150 +++++++++--------
59 files changed, 584 insertions(+), 649 deletions(-)
diff --git a/site2/docs/getting-started-docker.md b/site2/docs/getting-started-docker.md
index af654b2675b..20189057311 100644
--- a/site2/docs/getting-started-docker.md
+++ b/site2/docs/getting-started-docker.md
@@ -32,9 +32,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
> **Tip**
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/docs/io-kafka-source.md b/site2/docs/io-kafka-source.md
index 1bce28381d6..3b34b18127e 100644
--- a/site2/docs/io-kafka-source.md
+++ b/site2/docs/io-kafka-source.md
@@ -17,18 +17,18 @@ The configuration of the Kafka source connector has the following properties.
| Name | Type| Required | Default | Description
|------|----------|---------|-------------|-------------|
-| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
+| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
| `groupId` |String| true | " " (empty string) | A unique string that identifies the group of consumer processes to which this consumer belongs. |
| `fetchMinBytes` | long|false | 1 | The minimum byte expected for each fetch response. |
| `autoCommitEnabled` | boolean |false | true | If set to true, the consumer's offset is periodically committed in the background.<br/><br/> This committed offset is used when the process fails as the position from which a new consumer begins. |
| `autoCommitIntervalMs` | long|false | 5000 | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true. |
| `heartbeatIntervalMs` | long| false | 3000 | The interval between heartbeats to the consumer when using Kafka's group management facilities. <br/><br/>**Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.|
| `sessionTimeoutMs` | long|false | 30000 | The timeout used to detect consumer failures when using Kafka's group management facility. |
-| `topic` | String|true | " " (empty string)| The Kafka topic which sends messages to Pulsar. |
-| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br/><br/>**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
+| `topic` | String|true | " " (empty string)| The Kafka topic that sends messages to Pulsar. |
+| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br/><br/>**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringDeserializer | The deserializer class for Kafka consumers to deserialize keys.<br/> The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java).
| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values.
-| `autoOffsetReset` | String | false | "earliest" | The default offset reset policy. |
+| `autoOffsetReset` | String | false | earliest | The default offset reset policy. |
### Schema Management
@@ -64,83 +64,65 @@ If you want to access the raw key, you can use the `Message#getKeyBytes()` API.
Before using the Kafka source connector, you need to create a configuration file through one of the following methods.
-* JSON
+- JSON
- ```json
+ ```json
{
- "configs": {
- "bootstrapServers": "pulsar-kafka:9092",
- "groupId": "test-pulsar-io",
- "topic": "my-topic",
- "sessionTimeoutMs": "10000",
- "autoCommitEnabled": false
- }
+ "bootstrapServers": "pulsar-kafka:9092",
+ "groupId": "test-pulsar-io",
+ "topic": "my-topic",
+ "sessionTimeoutMs": "10000",
+ "autoCommitEnabled": false
}
- ```
+ ```
-* YAML
+- YAML
- ```yaml
+ ```yaml
configs:
- bootstrapServers: "pulsar-kafka:9092"
- groupId: "test-pulsar-io"
- topic: "my-topic"
- sessionTimeoutMs: "10000"
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
autoCommitEnabled: false
```
## Usage
-Here is an example of using the Kafka source connector with the configuration file as shown previously.
+You can make the Kafka source connector as a Pulsar built-in connector and use it on a standalone cluster or an on-premises cluster.
-1. Download a Kafka client and a Kafka connector.
+### Standalone cluster
- ```bash
- $ wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar
+This example describes how to use the Kafka source connector to feed data from Kafka and write data to Pulsar topics in the standalone mode.
- $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/connectors/pulsar-io-kafka-2.4.0.nar
- ```
+#### Prerequisites
-2. Create a network.
-
- ```bash
- $ docker network create kafka-pulsar
- ```
+- Install [Docker](https://docs.docker.com/get-docker/)(Community Edition).
-3. Pull a ZooKeeper image and start ZooKeeper.
-
- ```bash
- $ docker pull wurstmeister/zookeeper
+#### Steps
- $ docker run -d -it -p 2181:2181 --name pulsar-kafka-zookeeper --network kafka-pulsar wurstmeister/zookeeper
- ```
+1. Download and start the Confluent Platform.
-4. Pull a Kafka image and start Kafka.
-
- ```bash
- $ docker pull wurstmeister/kafka:2.11-1.0.2
-
- $ docker run -d -it --network kafka-pulsar -p 6667:6667 -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=pulsar-kafka -e KAFKA_ZOOKEEPER_CONNECT=pulsar-kafka-zookeeper:2181 --name pulsar-kafka wurstmeister/kafka:2.11-1.0.2
- ```
+For details, see the [documentation](https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#step-1-download-and-start-cp) to install the Kafka service locally.
-5. Pull a Pulsar image and start Pulsar standalone.
-
- ```bash
- $ docker pull apachepulsar/pulsar:{{pulsar:version}}
-
- $ docker run -d -it --network kafka-pulsar -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone
- ```
+2. Pull a Pulsar image and start Pulsar in standalone mode.
+
+ ```bash
+ docker pull apachepulsar/pulsar:latest
+
+ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:latest bin/pulsar standalone
+ ```
+
+3. Create a producer file _kafka-producer.py_.
-6. Create a producer file _kafka-producer.py_.
-
```python
from kafka import KafkaProducer
- producer = KafkaProducer(bootstrap_servers='pulsar-kafka:9092')
+ producer = KafkaProducer(bootstrap_servers='localhost:9092')
future = producer.send('my-topic', b'hello world')
future.get()
```
-7. Create a consumer file _pulsar-client.py_.
+4. Create a consumer file _pulsar-client.py_.
```python
import pulsar
@@ -158,23 +140,20 @@ Here is an example of using the Kafka source connector with the configuration fi
client.close()
```
-8. Copy the following files to Pulsar.
-
+5. Copy the following files to Pulsar.
+
```bash
- $ docker cp pulsar-io-kafka-{{pulsar:version}}.nar pulsar-kafka-standalone:/pulsar
- $ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
- $ docker cp pulsar-client.py pulsar-kafka-standalone:/pulsar/
- $ docker cp kafka-producer.py pulsar-kafka-standalone:/pulsar/
+ docker cp pulsar-io-kafka.nar pulsar-kafka-standalone:/pulsar
+ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
```
-9. Open a new terminal window and start the Kafka source connector in local run mode.
+6. Open a new terminal window and start the Kafka source connector in local run mode.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ docker exec -it pulsar-kafka-standalone /bin/bash
- $ ./bin/pulsar-admin source localrun \
- --archive ./pulsar-io-kafka-{{pulsar:version}}.nar \
- --classname org.apache.pulsar.io.kafka.KafkaBytesSource \
+ ./bin/pulsar-admin source localrun \
+ --archive ./pulsar-io-kafka.nar \
--tenant public \
--namespace default \
--name kafka \
@@ -183,18 +162,49 @@ Here is an example of using the Kafka source connector with the configuration fi
--parallelism 1
```
-10. Open a new terminal window and run the consumer.
+7. Open a new terminal window and run the Kafka producer locally.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ python3 kafka-producer.py
+ ```
- $ pip install kafka-python
+8. Open a new terminal window and run the Pulsar consumer locally.
- $ python3 kafka-producer.py
+ ```bash
+ python3 pulsar-client.py
```
- The following information appears on the consumer terminal window.
+The following information appears on the consumer terminal window.
```bash
Received message: 'hello world'
```
+
+### On-premises cluster
+
+This example explains how to create a Kafka source connector in an on-premises cluster.
+
+1. Copy the NAR package of the Kafka connector to the Pulsar connectors directory.
+
+ ```
+ cp pulsar-io-kafka-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-kafka-{{connector:version}}.nar
+ ```
+
+2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources reload
+ ```
+
+3. Check whether the Kafka source connector is available on the list or not.
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources available-sources
+ ```
+
+4. Create a Kafka source connector on a Pulsar cluster using the [`pulsar-admin sources create`](http://pulsar.apache.org/tools/pulsar-admin/2.11.0-SNAPSHOT/#-em-create-em--14) command.
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources create \
+ --source-config-file <kafka-source-config.yaml>
+ ```
\ No newline at end of file
diff --git a/site2/website-next/docs/getting-started-docker.md b/site2/website-next/docs/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/docs/getting-started-docker.md
+++ b/site2/website-next/docs/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/docs/io-kafka-source.md b/site2/website-next/docs/io-kafka-source.md
index befd0d85c6d..69185c8d75e 100644
--- a/site2/website-next/docs/io-kafka-source.md
+++ b/site2/website-next/docs/io-kafka-source.md
@@ -17,18 +17,18 @@ The configuration of the Kafka source connector has the following properties.
| Name | Type| Required | Default | Description
|------|----------|---------|-------------|-------------|
-| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
+| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
| `groupId` |String| true | " " (empty string) | A unique string that identifies the group of consumer processes to which this consumer belongs. |
| `fetchMinBytes` | long|false | 1 | The minimum byte expected for each fetch response. |
| `autoCommitEnabled` | boolean |false | true | If set to true, the consumer's offset is periodically committed in the background.<br /><br /> This committed offset is used when the process fails as the position from which a new consumer begins. |
| `autoCommitIntervalMs` | long|false | 5000 | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true. |
| `heartbeatIntervalMs` | long| false | 3000 | The interval between heartbeats to the consumer when using Kafka's group management facilities. <br /><br />**Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.|
| `sessionTimeoutMs` | long|false | 30000 | The timeout used to detect consumer failures when using Kafka's group management facility. |
-| `topic` | String|true | " " (empty string)| The Kafka topic which sends messages to Pulsar. |
-| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br /><br />**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
+| `topic` | String|true | " " (empty string)| The Kafka topic that sends messages to Pulsar. |
+| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br /><br />**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringDeserializer | The deserializer class for Kafka consumers to deserialize keys.<br /> The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java).
| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values.
-| `autoOffsetReset` | String | false | "earliest" | The default offset reset policy. |
+| `autoOffsetReset` | String | false | earliest | The default offset reset policy. |
### Schema Management
@@ -64,99 +64,73 @@ If you want to access the raw key, you can use the `Message#getKeyBytes()` API.
Before using the Kafka source connector, you need to create a configuration file through one of the following methods.
-* JSON
-
- ```json
-
- {
- "configs": {
- "bootstrapServers": "pulsar-kafka:9092",
- "groupId": "test-pulsar-io",
- "topic": "my-topic",
- "sessionTimeoutMs": "10000",
- "autoCommitEnabled": false
- }
- }
-
- ```
-
-* YAML
-
- ```yaml
-
- configs:
- bootstrapServers: "pulsar-kafka:9092"
- groupId: "test-pulsar-io"
- topic: "my-topic"
- sessionTimeoutMs: "10000"
- autoCommitEnabled: false
-
- ```
-
-## Usage
-
-Here is an example of using the Kafka source connector with the configuration file as shown previously.
+- JSON
-1. Download a Kafka client and a Kafka connector.
-
- ```bash
+ ```json
- $ wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar
-
- $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/connectors/pulsar-io-kafka-2.4.0.nar
+ {
+ "bootstrapServers": "pulsar-kafka:9092",
+ "groupId": "test-pulsar-io",
+ "topic": "my-topic",
+ "sessionTimeoutMs": "10000",
+ "autoCommitEnabled": false
+ }
```
-2. Create a network.
+- YAML
- ```bash
+ ```yaml
- $ docker network create kafka-pulsar
+ configs:
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
+ autoCommitEnabled: false
```
-3. Pull a ZooKeeper image and start ZooKeeper.
+## Usage
- ```bash
-
- $ docker pull wurstmeister/zookeeper
+You can make the Kafka source connector as a Pulsar built-in connector and use it on a standalone cluster or an on-premises cluster.
- $ docker run -d -it -p 2181:2181 --name pulsar-kafka-zookeeper --network kafka-pulsar wurstmeister/zookeeper
-
- ```
+### Standalone cluster
-4. Pull a Kafka image and start Kafka.
+This example describes how to use the Kafka source connector to feed data from Kafka and write data to Pulsar topics in the standalone mode.
- ```bash
-
- $ docker pull wurstmeister/kafka:2.11-1.0.2
-
- $ docker run -d -it --network kafka-pulsar -p 6667:6667 -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=pulsar-kafka -e KAFKA_ZOOKEEPER_CONNECT=pulsar-kafka-zookeeper:2181 --name pulsar-kafka wurstmeister/kafka:2.11-1.0.2
-
- ```
+#### Prerequisites
+
+- Install [Docker](https://docs.docker.com/get-docker/)(Community Edition).
-5. Pull a Pulsar image and start Pulsar standalone.
+#### Steps
+
+1. Download and start the Confluent Platform.
+
+For details, see the [documentation](https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#step-1-download-and-start-cp) to install the Kafka service locally.
+
+2. Pull a Pulsar image and start Pulsar in standalone mode.
```bash
- $ docker pull apachepulsar/pulsar:@pulsar:version@
-
- $ docker run -d -it --network kafka-pulsar -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone
+ docker pull apachepulsar/pulsar:latest
+
+ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:latest bin/pulsar standalone
```
-6. Create a producer file _kafka-producer.py_.
+3. Create a producer file _kafka-producer.py_.
```python
from kafka import KafkaProducer
- producer = KafkaProducer(bootstrap_servers='pulsar-kafka:9092')
+ producer = KafkaProducer(bootstrap_servers='localhost:9092')
future = producer.send('my-topic', b'hello world')
future.get()
```
-7. Create a consumer file _pulsar-client.py_.
+4. Create a consumer file _pulsar-client.py_.
```python
@@ -176,26 +150,23 @@ Here is an example of using the Kafka source connector with the configuration fi
```
-8. Copy the following files to Pulsar.
+5. Copy the following files to Pulsar.
```bash
- $ docker cp pulsar-io-kafka-@pulsar:version@.nar pulsar-kafka-standalone:/pulsar
- $ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
- $ docker cp pulsar-client.py pulsar-kafka-standalone:/pulsar/
- $ docker cp kafka-producer.py pulsar-kafka-standalone:/pulsar/
+ docker cp pulsar-io-kafka.nar pulsar-kafka-standalone:/pulsar
+ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
```
-9. Open a new terminal window and start the Kafka source connector in local run mode.
+6. Open a new terminal window and start the Kafka source connector in local run mode.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ docker exec -it pulsar-kafka-standalone /bin/bash
- $ ./bin/pulsar-admin source localrun \
- --archive ./pulsar-io-kafka-@pulsar:version@.nar \
- --classname org.apache.pulsar.io.kafka.KafkaBytesSource \
+ ./bin/pulsar-admin source localrun \
+ --archive ./pulsar-io-kafka.nar \
--tenant public \
--namespace default \
--name kafka \
@@ -205,23 +176,64 @@ Here is an example of using the Kafka source connector with the configuration fi
```
-10. Open a new terminal window and run the consumer.
+7. Open a new terminal window and run the Kafka producer locally.
+
+ ```bash
+
+ python3 kafka-producer.py
+
+ ```
+
+8. Open a new terminal window and run the Pulsar consumer locally.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ python3 pulsar-client.py
+
+ ```
+
+The following information appears on the consumer terminal window.
- $ pip install kafka-python
+ ```bash
+
+ Received message: 'hello world'
+
+ ```
- $ python3 kafka-producer.py
+### On-premises cluster
+
+This example explains how to create a Kafka source connector in an on-premises cluster.
+
+1. Copy the NAR package of the Kafka connector to the Pulsar connectors directory.
+
+ ```
+
+ cp pulsar-io-kafka-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-kafka-{{connector:version}}.nar
```
- The following information appears on the consumer terminal window.
+2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
- ```bash
+ ```
+
+ PULSAR_HOME/bin/pulsar-admin sources reload
+
+ ```
+
+3. Check whether the Kafka source connector is available on the list or not.
+
+ ```
+
+ PULSAR_HOME/bin/pulsar-admin sources available-sources
+
+ ```
+
+4. Create a Kafka source connector on a Pulsar cluster using the [`pulsar-admin sources create`](http://pulsar.apache.org/tools/pulsar-admin/2.11.0-SNAPSHOT/#-em-create-em--14) command.
+
+ ```
- Received message: 'hello world'
+ PULSAR_HOME/bin/pulsar-admin sources create \
+ --source-config-file <kafka-source-config.yaml>
```
diff --git a/site2/website-next/versioned_docs/version-2.1.1-incubating/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.1.1-incubating/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/versioned_docs/version-2.1.1-incubating/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.1.1-incubating/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.10.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.10.0/getting-started-docker.md
index 10f9c7c6c8c..3719463d23b 100644
--- a/site2/website-next/versioned_docs/version-2.10.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.10.0/getting-started-docker.md
@@ -38,13 +38,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.10.0/io-kafka-source.md b/site2/website-next/versioned_docs/version-2.10.0/io-kafka-source.md
index d50001755a2..b9a6ab07ad8 100644
--- a/site2/website-next/versioned_docs/version-2.10.0/io-kafka-source.md
+++ b/site2/website-next/versioned_docs/version-2.10.0/io-kafka-source.md
@@ -18,18 +18,18 @@ The configuration of the Kafka source connector has the following properties.
| Name | Type| Required | Default | Description
|------|----------|---------|-------------|-------------|
-| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
+| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
| `groupId` |String| true | " " (empty string) | A unique string that identifies the group of consumer processes to which this consumer belongs. |
| `fetchMinBytes` | long|false | 1 | The minimum byte expected for each fetch response. |
| `autoCommitEnabled` | boolean |false | true | If set to true, the consumer's offset is periodically committed in the background.<br /><br /> This committed offset is used when the process fails as the position from which a new consumer begins. |
| `autoCommitIntervalMs` | long|false | 5000 | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true. |
| `heartbeatIntervalMs` | long| false | 3000 | The interval between heartbeats to the consumer when using Kafka's group management facilities. <br /><br />**Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.|
| `sessionTimeoutMs` | long|false | 30000 | The timeout used to detect consumer failures when using Kafka's group management facility. |
-| `topic` | String|true | " " (empty string)| The Kafka topic which sends messages to Pulsar. |
-| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br /><br />**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
+| `topic` | String|true | " " (empty string)| The Kafka topic that sends messages to Pulsar. |
+| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br /><br />**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringDeserializer | The deserializer class for Kafka consumers to deserialize keys.<br /> The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java).
| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values.
-| `autoOffsetReset` | String | false | "earliest" | The default offset reset policy. |
+| `autoOffsetReset` | String | false | earliest | The default offset reset policy. |
### Schema Management
@@ -65,99 +65,73 @@ If you want to access the raw key, you can use the `Message#getKeyBytes()` API.
Before using the Kafka source connector, you need to create a configuration file through one of the following methods.
-* JSON
-
- ```json
-
- {
- "configs": {
- "bootstrapServers": "pulsar-kafka:9092",
- "groupId": "test-pulsar-io",
- "topic": "my-topic",
- "sessionTimeoutMs": "10000",
- "autoCommitEnabled": false
- }
- }
-
- ```
-
-* YAML
-
- ```yaml
-
- configs:
- bootstrapServers: "pulsar-kafka:9092"
- groupId: "test-pulsar-io"
- topic: "my-topic"
- sessionTimeoutMs: "10000"
- autoCommitEnabled: false
-
- ```
-
-## Usage
-
-Here is an example of using the Kafka source connector with the configuration file as shown previously.
+- JSON
-1. Download a Kafka client and a Kafka connector.
-
- ```bash
+ ```json
- $ wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar
-
- $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/connectors/pulsar-io-kafka-2.4.0.nar
+ {
+ "bootstrapServers": "pulsar-kafka:9092",
+ "groupId": "test-pulsar-io",
+ "topic": "my-topic",
+ "sessionTimeoutMs": "10000",
+ "autoCommitEnabled": false
+ }
```
-2. Create a network.
+- YAML
- ```bash
+ ```yaml
- $ docker network create kafka-pulsar
+ configs:
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
+ autoCommitEnabled: false
```
-3. Pull a ZooKeeper image and start ZooKeeper.
+## Usage
- ```bash
-
- $ docker pull wurstmeister/zookeeper
+You can make the Kafka source connector as a Pulsar built-in connector and use it on a standalone cluster or an on-premises cluster.
- $ docker run -d -it -p 2181:2181 --name pulsar-kafka-zookeeper --network kafka-pulsar wurstmeister/zookeeper
-
- ```
+### Standalone cluster
-4. Pull a Kafka image and start Kafka.
+This example describes how to use the Kafka source connector to feed data from Kafka and write data to Pulsar topics in the standalone mode.
- ```bash
-
- $ docker pull wurstmeister/kafka:2.11-1.0.2
-
- $ docker run -d -it --network kafka-pulsar -p 6667:6667 -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=pulsar-kafka -e KAFKA_ZOOKEEPER_CONNECT=pulsar-kafka-zookeeper:2181 --name pulsar-kafka wurstmeister/kafka:2.11-1.0.2
-
- ```
+#### Prerequisites
+
+- Install [Docker](https://docs.docker.com/get-docker/)(Community Edition).
-5. Pull a Pulsar image and start Pulsar standalone.
+#### Steps
+
+1. Download and start the Confluent Platform.
+
+For details, see the [documentation](https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#step-1-download-and-start-cp) to install the Kafka service locally.
+
+2. Pull a Pulsar image and start Pulsar in standalone mode.
```bash
- $ docker pull apachepulsar/pulsar:@pulsar:version@
-
- $ docker run -d -it --network kafka-pulsar -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone
+ docker pull apachepulsar/pulsar:latest
+
+ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:latest bin/pulsar standalone
```
-6. Create a producer file _kafka-producer.py_.
+3. Create a producer file _kafka-producer.py_.
```python
from kafka import KafkaProducer
- producer = KafkaProducer(bootstrap_servers='pulsar-kafka:9092')
+ producer = KafkaProducer(bootstrap_servers='localhost:9092')
future = producer.send('my-topic', b'hello world')
future.get()
```
-7. Create a consumer file _pulsar-client.py_.
+4. Create a consumer file _pulsar-client.py_.
```python
@@ -177,26 +151,23 @@ Here is an example of using the Kafka source connector with the configuration fi
```
-8. Copy the following files to Pulsar.
+5. Copy the following files to Pulsar.
```bash
- $ docker cp pulsar-io-kafka-@pulsar:version@.nar pulsar-kafka-standalone:/pulsar
- $ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
- $ docker cp pulsar-client.py pulsar-kafka-standalone:/pulsar/
- $ docker cp kafka-producer.py pulsar-kafka-standalone:/pulsar/
+ docker cp pulsar-io-kafka.nar pulsar-kafka-standalone:/pulsar
+ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
```
-9. Open a new terminal window and start the Kafka source connector in local run mode.
+6. Open a new terminal window and start the Kafka source connector in local run mode.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ docker exec -it pulsar-kafka-standalone /bin/bash
- $ ./bin/pulsar-admin source localrun \
- --archive ./pulsar-io-kafka-@pulsar:version@.nar \
- --classname org.apache.pulsar.io.kafka.KafkaBytesSource \
+ ./bin/pulsar-admin source localrun \
+ --archive ./pulsar-io-kafka.nar \
--tenant public \
--namespace default \
--name kafka \
@@ -206,23 +177,64 @@ Here is an example of using the Kafka source connector with the configuration fi
```
-10. Open a new terminal window and run the consumer.
+7. Open a new terminal window and run the Kafka producer locally.
+
+ ```bash
+
+ python3 kafka-producer.py
+
+ ```
+
+8. Open a new terminal window and run the Pulsar consumer locally.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ python3 pulsar-client.py
+
+ ```
+
+The following information appears on the consumer terminal window.
- $ pip install kafka-python
+ ```bash
+
+ Received message: 'hello world'
+
+ ```
- $ python3 kafka-producer.py
+### On-premises cluster
+
+This example explains how to create a Kafka source connector in an on-premises cluster.
+
+1. Copy the NAR package of the Kafka connector to the Pulsar connectors directory.
+
+ ```
+
+ cp pulsar-io-kafka-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-kafka-{{connector:version}}.nar
```
- The following information appears on the consumer terminal window.
+2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
- ```bash
+ ```
+
+ PULSAR_HOME/bin/pulsar-admin sources reload
+
+ ```
+
+3. Check whether the Kafka source connector is available on the list or not.
+
+ ```
+
+ PULSAR_HOME/bin/pulsar-admin sources available-sources
+
+ ```
+
+4. Create a Kafka source connector on a Pulsar cluster using the [`pulsar-admin sources create`](http://pulsar.apache.org/tools/pulsar-admin/2.10.0-SNAPSHOT/#-em-create-em--14) command.
+
+ ```
- Received message: 'hello world'
+ PULSAR_HOME/bin/pulsar-admin sources create \
+ --source-config-file <kafka-source-config.yaml>
```
diff --git a/site2/website-next/versioned_docs/version-2.2.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.2.0/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/versioned_docs/version-2.2.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.2.0/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.3.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.3.0/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/versioned_docs/version-2.3.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.3.0/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.3.1/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.3.1/getting-started-docker.md
index 01c81b10b5f..47b0b7876a8 100644
--- a/site2/website-next/versioned_docs/version-2.3.1/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.3.1/getting-started-docker.md
@@ -53,9 +53,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
> #### Automatically created namespace
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Start publishing and consuming messages
diff --git a/site2/website-next/versioned_docs/version-2.3.2/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.3.2/getting-started-docker.md
index 0b5a2fefd5b..8387c074c00 100644
--- a/site2/website-next/versioned_docs/version-2.3.2/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.3.2/getting-started-docker.md
@@ -56,13 +56,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.4.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.4.0/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.4.1/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.4.1/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.4.2/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.4.2/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.5.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.5.0/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.5.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.5.0/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.5.1/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.5.1/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.5.1/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.5.1/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.5.2/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.5.2/getting-started-docker.md
index cea482efe6f..c0723a61f25 100644
--- a/site2/website-next/versioned_docs/version-2.5.2/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.5.2/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.6.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.6.0/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.6.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.6.0/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.6.1/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.6.1/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.6.1/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.6.1/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.6.2/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.6.2/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.6.2/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.6.2/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.6.3/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.6.3/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.6.3/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.6.3/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.6.4/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.6.4/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.6.4/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.6.4/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.7.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.7.0/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.7.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.7.0/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.7.1/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.7.1/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.7.1/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.7.1/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.7.2/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.7.2/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.7.2/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.7.2/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.7.3/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.7.3/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.7.3/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.7.3/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.7.4/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.7.4/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.7.4/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.7.4/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.8.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.8.0/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.8.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.8.0/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.8.1/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.8.1/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.8.1/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.8.1/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.8.2/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.8.2/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.8.2/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.8.2/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.8.3/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.8.3/getting-started-docker.md
index 02ca2e9eb8f..0219b58f5c8 100644
--- a/site2/website-next/versioned_docs/version-2.8.3/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.8.3/getting-started-docker.md
@@ -44,13 +44,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.9.0/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.9.0/getting-started-docker.md
index d6c6bd7b1ff..26fe54abe74 100644
--- a/site2/website-next/versioned_docs/version-2.9.0/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.9.0/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.9.1/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.9.1/getting-started-docker.md
index d6c6bd7b1ff..26fe54abe74 100644
--- a/site2/website-next/versioned_docs/version-2.9.1/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.9.1/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.9.2/getting-started-docker.md b/site2/website-next/versioned_docs/version-2.9.2/getting-started-docker.md
index d6c6bd7b1ff..26fe54abe74 100644
--- a/site2/website-next/versioned_docs/version-2.9.2/getting-started-docker.md
+++ b/site2/website-next/versioned_docs/version-2.9.2/getting-started-docker.md
@@ -37,13 +37,10 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
:::tip
-When you start a local standalone cluster, a `public/default`
+When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
:::
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
-
## Use Pulsar in Docker
Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python)
diff --git a/site2/website-next/versioned_docs/version-2.9.2/io-kafka-source.md b/site2/website-next/versioned_docs/version-2.9.2/io-kafka-source.md
index 53448699e21..35b04c52b41 100644
--- a/site2/website-next/versioned_docs/version-2.9.2/io-kafka-source.md
+++ b/site2/website-next/versioned_docs/version-2.9.2/io-kafka-source.md
@@ -18,18 +18,18 @@ The configuration of the Kafka source connector has the following properties.
| Name | Type| Required | Default | Description
|------|----------|---------|-------------|-------------|
-| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
+| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
| `groupId` |String| true | " " (empty string) | A unique string that identifies the group of consumer processes to which this consumer belongs. |
| `fetchMinBytes` | long|false | 1 | The minimum byte expected for each fetch response. |
| `autoCommitEnabled` | boolean |false | true | If set to true, the consumer's offset is periodically committed in the background.<br /><br /> This committed offset is used when the process fails as the position from which a new consumer begins. |
| `autoCommitIntervalMs` | long|false | 5000 | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true. |
| `heartbeatIntervalMs` | long| false | 3000 | The interval between heartbeats to the consumer when using Kafka's group management facilities. <br /><br />**Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.|
| `sessionTimeoutMs` | long|false | 30000 | The timeout used to detect consumer failures when using Kafka's group management facility. |
-| `topic` | String|true | " " (empty string)| The Kafka topic which sends messages to Pulsar. |
-| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br /><br />**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
+| `topic` | String|true | " " (empty string)| The Kafka topic that sends messages to Pulsar. |
+| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br /><br />**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringDeserializer | The deserializer class for Kafka consumers to deserialize keys.<br /> The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java).
| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values.
-| `autoOffsetReset` | String | false | "earliest" | The default offset reset policy. |
+| `autoOffsetReset` | String | false | earliest | The default offset reset policy. |
### Schema Management
@@ -65,97 +65,73 @@ If you want to access the raw key, you can use the `Message#getKeyBytes()` API.
Before using the Kafka source connector, you need to create a configuration file through one of the following methods.
-* JSON
-
- ```json
-
- {
- "bootstrapServers": "pulsar-kafka:9092",
- "groupId": "test-pulsar-io",
- "topic": "my-topic",
- "sessionTimeoutMs": "10000",
- "autoCommitEnabled": false
- }
-
- ```
-
-* YAML
-
- ```yaml
-
- configs:
- bootstrapServers: "pulsar-kafka:9092"
- groupId: "test-pulsar-io"
- topic: "my-topic"
- sessionTimeoutMs: "10000"
- autoCommitEnabled: false
-
- ```
+- JSON
-## Usage
-
-Here is an example of using the Kafka source connector with the configuration file as shown previously.
-
-1. Download a Kafka client and a Kafka connector.
-
- ```bash
+ ```json
- $ wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar
-
- $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/connectors/pulsar-io-kafka-2.4.0.nar
+ {
+ "bootstrapServers": "pulsar-kafka:9092",
+ "groupId": "test-pulsar-io",
+ "topic": "my-topic",
+ "sessionTimeoutMs": "10000",
+ "autoCommitEnabled": false
+ }
```
-2. Create a network.
+- YAML
- ```bash
+ ```yaml
- $ docker network create kafka-pulsar
+ configs:
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
+ autoCommitEnabled: false
```
-3. Pull a ZooKeeper image and start ZooKeeper.
+## Usage
- ```bash
-
- $ docker pull wurstmeister/zookeeper
+You can make the Kafka source connector as a Pulsar built-in connector and use it on a standalone cluster or an on-premises cluster.
- $ docker run -d -it -p 2181:2181 --name pulsar-kafka-zookeeper --network kafka-pulsar wurstmeister/zookeeper
-
- ```
+### Standalone cluster
-4. Pull a Kafka image and start Kafka.
+This example describes how to use the Kafka source connector to feed data from Kafka and write data to Pulsar topics in the standalone mode.
- ```bash
-
- $ docker pull wurstmeister/kafka:2.11-1.0.2
-
- $ docker run -d -it --network kafka-pulsar -p 6667:6667 -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=pulsar-kafka -e KAFKA_ZOOKEEPER_CONNECT=pulsar-kafka-zookeeper:2181 --name pulsar-kafka wurstmeister/kafka:2.11-1.0.2
-
- ```
+#### Prerequisites
+
+- Install [Docker](https://docs.docker.com/get-docker/)(Community Edition).
+
+#### Steps
-5. Pull a Pulsar image and start Pulsar standalone.
+1. Download and start the Confluent Platform.
+
+For details, see the [documentation](https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#step-1-download-and-start-cp) to install the Kafka service locally.
+
+2. Pull a Pulsar image and start Pulsar in standalone mode.
```bash
- $ docker pull apachepulsar/pulsar:@pulsar:version@
-
- $ docker run -d -it --network kafka-pulsar -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone
+ docker pull apachepulsar/pulsar:latest
+
+ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:latest bin/pulsar standalone
```
-6. Create a producer file _kafka-producer.py_.
+3. Create a producer file _kafka-producer.py_.
```python
from kafka import KafkaProducer
- producer = KafkaProducer(bootstrap_servers='pulsar-kafka:9092')
+ producer = KafkaProducer(bootstrap_servers='localhost:9092')
future = producer.send('my-topic', b'hello world')
future.get()
```
-7. Create a consumer file _pulsar-client.py_.
+4. Create a consumer file _pulsar-client.py_.
```python
@@ -175,26 +151,23 @@ Here is an example of using the Kafka source connector with the configuration fi
```
-8. Copy the following files to Pulsar.
+5. Copy the following files to Pulsar.
```bash
- $ docker cp pulsar-io-kafka-@pulsar:version@.nar pulsar-kafka-standalone:/pulsar
- $ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
- $ docker cp pulsar-client.py pulsar-kafka-standalone:/pulsar/
- $ docker cp kafka-producer.py pulsar-kafka-standalone:/pulsar/
+ docker cp pulsar-io-kafka.nar pulsar-kafka-standalone:/pulsar
+ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
```
-9. Open a new terminal window and start the Kafka source connector in local run mode.
+6. Open a new terminal window and start the Kafka source connector in local run mode.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ docker exec -it pulsar-kafka-standalone /bin/bash
- $ ./bin/pulsar-admin source localrun \
- --archive ./pulsar-io-kafka-@pulsar:version@.nar \
- --classname org.apache.pulsar.io.kafka.KafkaBytesSource \
+ ./bin/pulsar-admin source localrun \
+ --archive ./pulsar-io-kafka.nar \
--tenant public \
--namespace default \
--name kafka \
@@ -204,23 +177,64 @@ Here is an example of using the Kafka source connector with the configuration fi
```
-10. Open a new terminal window and run the consumer.
+7. Open a new terminal window and run the Kafka producer locally.
+
+ ```bash
+
+ python3 kafka-producer.py
+
+ ```
+
+8. Open a new terminal window and run the Pulsar consumer locally.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ python3 pulsar-client.py
+
+ ```
+
+The following information appears on the consumer terminal window.
+
+ ```bash
+
+ Received message: 'hello world'
+
+ ```
- $ pip install kafka-python
+### On-premises cluster
- $ python3 kafka-producer.py
+This example explains how to create a Kafka source connector in an on-premises cluster.
+
+1. Copy the NAR package of the Kafka connector to the Pulsar connectors directory.
+
+ ```
+
+ cp pulsar-io-kafka-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-kafka-{{connector:version}}.nar
```
- The following information appears on the consumer terminal window.
+2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
- ```bash
+ ```
+
+ PULSAR_HOME/bin/pulsar-admin sources reload
+
+ ```
+
+3. Check whether the Kafka source connector is available on the list or not.
+
+ ```
+
+ PULSAR_HOME/bin/pulsar-admin sources available-sources
+
+ ```
+
+4. Create a Kafka source connector on a Pulsar cluster using the [`pulsar-admin sources create`](http://pulsar.apache.org/tools/pulsar-admin/2.9.0-SNAPSHOT/#-em-create-em--14) command.
+
+ ```
- Received message: 'hello world'
+ PULSAR_HOME/bin/pulsar-admin sources create \
+ --source-config-file <kafka-source-config.yaml>
```
diff --git a/site2/website/versioned_docs/version-2.10.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.10.0/getting-started-docker.md
index 390a1118e4b..2f614957188 100644
--- a/site2/website/versioned_docs/version-2.10.0/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.10.0/getting-started-docker.md
@@ -33,9 +33,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
> **Tip**
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.10.0/io-kafka-source.md b/site2/website/versioned_docs/version-2.10.0/io-kafka-source.md
index 21611696648..d0d779786f0 100644
--- a/site2/website/versioned_docs/version-2.10.0/io-kafka-source.md
+++ b/site2/website/versioned_docs/version-2.10.0/io-kafka-source.md
@@ -18,18 +18,18 @@ The configuration of the Kafka source connector has the following properties.
| Name | Type| Required | Default | Description
|------|----------|---------|-------------|-------------|
-| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
+| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
| `groupId` |String| true | " " (empty string) | A unique string that identifies the group of consumer processes to which this consumer belongs. |
| `fetchMinBytes` | long|false | 1 | The minimum byte expected for each fetch response. |
| `autoCommitEnabled` | boolean |false | true | If set to true, the consumer's offset is periodically committed in the background.<br/><br/> This committed offset is used when the process fails as the position from which a new consumer begins. |
| `autoCommitIntervalMs` | long|false | 5000 | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true. |
| `heartbeatIntervalMs` | long| false | 3000 | The interval between heartbeats to the consumer when using Kafka's group management facilities. <br/><br/>**Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.|
| `sessionTimeoutMs` | long|false | 30000 | The timeout used to detect consumer failures when using Kafka's group management facility. |
-| `topic` | String|true | " " (empty string)| The Kafka topic which sends messages to Pulsar. |
-| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br/><br/>**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
+| `topic` | String|true | " " (empty string)| The Kafka topic that sends messages to Pulsar. |
+| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br/><br/>**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringDeserializer | The deserializer class for Kafka consumers to deserialize keys.<br/> The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java).
| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values.
-| `autoOffsetReset` | String | false | "earliest" | The default offset reset policy. |
+| `autoOffsetReset` | String | false | earliest | The default offset reset policy. |
### Schema Management
@@ -65,83 +65,65 @@ If you want to access the raw key, you can use the `Message#getKeyBytes()` API.
Before using the Kafka source connector, you need to create a configuration file through one of the following methods.
-* JSON
+- JSON
- ```json
+ ```json
{
- "configs": {
- "bootstrapServers": "pulsar-kafka:9092",
- "groupId": "test-pulsar-io",
- "topic": "my-topic",
- "sessionTimeoutMs": "10000",
- "autoCommitEnabled": false
- }
+ "bootstrapServers": "pulsar-kafka:9092",
+ "groupId": "test-pulsar-io",
+ "topic": "my-topic",
+ "sessionTimeoutMs": "10000",
+ "autoCommitEnabled": false
}
- ```
+ ```
-* YAML
+- YAML
- ```yaml
+ ```yaml
configs:
- bootstrapServers: "pulsar-kafka:9092"
- groupId: "test-pulsar-io"
- topic: "my-topic"
- sessionTimeoutMs: "10000"
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
autoCommitEnabled: false
```
## Usage
-Here is an example of using the Kafka source connector with the configuration file as shown previously.
+You can make the Kafka source connector as a Pulsar built-in connector and use it on a standalone cluster or an on-premises cluster.
-1. Download a Kafka client and a Kafka connector.
+### Standalone cluster
- ```bash
- $ wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar
+This example describes how to use the Kafka source connector to feed data from Kafka and write data to Pulsar topics in the standalone mode.
- $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/connectors/pulsar-io-kafka-2.4.0.nar
- ```
+#### Prerequisites
-2. Create a network.
-
- ```bash
- $ docker network create kafka-pulsar
- ```
+- Install [Docker](https://docs.docker.com/get-docker/)(Community Edition).
-3. Pull a ZooKeeper image and start ZooKeeper.
-
- ```bash
- $ docker pull wurstmeister/zookeeper
+#### Steps
- $ docker run -d -it -p 2181:2181 --name pulsar-kafka-zookeeper --network kafka-pulsar wurstmeister/zookeeper
- ```
+1. Download and start the Confluent Platform.
-4. Pull a Kafka image and start Kafka.
-
- ```bash
- $ docker pull wurstmeister/kafka:2.11-1.0.2
-
- $ docker run -d -it --network kafka-pulsar -p 6667:6667 -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=pulsar-kafka -e KAFKA_ZOOKEEPER_CONNECT=pulsar-kafka-zookeeper:2181 --name pulsar-kafka wurstmeister/kafka:2.11-1.0.2
- ```
+For details, see the [documentation](https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#step-1-download-and-start-cp) to install the Kafka service locally.
-5. Pull a Pulsar image and start Pulsar standalone.
-
- ```bash
- $ docker pull apachepulsar/pulsar:{{pulsar:version}}
-
- $ docker run -d -it --network kafka-pulsar -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone
- ```
+2. Pull a Pulsar image and start Pulsar in standalone mode.
+
+ ```bash
+ docker pull apachepulsar/pulsar:latest
+
+ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:latest bin/pulsar standalone
+ ```
+
+3. Create a producer file _kafka-producer.py_.
-6. Create a producer file _kafka-producer.py_.
-
```python
from kafka import KafkaProducer
- producer = KafkaProducer(bootstrap_servers='pulsar-kafka:9092')
+ producer = KafkaProducer(bootstrap_servers='localhost:9092')
future = producer.send('my-topic', b'hello world')
future.get()
```
-7. Create a consumer file _pulsar-client.py_.
+4. Create a consumer file _pulsar-client.py_.
```python
import pulsar
@@ -159,23 +141,20 @@ Here is an example of using the Kafka source connector with the configuration fi
client.close()
```
-8. Copy the following files to Pulsar.
-
+5. Copy the following files to Pulsar.
+
```bash
- $ docker cp pulsar-io-kafka-{{pulsar:version}}.nar pulsar-kafka-standalone:/pulsar
- $ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
- $ docker cp pulsar-client.py pulsar-kafka-standalone:/pulsar/
- $ docker cp kafka-producer.py pulsar-kafka-standalone:/pulsar/
+ docker cp pulsar-io-kafka.nar pulsar-kafka-standalone:/pulsar
+ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
```
-9. Open a new terminal window and start the Kafka source connector in local run mode.
+6. Open a new terminal window and start the Kafka source connector in local run mode.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ docker exec -it pulsar-kafka-standalone /bin/bash
- $ ./bin/pulsar-admin source localrun \
- --archive ./pulsar-io-kafka-{{pulsar:version}}.nar \
- --classname org.apache.pulsar.io.kafka.KafkaBytesSource \
+ ./bin/pulsar-admin source localrun \
+ --archive ./pulsar-io-kafka.nar \
--tenant public \
--namespace default \
--name kafka \
@@ -184,18 +163,49 @@ Here is an example of using the Kafka source connector with the configuration fi
--parallelism 1
```
-10. Open a new terminal window and run the consumer.
+7. Open a new terminal window and run the Kafka producer locally.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ python3 kafka-producer.py
+ ```
- $ pip install kafka-python
+8. Open a new terminal window and run the Pulsar consumer locally.
- $ python3 kafka-producer.py
+ ```bash
+ python3 pulsar-client.py
```
- The following information appears on the consumer terminal window.
+The following information appears on the consumer terminal window.
```bash
Received message: 'hello world'
```
+
+### On-premises cluster
+
+This example explains how to create a Kafka source connector in an on-premises cluster.
+
+1. Copy the NAR package of the Kafka connector to the Pulsar connectors directory.
+
+ ```
+ cp pulsar-io-kafka-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-kafka-{{connector:version}}.nar
+ ```
+
+2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources reload
+ ```
+
+3. Check whether the Kafka source connector is available on the list or not.
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources available-sources
+ ```
+
+4. Create a Kafka source connector on a Pulsar cluster using the [`pulsar-admin sources create`](http://pulsar.apache.org/tools/pulsar-admin/2.10.0-SNAPSHOT/#-em-create-em--14) command.
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources create \
+ --source-config-file <kafka-source-config.yaml>
+ ```
\ No newline at end of file
diff --git a/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md
index 6409d05eb73..deaef81d321 100644
--- a/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md
@@ -47,9 +47,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
> #### Automatically created namespace
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Start publishing and consuming messages
diff --git a/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md
index 0f88a8f115a..02f1480489c 100644
--- a/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md
@@ -50,9 +50,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md
index 7c2f91e37dd..68f66a5e429 100644
--- a/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.5.1/standalone-docker.md b/site2/website/versioned_docs/version-2.5.1/standalone-docker.md
index ce16776faf5..2e715109f9f 100644
--- a/site2/website/versioned_docs/version-2.5.1/standalone-docker.md
+++ b/site2/website/versioned_docs/version-2.5.1/standalone-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md
index 9cb280c423c..6c652faca0f 100644
--- a/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md
index e6142a774fa..63e6cfabbdd 100644
--- a/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md
index b3ea394af4d..e4f71b82e93 100644
--- a/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md
index eaa28642675..7414bc30614 100644
--- a/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md
index f9b2e99087a..b0ed4e487f1 100644
--- a/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md
index 7fecb696527..d9bb600ea7d 100644
--- a/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md
index 66c07f3eb25..a2b66468305 100644
--- a/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md
index 03ad2c21932..6767c4431b5 100644
--- a/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md
index 69b3b913735..affc4ea8e7c 100644
--- a/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md
index 982aab5b82c..3300ad52562 100644
--- a/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.8.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.0/getting-started-docker.md
index 68939e5f85f..24d7791f915 100644
--- a/site2/website/versioned_docs/version-2.8.0/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.8.0/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.8.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.1/getting-started-docker.md
index d5bbb04e98d..7da8f96387f 100644
--- a/site2/website/versioned_docs/version-2.8.1/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.8.1/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.8.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.2/getting-started-docker.md
index 45b7ddd2d2a..5a2a18c7e77 100644
--- a/site2/website/versioned_docs/version-2.8.2/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.8.2/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.8.3/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.3/getting-started-docker.md
index 5a1aaaaa274..d59299c767e 100644
--- a/site2/website/versioned_docs/version-2.8.3/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.8.3/getting-started-docker.md
@@ -40,9 +40,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
> #### Tip
>
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.9.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.9.0/getting-started-docker.md
index 87e2a89587f..49dc73b462f 100644
--- a/site2/website/versioned_docs/version-2.9.0/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.9.0/getting-started-docker.md
@@ -32,9 +32,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
> **Tip**
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.9.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.9.1/getting-started-docker.md
index b71405e93fd..3f44b1a2760 100644
--- a/site2/website/versioned_docs/version-2.9.1/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.9.1/getting-started-docker.md
@@ -32,9 +32,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
> **Tip**
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.9.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.9.2/getting-started-docker.md
index eca448b76e7..9c459977709 100644
--- a/site2/website/versioned_docs/version-2.9.2/getting-started-docker.md
+++ b/site2/website/versioned_docs/version-2.9.2/getting-started-docker.md
@@ -32,9 +32,7 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
> **Tip**
-> When you start a local standalone cluster, a `public/default`
-namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces.
-For more information, see [Topics](concepts-messaging.md#topics).
+> When you start a local standalone cluster, a `public/default` namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics).
## Use Pulsar in Docker
diff --git a/site2/website/versioned_docs/version-2.9.2/io-kafka-source.md b/site2/website/versioned_docs/version-2.9.2/io-kafka-source.md
index 71122fdf930..926576d5aa7 100644
--- a/site2/website/versioned_docs/version-2.9.2/io-kafka-source.md
+++ b/site2/website/versioned_docs/version-2.9.2/io-kafka-source.md
@@ -18,18 +18,18 @@ The configuration of the Kafka source connector has the following properties.
| Name | Type| Required | Default | Description
|------|----------|---------|-------------|-------------|
-| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
+| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
| `groupId` |String| true | " " (empty string) | A unique string that identifies the group of consumer processes to which this consumer belongs. |
| `fetchMinBytes` | long|false | 1 | The minimum byte expected for each fetch response. |
| `autoCommitEnabled` | boolean |false | true | If set to true, the consumer's offset is periodically committed in the background.<br/><br/> This committed offset is used when the process fails as the position from which a new consumer begins. |
| `autoCommitIntervalMs` | long|false | 5000 | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true. |
| `heartbeatIntervalMs` | long| false | 3000 | The interval between heartbeats to the consumer when using Kafka's group management facilities. <br/><br/>**Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.|
| `sessionTimeoutMs` | long|false | 30000 | The timeout used to detect consumer failures when using Kafka's group management facility. |
-| `topic` | String|true | " " (empty string)| The Kafka topic which sends messages to Pulsar. |
-| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br/><br/>**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
+| `topic` | String|true | " " (empty string)| The Kafka topic that sends messages to Pulsar. |
+| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers. <br/><br/>**Note: other properties specified in the connector configuration file take precedence over this configuration**. |
| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringDeserializer | The deserializer class for Kafka consumers to deserialize keys.<br/> The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java).
| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values.
-| `autoOffsetReset` | String | false | "earliest" | The default offset reset policy. |
+| `autoOffsetReset` | String | false | earliest | The default offset reset policy. |
### Schema Management
@@ -65,81 +65,65 @@ If you want to access the raw key, you can use the `Message#getKeyBytes()` API.
Before using the Kafka source connector, you need to create a configuration file through one of the following methods.
-* JSON
+- JSON
- ```json
+ ```json
{
- "bootstrapServers": "pulsar-kafka:9092",
- "groupId": "test-pulsar-io",
- "topic": "my-topic",
- "sessionTimeoutMs": "10000",
- "autoCommitEnabled": false
+ "bootstrapServers": "pulsar-kafka:9092",
+ "groupId": "test-pulsar-io",
+ "topic": "my-topic",
+ "sessionTimeoutMs": "10000",
+ "autoCommitEnabled": false
}
- ```
+ ```
-* YAML
+- YAML
- ```yaml
+ ```yaml
configs:
- bootstrapServers: "pulsar-kafka:9092"
- groupId: "test-pulsar-io"
- topic: "my-topic"
- sessionTimeoutMs: "10000"
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
autoCommitEnabled: false
```
## Usage
-Here is an example of using the Kafka source connector with the configuration file as shown previously.
+You can make the Kafka source connector as a Pulsar built-in connector and use it on a standalone cluster or an on-premises cluster.
-1. Download a Kafka client and a Kafka connector.
+### Standalone cluster
- ```bash
- $ wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar
+This example describes how to use the Kafka source connector to feed data from Kafka and write data to Pulsar topics in the standalone mode.
- $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/connectors/pulsar-io-kafka-2.4.0.nar
- ```
+#### Prerequisites
-2. Create a network.
-
- ```bash
- $ docker network create kafka-pulsar
- ```
+- Install [Docker](https://docs.docker.com/get-docker/)(Community Edition).
-3. Pull a ZooKeeper image and start ZooKeeper.
-
- ```bash
- $ docker pull wurstmeister/zookeeper
+#### Steps
- $ docker run -d -it -p 2181:2181 --name pulsar-kafka-zookeeper --network kafka-pulsar wurstmeister/zookeeper
- ```
+1. Download and start the Confluent Platform.
-4. Pull a Kafka image and start Kafka.
-
- ```bash
- $ docker pull wurstmeister/kafka:2.11-1.0.2
-
- $ docker run -d -it --network kafka-pulsar -p 6667:6667 -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=pulsar-kafka -e KAFKA_ZOOKEEPER_CONNECT=pulsar-kafka-zookeeper:2181 --name pulsar-kafka wurstmeister/kafka:2.11-1.0.2
- ```
+For details, see the [documentation](https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#step-1-download-and-start-cp) to install the Kafka service locally.
-5. Pull a Pulsar image and start Pulsar standalone.
-
- ```bash
- $ docker pull apachepulsar/pulsar:{{pulsar:version}}
-
- $ docker run -d -it --network kafka-pulsar -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone
- ```
+2. Pull a Pulsar image and start Pulsar in standalone mode.
+
+ ```bash
+ docker pull apachepulsar/pulsar:latest
+
+ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:latest bin/pulsar standalone
+ ```
+
+3. Create a producer file _kafka-producer.py_.
-6. Create a producer file _kafka-producer.py_.
-
```python
from kafka import KafkaProducer
- producer = KafkaProducer(bootstrap_servers='pulsar-kafka:9092')
+ producer = KafkaProducer(bootstrap_servers='localhost:9092')
future = producer.send('my-topic', b'hello world')
future.get()
```
-7. Create a consumer file _pulsar-client.py_.
+4. Create a consumer file _pulsar-client.py_.
```python
import pulsar
@@ -157,23 +141,20 @@ Here is an example of using the Kafka source connector with the configuration fi
client.close()
```
-8. Copy the following files to Pulsar.
-
+5. Copy the following files to Pulsar.
+
```bash
- $ docker cp pulsar-io-kafka-{{pulsar:version}}.nar pulsar-kafka-standalone:/pulsar
- $ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
- $ docker cp pulsar-client.py pulsar-kafka-standalone:/pulsar/
- $ docker cp kafka-producer.py pulsar-kafka-standalone:/pulsar/
+ docker cp pulsar-io-kafka.nar pulsar-kafka-standalone:/pulsar
+ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf
```
-9. Open a new terminal window and start the Kafka source connector in local run mode.
+6. Open a new terminal window and start the Kafka source connector in local run mode.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ docker exec -it pulsar-kafka-standalone /bin/bash
- $ ./bin/pulsar-admin source localrun \
- --archive ./pulsar-io-kafka-{{pulsar:version}}.nar \
- --classname org.apache.pulsar.io.kafka.KafkaBytesSource \
+ ./bin/pulsar-admin source localrun \
+ --archive ./pulsar-io-kafka.nar \
--tenant public \
--namespace default \
--name kafka \
@@ -182,18 +163,49 @@ Here is an example of using the Kafka source connector with the configuration fi
--parallelism 1
```
-10. Open a new terminal window and run the consumer.
+7. Open a new terminal window and run the Kafka producer locally.
```bash
- $ docker exec -it pulsar-kafka-standalone /bin/bash
+ python3 kafka-producer.py
+ ```
- $ pip install kafka-python
+8. Open a new terminal window and run the Pulsar consumer locally.
- $ python3 kafka-producer.py
+ ```bash
+ python3 pulsar-client.py
```
- The following information appears on the consumer terminal window.
+The following information appears on the consumer terminal window.
```bash
Received message: 'hello world'
```
+
+### On-premises cluster
+
+This example explains how to create a Kafka source connector in an on-premises cluster.
+
+1. Copy the NAR package of the Kafka connector to the Pulsar connectors directory.
+
+ ```
+ cp pulsar-io-kafka-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-kafka-{{connector:version}}.nar
+ ```
+
+2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources reload
+ ```
+
+3. Check whether the Kafka source connector is available on the list or not.
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources available-sources
+ ```
+
+4. Create a Kafka source connector on a Pulsar cluster using the [`pulsar-admin sources create`](http://pulsar.apache.org/tools/pulsar-admin/2.9.0-SNAPSHOT/#-em-create-em--14) command.
+
+ ```
+ PULSAR_HOME/bin/pulsar-admin sources create \
+ --source-config-file <kafka-source-config.yaml>
+ ```
\ No newline at end of file