You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/08/30 10:00:41 UTC
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #5071:
[Doc] Add *How to Use Pulsar Connectors* Guide
Anonymitaet commented on a change in pull request #5071: [Doc] Add *How to Use Pulsar Connectors* Guide
URL: https://github.com/apache/pulsar/pull/5071#discussion_r319444007
##########
File path: site2/docs/io-use.md
##########
@@ -0,0 +1,1411 @@
+---
+id: io-use
+title: How to use Pulsar connectors
+sidebar_label: Use
+---
+
+This guide describes how to use Pulsar connectors.
+
+## Install a connector
+
+Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors.
+
+> #### Note
+>
+> When using a non-builtin connector, you need to specify the path of a archive file for the connector.
+
+To set up a builtin connector, follow
+the instructions [here](getting-started-standalone.md#installing-builtin-connectors).
+
+After the setup, the builtin connector is automatically discovered by Pulsar brokers (or function-workers), so no additional installation steps are required.
+
+## Configure a connector
+
+You can configure the following information:
+
+* [Configure a default storage location for a connector](#configure-a-default-storage-location-for-a-connector)
+
+* [Configure a connector with a YAML file](#configure-a-connector-with-yaml-file)
+
+### Configure a default storage location for a connector
+
+To configure a default folder for builtin connectors, set the `connectorsDirectory` parameter in the `./conf/functions_worker.yml` configuration file.
+
+**Example**
+
+Set the `./connectors` folder as the default storage location for builtin connectors.
+
+```
+########################
+# Connectors
+########################
+
+connectorsDirectory: ./connectors
+```
+
+### Configure a connector with a YAML file
+
+To configure a connector, you need to provide a YAML configuration file when creating a connector.
+
+The YAML configuration file tells Pulsar where to locate connectors and how to connect connectors with Pulsar topics.
+
+**Example 1**
+
+Below is a YAML configuration file of a Cassandra sink, which tells Pulsar:
+
+* Which Cassandra cluster to connect
+
+* What is the `keyspace` and `columnFamily` to be used in Cassandra for collecting data
+
+* How to map Pulsar messages into Cassandra table key and columns
+
+```shell
+tenant: public
+namespace: default
+name: cassandra-test-sink
+...
+# cassandra specific config
+configs:
+ roots: "localhost:9042"
+ keyspace: "pulsar_test_keyspace"
+ columnFamily: "pulsar_test_table"
+ keyname: "key"
+ columnName: "col"
+```
+
+**Example 2**
+
+Below is a YAML configuration file of a Kafka source.
+
+```shell
+configs:
+ bootstrapServers: "pulsar-kafka:9092"
+ groupId: "test-pulsar-io"
+ topic: "my-topic"
+ sessionTimeoutMs: "10000"
+ autoCommitEnabled: "false"
+```
+
+**Example 3**
+
+Below is a YAML configuration file of a MySQL JDBC sink.
+
+```shell
+configs:
+ userName: "root"
+ password: "jdbc"
+ jdbcUrl: "jdbc:mysql://127.0.0.1:3306/test_jdbc"
+ tableName: "test_jdbc"
+```
+
+## Get available connectors
+
+Before starting using connectors, you can perform the following operations:
+
+* [Reload connectors](#reload)
+
+* [Get a list of available connectors](#get-available-connectors)
+
+### `reload`
+
+If you add or delete a nar file in a connector folder, reload the available builtin connector before using it.
+
+#### Source
+
+Use the `reload` subcommand.
+
+```shell
+$ pulsar-admin sources reload
+```
+
+For more information, see [`here`](reference-connector-admin/#reload).
+
+#### Sink
+
+Use the `reload` subcommand.
+
+```shell
+$ pulsar-admin sinks reload
+```
+
+For more information, see [`here`](reference-connector-admin/#reload-1).
+
+### `available`
+
+After reloading connectors (optional), you can get a list of available connectors.
+
+#### Source
+
+Use the `available-sources` subcommand.
+
+```shell
+$ pulsar-admin sources available-sources
+```
+
+#### Sink
+
+Use the `available-sinks` subcommand.
+
+```shell
+$ pulsar-admin sinks available-sinks
+```
+
+## Run a connector
+
+To run a connector, you can perform the following operations:
+
+* [Create a connector](#create)
+
+* [Start a connector](#start)
+
+* [Run a connector locally](#localrun)
+
+### `create`
+
+You can create a connector using **Admin CLI**, **REST API** or **JAVA admin API**.
+
+#### Source
+
+Create a source connector.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--Admin CLI-->
+
+Use the `create` subcommand.
+
+```
+$ pulsar-admin sources create options
+```
+
+For more information, see [here](reference-connector-admin.md#create).
+
+<!--REST API-->
+
+Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName|operation/registerSource}
+
+<!--Java Admin API-->
+
+* Create a source connector with a **local file**.
+
+ ```java
+ void createSource(SourceConfig sourceConfig,
+ String fileName)
+ throws PulsarAdminException
+ ```
+
+ **Parameter**
+
+ |Name|Description
+ |---|---
+ `sourceConfig` | The source configuration object
+
+ **Exception**
+
+ |Name|Description|
+ |---|---
+ | `PulsarAdminException` | Unexpected error
+
+ For more information, see [`createSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSource-SourceConfig-java.lang.String-).
+
+* Create a source connector using a **remote file** with a URL from which fun-pkg can be downloaded.
+
+ ```java
+ void createSourceWithUrl(SourceConfig sourceConfig,
+ String pkgUrl)
+ throws PulsarAdminException
+ ```
+
+ Supported URLs are `http` and `file`.
+
+ **Example**
+
+ * HTTP: http://www.repo.com/fileName.jar
+
+ * File: file:/dir/fileName.jar
+
+ **Parameter**
+
+ Parameter| Description
+ |---|---
+ `sourceConfig` | The source configuration object
+ `pkgUrl` | URL from which pkg can be downloaded
+
+ **Exception**
+
+ |Name|Description|
+ |---|---
+ | `PulsarAdminException` | Unexpected error
+
+ For more information, see [`createSourceWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSourceWithUrl-SourceConfig-java.lang.String-).
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sink
+
+Create a sink connector.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--Admin CLI-->
+
+Use the `create` subcommand.
+
+```
+$ pulsar-admin sinks create options
+```
+
+For more information, see [here](reference-connector-admin.md#create-1).
+
+<!--REST API-->
+
+Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/:tenant/:namespace/:sinkName|operation/registerSink}
+
+<!--Java Admin API-->
+
+* Create a sink connector with a **local file**.
+
+ ```java
+ void createSink(SinkConfig sinkConfig,
+ String fileName)
+ throws PulsarAdminException
+ ```
+
+ **Parameter**
+
+ |Name|Description
+ |---|---
+ `sinkConfig` | The sink configuration object
+
+ **Exception**
+
+ |Name|Description|
+ |---|---
+ | `PulsarAdminException` | Unexpected error
+
+ For more information, see [`createSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSink-SinkConfig-java.lang.String-).
+
+* Create a sink connector using a **remote file** with a URL from which fun-pkg can be downloaded.
+
+ ```java
+ void createSinkWithUrl(SinkConfig sinkConfig,
+ String pkgUrl)
+ throws PulsarAdminException
+ ```
+
+ Supported URLs are `http` and `file`.
+
+ **Example**
+
+ * HTTP: http://www.repo.com/fileName.jar
+
+ * File: file:/dir/fileName.jar
+
+ **Parameter**
+
+ Parameter| Description
+ |---|---
+ `sinkConfig` | The sink configuration object
+ `pkgUrl` | URL from which pkg can be downloaded
+
+ **Exception**
+
+ |Name|Description|
+ |---|---
+ | `PulsarAdminException` | Unexpected error
+
+ For more information, see [`createSinkWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSinkWithUrl-SinkConfig-java.lang.String-).
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+### `start`
+
+You can start a connector using **Admin CLI** or **REST API**.
+
+#### Source
+
+Start a source connector.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--Admin CLI-->
+
+Use the `start` subcommand.
+
+```
+$ pulsar-admin sources start options
+```
+
+For more information, see [here](reference-connector-admin.md#start).
+
+<!--REST API-->
+
+* Start **all** source connectors.
+
+ Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/start|operation/startSource}
+
+* Start a **specified** source connector.
+
+ Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSource}
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sink
+
+Start a sink connector.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--Admin CLI-->
+
+Use the `start` subcommand.
+
+```
+$ pulsar-admin sinks start options
+```
+
+For more information, see [here](reference-connector-admin.md#start-1).
+
+<!--REST API-->
+
+* Start **all** sink connectors.
+
+ Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sinkName/start|operation/startSink}
+
+* Start a **specified** sink connector.
+
+ Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSink}
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+### `localrun`
+
+You can run a connector locally rather than deploying it on a Pulsar cluster using **Admin CLI**.
+
+#### Source
+
+Run a source connector locally.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--Admin CLI-->
+
+Use the `localrun` subcommand.
+
+```
+$ pulsar-admin sources localrun options
+```
+
+For more information, see [here](reference-connector-admin.md#localrun).
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sink
+
+Run a sink connector locally.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--Admin CLI-->
+
+Use the `localrun` subcommand.
+
+```
+$ pulsar-admin sinks localrun options
+```
+
+For more information, see [here](reference-connector-admin.md#localrun-1).
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+## Monitor a connector
+
+To monitor a connector, you can perform the following operations:
+
+* [Get the information of a connector](#get)
+
+* [Get the list of all running connectors](#list)
+
+* [Get the current status of a connector](#status)
+
+### `get`
+
+You can get the information of a connector using **Admin CLI**, **REST API** or **JAVA admin API**.
+
+#### Source
+
+Get the information of a source connector.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--Admin CLI-->
+
+Use the `get` subcommand.
+
+```
+$ pulsar-admin sources get options
+```
+
+For more information, see [here](reference-connector-admin.md#get).
+
+<!--REST API-->
+
+Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sources/:tenant/:namespace/:sourceName|operation/getSourceInfo}
+
+<!--Java Admin API-->
+
+```java
+SourceConfig getSource(String tenant,
+ String namespace,
+ String source)
+ throws PulsarAdminException
+```
+
+**Response example**
+
+```java
+serviceUrl : "http://my-broker.example.com:8080/"
Review comment:
This is from [JAVA admin API website](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSource-java.lang.String-java.lang.String-java.lang.String-)
What is the correct response?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services