You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by vi...@apache.org on 2024/02/27 18:53:25 UTC
(pulsar-site) branch main updated: Documentation added for Azure data explorer IO-sink (#790)
This is an automated email from the ASF dual-hosted git repository.
visortelle pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 54db3530d6e4 Documentation added for Azure data explorer IO-sink (#790)
54db3530d6e4 is described below
commit 54db3530d6e4c78312221f26025f4b50c7260832
Author: Abhishek Saharn <10...@users.noreply.github.com>
AuthorDate: Wed Feb 28 00:23:19 2024 +0530
Documentation added for Azure data explorer IO-sink (#790)
* Added ADX conf page
* Update io-connectors.md
---
docs/io-azuredataexplorer-sink.md | 77 +++++++++++++++++++++++++++++++++++++++
docs/io-connectors.md | 8 ++++
2 files changed, 85 insertions(+)
diff --git a/docs/io-azuredataexplorer-sink.md b/docs/io-azuredataexplorer-sink.md
new file mode 100644
index 000000000000..25736107ef15
--- /dev/null
+++ b/docs/io-azuredataexplorer-sink.md
@@ -0,0 +1,77 @@
+---
+id: io-azuredataexplorer-sink
+title: Azure Data Explorer sink connector
+sidebar_label: "Azure Data Explorer sink connector"
+---
+
+:::note
+
+You can download all the Pulsar connectors on [download page](pathname:///download).
+
+:::
+
+The Azure Data Explorer(ADX) sink connector pulls messages from Pulsar topics and sink the messages to ADX clusters.
+
+## Configuration
+
+The configuration of the ADX sink connector has the following properties.
+
+### Property
+
+| Name | Type | Required | Default | Description |
+|-----------------------|--------|----------|----------------|--------------------------------------------------------------------------------------|
+| `clusterUrl` | String | true | (empty string) | The ADX cluster URL. |
+| `database` | String | true | (empty string) | The database name to which data need to be ingested |
+| `table` | String | true | (empty string) | Table name to which pulsar data need to be ingested |
+| `appId` | String | false | (empty string) | The AAD app Id for authentication |
+| `appKey` | String | false | (empty string) | The AAD app secret for authentication |
+| `tenantId` | String | false | (empty string) | The tenant Id for authentication |
+| `managedIdentityId` | String | false | (empty string) | The Managed Identity credential for authentication. Set this with clientId in case of user assigned MI and 'system' in case of system assigned managed identity|
+| `mappingRefName` | String | false | (empty string) | The mapping reference for ingestion |
+| `mappingRefType` | String | false | (empty string) | The type of mapping reference provided. Eg. CSV, JSON etc. |
+| `flushImmediately` | Boolean| false | false | Denotes if flush should happen immediately without aggregation. Not recommended to enable flushImmediately for production workloads |
+| `batchSize` | Int | false | 100 | For batching, this defines the number of records to hold for batching, to sink data to ADX |
+| `batchTimeMs` | Long | false | 10000 | For batching, this defines the time(in ms) to hold records before sink to ADX |
+| `maxRetryAttempts`| Int | false | 1 | Max retry attempts, In case of transient ingestion errors |
+| `retryBackOffTime`| Long |false | 10 | Period of time in milliseconds to backoff before retry for transient errors |
+
+### Example
+
+Before using the ADX sink connector, you need to create a configuration file through one of the following methods.
+
+* JSON
+
+ ```json
+ {
+ "configs": {
+ "clusterUrl":"https://somecluster.eastus.kusto.windows.net",
+ "database":"somedb",
+ "table": "tableName",
+ "appId": "xxxx-xxxx-xxxx-xxxx",
+ "appKey": "xxxx-xxxx-xxxx-xxxx",
+ "tenantId": "xxxx-xxxx-xxxx-xxxx",
+ "mappingRefName": "mapping ref name",
+ "mappingRefType":"CSV",
+ "flushImmediately":true,
+ "batchSize":100,
+ "batchTimeMs":10000,
+ }
+ }
+ ```
+
+* YAML
+
+ ```yaml
+ configs:
+ clusterUrl: https://somecluster.eastus.kusto.windows.net,
+ database: somedb,
+ table: tableName,
+ appId: xxxx-xxxx-xxxx-xxxx,
+ appKey: xxxx-xxxx-xxxx-xxxx,
+ tenantId: xxxx-xxxx-xxxx-xxxx,
+ mappingRefName: mapping ref name,
+ mappingRefType: CSV,
+ flushImmediately: true,
+ batchSize: 100,
+ batchTimeMs: 10000,
+ ```
diff --git a/docs/io-connectors.md b/docs/io-connectors.md
index 55c5ef56bda5..1b94b6d6dd0f 100644
--- a/docs/io-connectors.md
+++ b/docs/io-connectors.md
@@ -147,6 +147,14 @@ Pulsar has various sink connectors, which are sorted alphabetically as below.
* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/alluxio/src/main/java/org/apache/pulsar/io/alluxio/sink/AlluxioSink.java)
+### Azure Data Explorer
+
+* [Configuration](io-azuredataexplorer-sink.md#configuration)
+
+* [Example](io-azuredataexplorer-sink.md#example)
+
+* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/azure-data-explorer/src/main/java/org/apache/pulsar/io/azuredataexplorer/ADXSink.java)
+
### Cassandra
* [Configuration](io-cassandra-sink.md#configuration)