You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by ur...@apache.org on 2022/06/06 12:06:31 UTC
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#0a6c6b6)
This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 333fae37bc4 Docs sync done from apache/pulsar(#0a6c6b6)
333fae37bc4 is described below
commit 333fae37bc4c3fe42d34774ed6834cbe7603eb7f
Author: Pulsar Site Updater <de...@pulsar.apache.org>
AuthorDate: Mon Jun 6 12:06:27 2022 +0000
Docs sync done from apache/pulsar(#0a6c6b6)
---
.../docs/developing-binary-protocol.md | 50 +++++++++++-----------
.../docs/sql-deployment-configurations.md | 47 ++++++++++++++++----
2 files changed, 64 insertions(+), 33 deletions(-)
diff --git a/site2/website-next/docs/developing-binary-protocol.md b/site2/website-next/docs/developing-binary-protocol.md
index d571e976b21..2d085254016 100644
--- a/site2/website-next/docs/developing-binary-protocol.md
+++ b/site2/website-next/docs/developing-binary-protocol.md
@@ -28,39 +28,39 @@ The Pulsar protocol allows for two types of commands:
Simple (payload-free) commands have this basic structure:
-| Component | Description | Size (in bytes) |
-|:--------------|:----------------------------------------------------------------------------------------|:----------------|
-| `totalSize` | The size of the frame, counting everything that comes after it (in bytes) | 4 |
-| `commandSize` | The size of the protobuf-serialized command | 4 |
-| `message` | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
+| Component | Description | Size (in bytes) |
+|:--------------|:--------------------------------------------------------------------------|:----------------|
+| `totalSize` | The size of the frame, counting everything that comes after it (in bytes) | 4 |
+| `commandSize` | The size of the protobuf-serialized command | 4 |
+| `command` | The protobuf serialized command | |
-### Payload commands
+### Message commands
Payload commands have this basic structure:
-| Component | Required or optional| Description | Size (in bytes) |
-|:-----------------------------------|:----------|:--------------------------------------------------------------------------------------------|:----------------|
-| `totalSize` | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 |
-| `commandSize` | Required | The size of the protobuf-serialized command | 4 |
-| `message` | Required | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
-| `magicNumberOfBrokerEntryMetadata` | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata <br /> **Note**: `magicNumberOfBrokerEntryMetadata` , `brokerEntryMetadataSize`, and `brokerEntryMetadata` should be used **together**. | 2 |
-| `brokerEntryMetadataSize` | Optional | The size of the broker entry metadata | 4 |
-| `brokerEntryMetadata` | Optional | The broker entry metadata stored as a binary protobuf message | |
-| `magicNumber` | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 |
-| `checksum` | Required | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 |
-| `metadataSize` | Required | The size of the message [metadata](#message-metadata) | 4 |
-| `metadata` | Required | The message [metadata](#message-metadata) stored as a binary protobuf message | |
-| `payload` | Required | Anything left in the frame is considered the payload and can include any sequence of bytes | |
+| Component | Required or optional | Description | Size (in bytes) |
+|:-----------------------------------|:---------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------|
+| `totalSize` | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 |
+| `commandSize` | Required | The size of the protobuf-serialized command | 4 |
+| `command` | Required | The protobuf serialized command | |
+| `magicNumberOfBrokerEntryMetadata` | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata <br /> **Note**: `magicNumberOfBrokerEntryMetadata` , `brokerEntryMetadataSize`, and `brokerEntryMetadata` should be used **together**. | 2 |
+| `brokerEntryMetadataSize` | Optional | The size of the broker entry metadata | 4 |
+| `brokerEntryMetadata` | Optional | The broker entry metadata stored as a binary protobuf message | |
+| `magicNumber` | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 |
+| `checksum` | Required | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 |
+| `metadataSize` | Required | The size of the message [metadata](#message-metadata) | 4 |
+| `metadata` | Required | The message [metadata](#message-metadata) stored as a binary protobuf message | |
+| `payload` | Required | Anything left in the frame is considered the payload and can include any sequence of bytes | |
## Broker entry metadata
Broker entry metadata is stored alongside the message metadata as a serialized protobuf message.
It is created by the broker when the message arrived at the broker and passed without changes to the consumer if configured.
-| Field | Required or optional | Description |
-|:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------|
-| `broker_timestamp` | Optional | The timestamp when a message arrived at the broker (`id est` as the number of milliseconds since January 1st, 1970 in UTC) |
-| `index` | Optional | The index of the message. It is assigned by the broker.
+| Field | Required or optional | Description |
+|:-------------------|:---------------------|:---------------------------------------------------------------------------------------------------------------------------|
+| `broker_timestamp` | Optional | The timestamp when a message arrived at the broker (`id est` as the number of milliseconds since January 1st, 1970 in UTC) |
+| `index` | Optional | The index of the message. It is assigned by the broker. |
If you want to use broker entry metadata for **brokers**, configure the [`brokerEntryMetadataInterceptors`](reference-configuration.md#broker) parameter in the `broker.conf` file.
@@ -246,7 +246,7 @@ Command `Send` is used to publish a new message within the context of an
already existing producer. If a producer has not yet been created for the
connection, the broker will terminate the connection. This command is used
in a frame that includes command as well as message payload, for which the
-complete format is specified in the [payload commands](#payload-commands) section.
+complete format is specified in the [message commands](#message-commands) section.
```protobuf
@@ -399,7 +399,7 @@ within the limits of the given permits.
This command is used in a frame that includes the message payload as well, for
-which the complete format is specified in the [payload commands](#payload-commands)
+which the complete format is specified in the [message commands](#message-commands)
section.
```protobuf
diff --git a/site2/website-next/docs/sql-deployment-configurations.md b/site2/website-next/docs/sql-deployment-configurations.md
index 10fb47ad7b4..1666414460b 100644
--- a/site2/website-next/docs/sql-deployment-configurations.md
+++ b/site2/website-next/docs/sql-deployment-configurations.md
@@ -14,10 +14,13 @@ You can configure Presto Pulsar Connector in the `${project.root}/conf/presto/ca
# name of the connector to be displayed in the catalog
connector.name=pulsar
-# the url of Pulsar broker service
+# the URL of Pulsar broker service
pulsar.web-service-url=http://localhost:8080
-# URI of Zookeeper cluster
+# the URL of Pulsar broker binary service
+pulsar.broker-binary-service-url=pulsar://localhost:6650
+
+# the URL of Zookeeper cluster
pulsar.zookeeper-uri=localhost:2181
# minimum number of entries to read at a single time
@@ -29,10 +32,10 @@ pulsar.target-num-splits=4
# max size of one batch message (default value is 5MB)
pulsar.max-message-size=5242880
-# number of split used when querying data from pulsar
+# number of split used when querying data from Pulsar
pulsar.target-num-splits=2
-# size of queue to buffer entry read from pulsar
+# size of queue to buffer entry read from Pulsar
pulsar.max-split-entry-queue-size=1000
# size of queue to buffer message extract from entries
@@ -77,6 +80,9 @@ pulsar.tls-hostname-verification-enable=null
# path for the trusted TLS certificate file of Pulsar broker
pulsar.tls-trust-cert-file-path=null
+## whether to enable Pulsar authorization
+pulsar.authorization-enabled=false
+
# set the threshold for BookKeeper request throttle, default is disabled
pulsar.bookkeeper-throttle-value=0
@@ -106,16 +112,41 @@ pulsar.nar-extraction-directory=System.getProperty("java.io.tmpdir")
```
-You can connect Presto to a Pulsar cluster with multiple hosts. To configure multiple hosts for brokers, add multiple URLs to `pulsar.web-service-url`. To configure multiple hosts for ZooKeeper, add multiple URIs to `pulsar.zookeeper-uri`. The following is an example.
+### Enable authentication and authorization between Pulsar and Pulsar SQL
+
+By default, the authentication and authorization between Pulsar and Pulsar SQL is disabled.
+
+To enable it, set the following configurations in the `${project.root}/conf/presto/catalog/pulsar.properties` properties file:
+
+```properties
+
+pulsar.authorization-enabled=true
+pulsar.broker-binary-service-url=pulsar://localhost:6650
```
+### Connect Presto to Pulsar with multiple hosts
+
+You can connect Presto to a Pulsar cluster with multiple hosts.
+* To configure multiple hosts for brokers, add multiple URLs to `pulsar.web-service-url`.
+* To configure multiple hosts for ZooKeeper, add multiple URIs to `pulsar.zookeeper-uri`.
+
+The following is an example.
+
+```properties
+
pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
pulsar.zookeeper-uri=localhost1,localhost2:2181
```
-**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger.
+### Get the last message in a topic
+
+:::note
+
+By default, Pulsar SQL **does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger.
+
+:::
If you want to get the last message in a topic, set the following configurations:
@@ -266,11 +297,11 @@ presto> SELECT * FROM system.runtime.nodes;
```
-For more information about deployment in Presto, refer to [Presto deployment](https://trino.io/docs/current/installation/deployment.html).
+For more information about the deployment in Presto, refer to [Presto deployment](https://trino.io/docs/current/installation/deployment.html).
:::note
-The broker does not advance LAC, so when Pulsar SQL bypass broker to query data, it can only read entries up to the LAC that all the bookies learned. You can enable periodically write LAC on the broker by setting "bookkeeperExplicitLacIntervalInMills" in the broker.conf.
+The broker does not advance LAC, so when Pulsar SQL bypasses broker to query data, it can only read entries up to the LAC that all the bookies learned. You can enable periodically write LAC on the broker by setting "bookkeeperExplicitLacIntervalInMills" in the `broker.conf` file.
:::