You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by mm...@apache.org on 2018/09/19 03:37:01 UTC
[incubator-pulsar] branch master updated: docs for 2.1.1-incubating
release (#2601)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 3097283 docs for 2.1.1-incubating release (#2601)
3097283 is described below
commit 3097283823d14929bccd8ce1140d12ded1a93afc
Author: Boyang Jerry Peng <je...@gmail.com>
AuthorDate: Tue Sep 18 20:36:57 2018 -0700
docs for 2.1.1-incubating release (#2601)
* docs for 2.1.1-incubating release
* removing docs for sql
---
site2/website/release-notes.md | 17 +
site2/website/releases.json | 1 +
site2/website/sidebars.json | 5 -
.../version-2.1.1-incubating/adaptors-kafka.md | 262 +++
.../administration-proxy.md | 67 +
.../client-libraries-cpp.md | 184 ++
.../client-libraries-python.md | 95 ++
.../cookbooks-tiered-storage.md | 221 +++
.../version-2.1.1-incubating/deploy-bare-metal.md | 357 ++++
.../version-2.1.1-incubating/functions-api.md | 721 ++++++++
.../functions-deploying.md | 227 +++
.../functions-guarantees.md | 42 +
.../version-2.1.1-incubating/functions-overview.md | 452 +++++
.../functions-quickstart.md | 266 +++
.../getting-started-clients.md | 58 +
.../getting-started-standalone.md | 198 +++
.../version-2.1.1-incubating/io-managing.md | 162 ++
.../version-2.1.1-incubating/io-quickstart.md | 400 +++++
.../reference-configuration.md | 468 +++++
.../reference-pulsar-admin.md | 1795 ++++++++++++++++++++
.../sql-deployment-configurations.md | 3 +-
.../sql-getting-started.md | 3 +-
.../version-2.1.1-incubating}/sql-overview.md | 3 +-
site2/website/versions.json | 1 +
24 files changed, 6000 insertions(+), 8 deletions(-)
diff --git a/site2/website/release-notes.md b/site2/website/release-notes.md
index 1745e9d..a12db34 100644
--- a/site2/website/release-notes.md
+++ b/site2/website/release-notes.md
@@ -1,6 +1,23 @@
## Apache incubator
+### 2.1.1-incubating — 2018-09-17 <a id="2.1.1-incubating"></a>
+
+This release fixes issues reported for 2.1.0-incubating.
+
+ * [#2473](https://github.com/apache/incubator-pulsar/pull/2473) - Downgrading ZK to stable version 3.4.13
+ * [#2219](https://github.com/apache/incubator-pulsar/pull/2219) - Cpp client: add PatternMultiTopicsConsumerImpl to support regex subscribe
+ * [#2387](https://github.com/apache/incubator-pulsar/pull/2387) - Fixed race condition during expansion of concurrent open hash maps
+ * [#2348](https://github.com/apache/incubator-pulsar/pull/2348) - Fix NPE when splitting and unloading bundle
+ * [#2223](https://github.com/apache/incubator-pulsar/pull/2223) - fix bug in FunctionRuntimeManager involving not cleaning up old invalid assignments
+ * [#2367](https://github.com/apache/incubator-pulsar/pull/2367) - [compaction] make topic compaction works with partitioned topic
+ * [#2203](https://github.com/apache/incubator-pulsar/pull/2203) - Make sure schema is initialized before the topic is loaded
+
+The complete list of changes can be found at:
+https://github.com/apache/incubator-pulsar/milestone/17?closed=1
+
+https://github.com/apache/incubator-pulsar/releases/tag/v2.1.1-incubating
+
### 2.1.0-incubating — 2018-08-02 <a id="2.1.0-incubating"></a>
This is the seventh release of Apache Pulsar since entering the ASF incubator.
diff --git a/site2/website/releases.json b/site2/website/releases.json
index 9ea62bc..2966e88 100644
--- a/site2/website/releases.json
+++ b/site2/website/releases.json
@@ -1,4 +1,5 @@
[
+ "2.1.1",
"2.1.0",
"2.0.1",
"1.22.1",
diff --git a/site2/website/sidebars.json b/site2/website/sidebars.json
index 5afda90..f321d30 100644
--- a/site2/website/sidebars.json
+++ b/site2/website/sidebars.json
@@ -34,11 +34,6 @@
"io-connectors",
"io-develop"
],
- "Pulsar SQL": [
- "sql-overview",
- "sql-getting-started",
- "sql-deployment-configurations"
- ],
"Deployment": [
"deploy-aws",
"deploy-kubernetes",
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/adaptors-kafka.md b/site2/website/versioned_docs/version-2.1.1-incubating/adaptors-kafka.md
new file mode 100644
index 0000000..3e6de03
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/adaptors-kafka.md
@@ -0,0 +1,262 @@
+---
+id: version-2.1.1-incubating-adaptors-kafka
+title: Pulsar adaptor for Apache Kafka
+sidebar_label: Kafka client wrapper
+original_id: adaptors-kafka
+---
+
+
+Pulsar provides an easy option for applications that are currently written using the [Apache Kafka](http://kafka.apache.org) Java client API.
+
+## Using the Pulsar Kafka compatibility wrapper
+
+In an existing application, change the regular Kafka client dependency and replace it with the Pulsar Kafka wrapper. Remove:
+
+```xml
+<dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kakfa-clients</artifactId>
+ <version>0.10.2.1</version>
+</dependency>
+```
+
+Then include this dependency for the Pulsar Kafka wrapper:
+
+```xml
+<dependency>
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>pulsar-client-kafka</artifactId>
+ <version>{{pulsar:version}}</version>
+</dependency>
+```
+
+With the new dependency, the existing code should work without any changes. The only
+thing that needs to be adjusted is the configuration, to make sure to point the
+producers and consumers to Pulsar service rather than Kafka and to use a particular
+Pulsar topic.
+
+## Using the Pulsar Kafka compatibility wrapper together with existing kafka client.
+
+When migrating from Kafka to Pulsar, the application might have to use the original kafka client
+and the pulsar kafka wrapper together during migration. Then you should consider using the
+unshaded pulsar kafka client wrapper.
+
+```xml
+<dependency>
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>pulsar-client-kafka-original</artifactId>
+ <version>{{pulsar:version}}</version>
+</dependency>
+```
+
+When using this dependency, you need to construct producer using `org.apache.kafka.clients.producer.PulsarKafkaProducer`
+instead of `org.apache.kafka.clients.producer.KafkaProducer` and `org.apache.kafka.clients.producer.PulsarKafkaConsumer` for consumers.
+
+## Producer example
+
+```java
+// Topic needs to be a regular Pulsar topic
+String topic = "persistent://public/default/my-topic";
+
+Properties props = new Properties();
+// Point to a Pulsar service
+props.put("bootstrap.servers", "pulsar://localhost:6650");
+
+props.put("key.serializer", IntegerSerializer.class.getName());
+props.put("value.serializer", StringSerializer.class.getName());
+
+Producer<Integer, String> producer = new KafkaProducer<>(props);
+
+for (int i = 0; i < 10; i++) {
+ producer.send(new ProducerRecord<Integer, String>(topic, i, "hello-" + i));
+ log.info("Message {} sent successfully", i);
+}
+
+producer.close();
+```
+
+## Consumer example
+
+```java
+String topic = "persistent://public/default/my-topic";
+
+Properties props = new Properties();
+// Point to a Pulsar service
+props.put("bootstrap.servers", "pulsar://localhost:6650");
+props.put("group.id", "my-subscription-name");
+props.put("enable.auto.commit", "false");
+props.put("key.deserializer", IntegerDeserializer.class.getName());
+props.put("value.deserializer", StringDeserializer.class.getName());
+
+Consumer<Integer, String> consumer = new KafkaConsumer<>(props);
+consumer.subscribe(Arrays.asList(topic));
+
+while (true) {
+ ConsumerRecords<Integer, String> records = consumer.poll(100);
+ records.forEach(record -> {
+ log.info("Received record: {}", record);
+ });
+
+ // Commit last offset
+ consumer.commitSync();
+}
+```
+
+## Complete Examples
+
+You can find the complete producer and consumer examples
+[here](https://github.com/apache/incubator-pulsar/tree/master/pulsar-client-kafka-compat/pulsar-client-kafka-tests/src/test/java/org/apache/pulsar/client/kafka/compat/examples).
+
+## Compatibility matrix
+
+Currently the Pulsar Kafka wrapper supports most of the operations offered by the Kafka API.
+
+#### Producer
+
+APIs:
+
+| Producer Method | Supported | Notes |
+|:------------------------------------------------------------------------------|:----------|:-------------------------------------------------------------------------|
+| `Future<RecordMetadata> send(ProducerRecord<K, V> record)` | Yes | Currently no support for explicitly set the partition id when publishing |
+| `Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback)` | Yes | |
+| `void flush()` | Yes | |
+| `List<PartitionInfo> partitionsFor(String topic)` | No | |
+| `Map<MetricName, ? extends Metric> metrics()` | No | |
+| `void close()` | Yes | |
+| `void close(long timeout, TimeUnit unit)` | Yes | |
+
+Properties:
+
+| Config property | Supported | Notes |
+|:----------------------------------------|:----------|:------------------------------------------------------------------------------|
+| `acks` | Ignored | Durability and quorum writes are configured at the namespace level |
+| `batch.size` | Ignored | |
+| `block.on.buffer.full` | Yes | If true it will block producer, otherwise give error |
+| `bootstrap.servers` | Yes | Needs to point to a single Pulsar service URL |
+| `buffer.memory` | Ignored | |
+| `client.id` | Ignored | |
+| `compression.type` | Yes | Allows `gzip` and `lz4`. No `snappy`. |
+| `connections.max.idle.ms` | Ignored | |
+| `interceptor.classes` | Ignored | |
+| `key.serializer` | Yes | |
+| `linger.ms` | Yes | Controls the group commit time when batching messages |
+| `max.block.ms` | Ignored | |
+| `max.in.flight.requests.per.connection` | Ignored | In Pulsar ordering is maintained even with multiple requests in flight |
+| `max.request.size` | Ignored | |
+| `metric.reporters` | Ignored | |
+| `metrics.num.samples` | Ignored | |
+| `metrics.sample.window.ms` | Ignored | |
+| `partitioner.class` | Ignored | |
+| `receive.buffer.bytes` | Ignored | |
+| `reconnect.backoff.ms` | Ignored | |
+| `request.timeout.ms` | Ignored | |
+| `retries` | Ignored | Pulsar client retries with exponential backoff until the send timeout expires |
+| `send.buffer.bytes` | Ignored | |
+| `timeout.ms` | Ignored | |
+| `value.serializer` | Yes | |
+
+
+#### Consumer
+
+APIs:
+
+| Consumer Method | Supported | Notes |
+|:--------------------------------------------------------------------------------------------------------|:----------|:------|
+| `Set<TopicPartition> assignment()` | No | |
+| `Set<String> subscription()` | Yes | |
+| `void subscribe(Collection<String> topics)` | Yes | |
+| `void subscribe(Collection<String> topics, ConsumerRebalanceListener callback)` | No | |
+| `void assign(Collection<TopicPartition> partitions)` | No | |
+| `void subscribe(Pattern pattern, ConsumerRebalanceListener callback)` | No | |
+| `void unsubscribe()` | Yes | |
+| `ConsumerRecords<K, V> poll(long timeoutMillis)` | Yes | |
+| `void commitSync()` | Yes | |
+| `void commitSync(Map<TopicPartition, OffsetAndMetadata> offsets)` | Yes | |
+| `void commitAsync()` | Yes | |
+| `void commitAsync(OffsetCommitCallback callback)` | Yes | |
+| `void commitAsync(Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback)` | Yes | |
+| `void seek(TopicPartition partition, long offset)` | Yes | |
+| `void seekToBeginning(Collection<TopicPartition> partitions)` | Yes | |
+| `void seekToEnd(Collection<TopicPartition> partitions)` | Yes | |
+| `long position(TopicPartition partition)` | Yes | |
+| `OffsetAndMetadata committed(TopicPartition partition)` | Yes | |
+| `Map<MetricName, ? extends Metric> metrics()` | No | |
+| `List<PartitionInfo> partitionsFor(String topic)` | No | |
+| `Map<String, List<PartitionInfo>> listTopics()` | No | |
+| `Set<TopicPartition> paused()` | No | |
+| `void pause(Collection<TopicPartition> partitions)` | No | |
+| `void resume(Collection<TopicPartition> partitions)` | No | |
+| `Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> timestampsToSearch)` | No | |
+| `Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions)` | No | |
+| `Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions)` | No | |
+| `void close()` | Yes | |
+| `void close(long timeout, TimeUnit unit)` | Yes | |
+| `void wakeup()` | No | |
+
+Properties:
+
+| Config property | Supported | Notes |
+|:--------------------------------|:----------|:------------------------------------------------------|
+| `group.id` | Yes | Maps to a Pulsar subscription name |
+| `max.poll.records` | Ignored | |
+| `max.poll.interval.ms` | Ignored | Messages are "pushed" from broker |
+| `session.timeout.ms` | Ignored | |
+| `heartbeat.interval.ms` | Ignored | |
+| `bootstrap.servers` | Yes | Needs to point to a single Pulsar service URL |
+| `enable.auto.commit` | Yes | |
+| `auto.commit.interval.ms` | Ignored | With auto-commit, acks are sent immediately to broker |
+| `partition.assignment.strategy` | Ignored | |
+| `auto.offset.reset` | Ignored | |
+| `fetch.min.bytes` | Ignored | |
+| `fetch.max.bytes` | Ignored | |
+| `fetch.max.wait.ms` | Ignored | |
+| `metadata.max.age.ms` | Ignored | |
+| `max.partition.fetch.bytes` | Ignored | |
+| `send.buffer.bytes` | Ignored | |
+| `receive.buffer.bytes` | Ignored | |
+| `client.id` | Ignored | |
+
+
+## Custom Pulsar configurations
+
+You can configure Pulsar authentication provider directly from the Kafka properties.
+
+### Pulsar client properties:
+
+| Config property | Default | Notes |
+|:---------------------------------------|:--------|:---------------------------------------------------------------------------------------|
+| [`pulsar.authentication.class`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setAuthentication-org.apache.pulsar.client.api.Authentication-) | | Configure to auth provider. Eg. `org.apache.pulsar.client.impl.auth.AuthenticationTls` |
+| [`pulsar.authentication.params.map`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setAuthentication-java.lang.String-java.util.Map-) | | Map which represents parameters for the Authentication-Plugin |
+| [`pulsar.authentication.params.string`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setAuthentication-java.lang.String-java.lang.String-) | | String which represents parameters for the Authentication-Plugin, Eg. `key1:val1,key2:val2` |
+| [`pulsar.use.tls`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setUseTls-boolean-) | `false` | Enable TLS transport encryption |
+| [`pulsar.tls.trust.certs.file.path`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setTlsTrustCertsFilePath-java.lang.String-) | | Path for the TLS trust certificate store |
+| [`pulsar.tls.allow.insecure.connection`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setTlsAllowInsecureConnection-boolean-) | `false` | Accept self-signed certificates from brokers |
+| [`pulsar.operation.timeout.ms`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setOperationTimeout-int-java.util.concurrent.TimeUnit-) | `30000` | General operations timeout |
+| [`pulsar.stats.interval.seconds`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setStatsInterval-long-java.util.concurrent.TimeUnit-) | `60` | Pulsar client lib stats printing interval |
+| [`pulsar.num.io.threads`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setIoThreads-int-) | `1` | Number of Netty IO threads to use |
+| [`pulsar.connections.per.broker`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setConnectionsPerBroker-int-) | `1` | Max number of connection to open to each broker |
+| [`pulsar.use.tcp.nodelay`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setUseTcpNoDelay-boolean-) | `true` | TCP no-delay |
+| [`pulsar.concurrent.lookup.requests`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setConcurrentLookupRequest-int-) | `50000` | Max number of concurrent topic lookups |
+| [`pulsar.max.number.rejected.request.per.connection`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ClientConfiguration.html#setMaxNumberOfRejectedRequestPerConnection-int-) | `50` | Threshold of errors to forcefully close a connection |
+
+
+### Pulsar producer properties
+
+| Config property | Default | Notes |
+|:---------------------------------------|:--------|:---------------------------------------------------------------------------------------|
+| [`pulsar.producer.name`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setProducerName-java.lang.String-) | | Specify producer name |
+| [`pulsar.producer.initial.sequence.id`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setInitialSequenceId-long-) | | Specify baseline for sequence id for this producer |
+| [`pulsar.producer.max.pending.messages`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setMaxPendingMessages-int-) | `1000` | Set the max size of the queue holding the messages pending to receive an acknowledgment from the broker. |
+| [`pulsar.producer.max.pending.messages.across.partitions`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setMaxPendingMessagesAcrossPartitions-int-) | `50000` | Set the number of max pending messages across all the partitions |
+| [`pulsar.producer.batching.enabled`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBatchingEnabled-boolean-) | `true` | Control whether automatic batching of messages is enabled for the producer |
+| [`pulsar.producer.batching.max.messages`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBatchingMaxMessages-int-) | `1000` | The maximum number of messages permitted in a batch |
+
+
+### Pulsar consumer Properties
+
+| Config property | Default | Notes |
+|:---------------------------------------|:--------|:---------------------------------------------------------------------------------------|
+| [`pulsar.consumer.name`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerConfiguration.html#setConsumerName-java.lang.String-) | | Set the consumer name |
+| [`pulsar.consumer.receiver.queue.size`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerConfiguration.html#setReceiverQueueSize-int-) | 1000 | Sets the size of the consumer receive queue |
+| [`pulsar.consumer.total.receiver.queue.size.across.partitions`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerConfiguration.html#setMaxTotalReceiverQueueSizeAcrossPartitions-int-) | 50000 | Set the max total receiver queue size across partitons |
+
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/administration-proxy.md b/site2/website/versioned_docs/version-2.1.1-incubating/administration-proxy.md
new file mode 100644
index 0000000..8f87782
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/administration-proxy.md
@@ -0,0 +1,67 @@
+---
+id: version-2.1.1-incubating-administration-proxy
+title: The Pulsar proxy
+sidebar_label: Pulsar proxy
+original_id: administration-proxy
+---
+
+The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) is an optional gateway that you can run over the brokers in a Pulsar cluster. We recommend running a Pulsar proxy in cases when direction connections between clients and Pulsar brokers are either infeasible, undesirable, or both, for example when running Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform.
+
+## Running the proxy
+
+In order to run the Pulsar proxy, you need to have both a local [ZooKeeper](https://zookeeper.apache.org) and configuration store quorum set up for use by your Pulsar cluster. For instructions, see [this document](deploy-bare-metal.md). Once you have ZooKeeper set up and have connection strings for both ZooKeeper quorums, you can use the [`proxy`](reference-cli-tools.md#pulsar-proxy) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool to start up the proxy (preferably on it [...]
+
+To start the proxy:
+
+```bash
+$ cd /path/to/pulsar/directory
+$ bin/pulsar proxy \
+ --zookeeper-servers zk-0,zk-1,zk-2 \
+ --global-zookeeper-servers zk-0,zk-1,zk-2
+```
+
+> You can run as many instances of the Pulsar proxy in a cluster as you would like.
+
+
+## Stopping the proxy
+
+The Pulsar proxy runs by default in the foreground. To stop the proxy, simply stop the process in which it's running.
+
+## Proxy frontends
+
+We recommend running the Pulsar proxy behind some kind of load-distributing frontend, such as an [HAProxy](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts) load balancer.
+
+## Using Pulsar clients with the proxy
+
+Once your Pulsar proxy is up and running, preferably behind a load-distributing [frontend](#proxy-frontends), clients can connect to the proxy via whichever address is used by the frontend. If the address were the DNS address `pulsar.cluster.default`, for example, then the connection URL for clients would be `pulsar://pulsar.cluster.default:6650`.
+
+## Proxy configuration
+
+The Pulsar proxy can be configured using the [`proxy.conf`](reference-configuration.md#proxy) configuration file. The following parameters are available in that file:
+
+|Name|Description|Default|
+|---|---|---|
+|zookeeperServers| The ZooKeeper quorum connection string (as a comma-separated list) ||
+|configurationStoreServers| Configuration store connection string (as a comma-separated list) ||
+|zookeeperSessionTimeoutMs| ZooKeeper session timeout (in milliseconds) |30000|
+|servicePort| The port to use for server binary Protobuf requests |6650|
+|servicePortTls| The port to use to server binary Protobuf TLS requests |6651|
+|statusFilePath | Path for the file used to determine the rotation status for the proxy instance when responding to service discovery health checks ||
+|authenticationEnabled| Whether authentication is enabled for the Pulsar proxy |false|
+|authenticationProviders| Authentication provider name list (a comma-separated list of class names) ||
+|authorizationEnabled| Whether authorization is enforced by the Pulsar proxy |false|
+|authorizationProvider| Authorization provider as a fully qualified class name |org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider|
+|brokerClientAuthenticationPlugin| The authentication plugin used by the Pulsar proxy to authenticate with Pulsar brokers ||
+|brokerClientAuthenticationParameters| The authentication parameters used by the Pulsar proxy to authenticate with Pulsar brokers ||
+|brokerClientTrustCertsFilePath| The path to trusted certificates used by the Pulsar proxy to authenticate with Pulsar brokers ||
+|superUserRoles| Role names that are treated as “super-users,” meaning that they will be able to perform all admin ||
+|forwardAuthorizationCredentials| Whether client authorization credentials are forwared to the broker for re-authorization. Authentication must be enabled via authenticationEnabled=true for this to take effect. |false|
+|maxConcurrentInboundConnections| Max concurrent inbound connections. The proxy will reject requests beyond that. |10000|
+|maxConcurrentLookupRequests| Max concurrent outbound connections. The proxy will error out requests beyond that. |50000|
+|tlsEnabledInProxy| Whether TLS is enabled for the proxy |false|
+|tlsEnabledWithBroker| Whether TLS is enabled when communicating with Pulsar brokers |false|
+|tlsCertificateFilePath| Path for the TLS certificate file ||
+|tlsKeyFilePath| Path for the TLS private key file ||
+|tlsTrustCertsFilePath| Path for the trusted TLS certificate pem file ||
+|tlsHostnameVerificationEnabled| Whether the hostname is validated when the proxy creates a TLS connection with brokers |false|
+|tlsRequireTrustedClientCertOnConnect| Whether client certificates are required for TLS. Connections are rejected if the client certificate isn’t trusted. |false|
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-cpp.md
new file mode 100644
index 0000000..547d1d9
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-cpp.md
@@ -0,0 +1,184 @@
+---
+id: version-2.1.1-incubating-client-libraries-cpp
+title: The Pulsar C++ client
+sidebar_label: C++
+original_id: client-libraries-cpp
+---
+
+## Supported platforms
+
+The Pulsar C++ client has been successfully tested on **MacOS** and **Linux**.
+
+## Linux
+
+### Install
+
+> Since the 2.1.0 release, Pulsar ships pre-built RPM and Debian packages. You can choose to download
+> and install those packages instead of building them yourself.
+
+#### RPM
+
+| Link | Crypto files |
+|------|--------------|
+| [client]({{pulsar:rpm:client}}) | [asc]({{pulsar:rpm:client}}.asc), [sha512]({{pulsar:rpm:client}}.sha512) |
+| [client-debuginfo]({{pulsar:rpm:client-debuginfo}}) | [asc]({{pulsar:rpm:client-debuginfo}}.asc), [sha512]({{pulsar:rpm:client-debuginfo}}.sha512) |
+| [client-devel]({{pulsar:rpm:client-devel}}) | [asc]({{pulsar:rpm:client-devel}}.asc), [sha512]({{pulsar:rpm:client-devel}}.sha512) |
+
+To install a RPM package, download the RPM packages and install them using the following command:
+
+```bash
+$ rpm -ivh apache-pulsar-client*.rpm
+```
+
+#### DEB
+
+| Link | Crypto files |
+|------|--------------|
+| [client]({{pulsar:deb:client}}) | [asc]({{pulsar:deb:client}}.asc), [sha512]({{pulsar:deb:client}}.sha512) |
+| [client-devel]({{pulsar:deb:client-devel}}) | [asc]({{pulsar:deb:client-devel}}.asc), [sha512]({{pulsar:deb:client-devel}}.sha512) |
+
+To install a DEB package, download the DEB packages and install them using the following command:
+
+```bash
+$ apt-install apache-pulsar-client*.deb
+```
+
+### Build
+
+> If you want to build RPM and Debian packages off latest master, you can follow the instructions
+> below to do so. All the instructions are run at the root directory of your cloned Pulsar
+> repo.
+
+There are recipes that build RPM and Debian packages containing a
+statically linked `libpulsar.so` / `libpulsar.a` with all the required
+dependencies.
+
+To build the C++ library packages, first build the Java packages:
+
+```shell
+mvn install -DskipTests
+```
+
+#### RPM
+
+```shell
+pulsar-client-cpp/pkg/rpm/docker-build-rpm.sh
+```
+
+This will build the RPM inside a Docker container and it will leave the RPMs
+in `pulsar-client-cpp/pkg/rpm/RPMS/x86_64/`.
+
+| Package name | Content |
+|-----|-----|
+| pulsar-client | Shared library `libpulsar.so` |
+| pulsar-client-devel | Static library `libpulsar.a` and C++ and C headers |
+| pulsar-client-debuginfo | Debug symbols for `libpulsar.so` |
+
+#### Deb
+
+To build Debian packages:
+
+```shell
+pulsar-client-cpp/pkg/deb/docker-build-deb.sh
+```
+
+Debian packages will be created at `pulsar-client-cpp/pkg/deb/BUILD/DEB/`
+
+| Package name | Content |
+|-----|-----|
+| pulsar-client | Shared library `libpulsar.so` |
+| pulsar-client-dev | Static library `libpulsar.a` and C++ and C headers |
+
+## MacOS
+
+Use the [Homebrew](https://brew.sh/) supplied recipe to build the Pulsar
+client lib on MacOS.
+
+```shell
+brew install https://raw.githubusercontent.com/apache/incubator-pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb
+```
+
+If using Python 3 on MacOS, add the flag `--with-python3` to the above command.
+
+This will install the package with the library and headers.
+
+## Connection URLs
+
+
+To connect to Pulsar using client libraries, you need to specify a Pulsar protocol URL.
+
+Pulsar protocol URLs are assigned to specific clusters, use the pulsar URI scheme and have a default port of 6650. Here’s an example for localhost:
+
+```http
+pulsar://localhost:6650
+```
+
+A URL for a production Pulsar cluster may look something like this:
+```http
+pulsar://pulsar.us-west.example.com:6650
+```
+
+If you’re using TLS authentication, the URL will look like something like this:
+```http
+pulsar+ssl://pulsar.us-west.example.com:6651
+```
+
+## Consumer
+
+```c++
+Client client("pulsar://localhost:6650");
+
+Consumer consumer;
+Result result = client.subscribe("my-topic", "my-subscribtion-name", consumer);
+if (result != ResultOk) {
+ LOG_ERROR("Failed to subscribe: " << result);
+ return -1;
+}
+
+Message msg;
+
+while (true) {
+ consumer.receive(msg);
+ LOG_INFO("Received: " << msg
+ << " with payload '" << msg.getDataAsString() << "'");
+
+ consumer.acknowledge(msg);
+}
+
+client.close();
+```
+
+
+## Producer
+
+```c++
+Client client("pulsar://localhost:6650");
+
+Producer producer;
+Result result = client.createProducer("my-topic", producer);
+if (result != ResultOk) {
+ LOG_ERROR("Error creating producer: " << result);
+ return -1;
+}
+
+// Publish 10 messages to the topic
+for (int i = 0; i < 10; i++){
+ Message msg = MessageBuilder().setContent("my-message").build();
+ Result res = producer.send(msg);
+ LOG_INFO("Message sent: " << res);
+}
+client.close();
+```
+
+## Authentication
+
+```cpp
+ClientConfiguration config = ClientConfiguration();
+config.setUseTls(true);
+config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
+config.setTlsAllowInsecureConnection(false);
+config.setAuth(pulsar::AuthTls::create(
+ "/path/to/client-cert.pem", "/path/to/client-key.pem"););
+
+Client client("pulsar+ssl://my-broker.com:6651", config);
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-python.md b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-python.md
new file mode 100644
index 0000000..45df94a
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-python.md
@@ -0,0 +1,95 @@
+---
+id: version-2.1.1-incubating-client-libraries-python
+title: The Pulsar Python client
+sidebar_label: Python
+original_id: client-libraries-python
+---
+
+The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/incubator-pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code.
+
+## Installation
+
+You can install the [`pulsar-client`](https://pypi.python.org/pypi/pulsar-client) library either via [PyPi](https://pypi.python.org/pypi), using [pip](#installation-using-pip), or by building the library from source.
+
+### Installation using pip
+
+To install the `pulsar-client` library as a pre-built package using the [pip](https://pip.pypa.io/en/stable/) package manager:
+
+```shell
+$ pip install pulsar-client=={{pulsar:version_number}}
+```
+
+Installation via PyPi is available for the following Python versions:
+
+Platform | Supported Python versions
+:--------|:-------------------------
+MacOS <br /> 10.11 (El Capitan) — 10.12 (Sierra) — 10.13 (High Sierra) | 2.7, 3.7
+Linux | 2.7, 3.3, 3.4, 3.5, 3.6, 3.7
+
+### Installing from source
+
+To install the `pulsar-client` library by building from source, follow [these instructions](client-libraries-cpp.md#compilation) and compile the Pulsar C++ client library. That will also build the Python binding for the library.
+
+To install the built Python bindings:
+
+```shell
+$ git clone https://github.com/apache/pulsar
+$ cd pulsar/pulsar-client-cpp/python
+$ sudo python setup.py install
+```
+
+## API Reference
+
+The complete Python API reference is available at [api/python]({{site.baseUrl}}/api/python).
+
+## Examples
+
+Below you'll find a variety of Python code examples for the `pulsar-client` library.
+
+### Producer example
+
+This creates a Python producer for the `my-topic` topic and send 10 messages on that topic:
+
+```python
+import pulsar
+
+client = pulsar.Client('pulsar://localhost:6650')
+
+producer = client.create_producer('my-topic')
+
+for i in range(10):
+ producer.send(('Hello-%d' % i).encode('utf-8'))
+
+client.close()
+```
+
+### Consumer example
+
+This creates a consumer with the `my-subscription` subscription on the `my-topic` topic, listen for incoming messages, print the content and ID of messages that arrive, and acknowledge each message to the Pulsar broker:
+
+```python
+consumer = client.subscribe('my-topic', 'my-subscription')
+
+while True:
+ msg = consumer.receive()
+ print("Received message '{}' id='{}'".format(msg.data(), msg.message_id()))
+ consumer.acknowledge(msg)
+
+client.close()
+```
+
+### Reader interface example
+
+You can use the Pulsar Python API to use the Pulsar [reader interface](concepts-clients.md#reader-interface). Here's an example:
+
+```python
+# MessageId taken from a previously fetched message
+msg_id = msg.message_id()
+
+reader = client.create_reader('my-topic', msg_id)
+
+while True:
+ msg = reader.receive()
+ print("Received message '{}' id='{}'".format(msg.data(), msg.message_id()))
+ # No acknowledgment
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-tiered-storage.md b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-tiered-storage.md
new file mode 100644
index 0000000..03b74da
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-tiered-storage.md
@@ -0,0 +1,221 @@
+---
+id: version-2.1.1-incubating-cookbooks-tiered-storage
+title: Tiered Storage
+sidebar_label: Tiered Storage
+original_id: cookbooks-tiered-storage
+---
+
+Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded to long term storage, thereby freeing up space in BookKeeper and reducing storage costs. This cookbook walks you through using tiered storage in your Pulsar cluster.
+
+Tiered storage currently uses [Apache Jclouds](https://jclouds.apache.org) to supports
+[Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage/)(GCS for short)
+for long term storage. With Jclouds, it is easy to add support for more
+[cloud storage providers](https://jclouds.apache.org/reference/providers/#blobstore-providers) in the future.
+
+## When should I use Tiered Storage?
+
+Tiered storage should be used when you have a topic for which you want to keep a very long backlog for a long time. For example, if you have a topic containing user actions which you use to train your recommendation systems, you may want to keep that data for a long time, so that if you change your recommendation algorithm you can rerun it against your full user history.
+
+## The offloading mechanism
+
+A topic in Pulsar is backed by a log, known as a managed ledger. This log is composed of an ordered list of segments. Pulsar only every writes to the final segment of the log. All previous segments are sealed. The data within the segment is immutable. This is known as a segment oriented architecture.
+
+
+
+The Tiered Storage offloading mechanism takes advantage of this segment oriented architecture. When offloading is requested, the segments of the log are copied, one-by-one, to tiered storage. All segments of the log, apart from the segment currently being written to can be offloaded.
+
+On the broker, the administrator must configure the bucket and credentials for the cloud storage service.
+The configured bucket must exist before attempting to offload. If it does not exist, the offload operation will fail.
+
+Pulsar uses multi-part objects to upload the segment data. It is possible that a broker could crash while uploading the data.
+We recommend you add a life cycle rule your bucket to expire incomplete multi-part upload after a day or two to avoid
+getting charged for incomplete uploads.
+
+## Configuring the offload driver
+
+Offloading is configured in ```broker.conf```.
+
+At a minimum, the administrator must configure the driver, the bucket and the authenticating credentials.
+There is also some other knobs to configure, like the bucket region, the max block size in backed storage, etc.
+
+Currently we support driver of types:
+
+- `aws-s3`: [Simple Cloud Storage Service](https://aws.amazon.com/s3/)
+- `google-cloud-storage`: [Google Cloud Storage](https://cloud.google.com/storage/)
+
+> Driver names are case-insensitive for driver's name. There is a third driver type, `s3`, which is identical to `aws-s3`,
+> though it requires that you specify an endpoint url using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if
+> using a S3 compatible data store, other than AWS.
+
+```conf
+managedLedgerOffloadDriver=aws-s3
+```
+
+### "aws-s3" Driver configuration
+
+#### Bucket and Region
+
+Buckets are the basic containers that hold your data.
+Everything that you store in Cloud Storage must be contained in a bucket.
+You can use buckets to organize your data and control access to your data,
+but unlike directories and folders, you cannot nest buckets.
+
+```conf
+s3ManagedLedgerOffloadBucket=pulsar-topic-offload
+```
+
+Bucket Region is the region where bucket located. Bucket Region is not a required
+but a recommended configuration. If it is not configured, It will use the default region.
+
+With AWS S3, the default region is `US East (N. Virginia)`. Page
+[AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) contains more information.
+
+```conf
+s3ManagedLedgerOffloadRegion=eu-west-3
+```
+
+#### Authentication with AWS
+
+To be able to access AWS S3, you need to authenticate with AWS S3.
+Pulsar does not provide any direct means of configuring authentication for AWS S3,
+but relies on the mechanisms supported by the
+[DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html).
+
+Once you have created a set of credentials in the AWS IAM console, they can be configured in a number of ways.
+
+1. Set the environment variables **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** in ```conf/pulsar_env.sh```.
+
+```bash
+export AWS_ACCESS_KEY_ID=ABC123456789
+export AWS_SECRET_ACCESS_KEY=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c
+```
+
+> \"export\" is important so that the variables are made available in the environment of spawned processes.
+
+
+2. Add the Java system properties *aws.accessKeyId* and *aws.secretKey* to **PULSAR_EXTRA_OPTS** in `conf/pulsar_env.sh`.
+
+```bash
+PULSAR_EXTRA_OPTS="${PULSAR_EXTRA_OPTS} ${PULSAR_MEM} ${PULSAR_GC} -Daws.accessKeyId=ABC123456789 -Daws.secretKey=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024"
+```
+
+3. Set the access credentials in ```~/.aws/credentials```.
+
+```conf
+[default]
+aws_access_key_id=ABC123456789
+aws_secret_access_key=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c
+```
+
+If you are running in EC2 you can also use instance profile credentials, provided through the EC2 metadata service, but that is out of scope for this cookbook.
+
+> The broker must be rebooted for credentials specified in pulsar_env to take effect.
+
+#### Configuring the size of block read/write
+
+Pulsar also provides some knobs to configure the size of requests sent to AWS S3.
+
+- ```s3ManagedLedgerOffloadMaxBlockSizeInBytes``` configures the maximum size of
+ a "part" sent during a multipart upload. This cannot be smaller than 5MB. Default is 64MB.
+- ```s3ManagedLedgerOffloadReadBufferSizeInBytes``` configures the block size for
+ each individual read when reading back data from AWS S3. Default is 1MB.
+
+In both cases, these should not be touched unless you know what you are doing.
+
+### "google-cloud-storage" Driver configuration
+
+Buckets are the basic containers that hold your data. Everything that you store in
+Cloud Storage must be contained in a bucket. You can use buckets to organize your data and
+control access to your data, but unlike directories and folders, you cannot nest buckets.
+
+```conf
+gcsManagedLedgerOffloadBucket=pulsar-topic-offload
+```
+
+Bucket Region is the region where bucket located. Bucket Region is not a required but
+a recommended configuration. If it is not configured, It will use the default region.
+
+Regarding GCS, buckets are default created in the `us multi-regional location`,
+page [Bucket Locations](https://cloud.google.com/storage/docs/bucket-locations) contains more information.
+
+```conf
+gcsManagedLedgerOffloadRegion=europe-west3
+```
+
+#### Authentication with GCS
+
+The administrator needs to configure `gcsManagedLedgerOffloadServiceAccountKeyFile` in `broker.conf`
+for the broker to be able to access the GCS service. `gcsManagedLedgerOffloadServiceAccountKeyFile` is
+a Json file, containing the GCS credentials of a service account.
+[Service Accounts section of this page](https://support.google.com/googleapi/answer/6158849) contains
+more information of how to create this key file for authentication. More information about google cloud IAM
+is available [here](https://cloud.google.com/storage/docs/access-control/iam).
+
+Usually these are the steps to create the authentication file:
+1. Open the API Console Credentials page.
+2. If it's not already selected, select the project that you're creating credentials for.
+3. To set up a new service account, click New credentials and then select Service account key.
+4. Choose the service account to use for the key.
+5. Download the service account's public/private key as a JSON file that can be loaded by a Google API client library.
+
+```conf
+gcsManagedLedgerOffloadServiceAccountKeyFile="/Users/hello/Downloads/project-804d5e6a6f33.json"
+```
+
+#### Configuring the size of block read/write
+
+Pulsar also provides some knobs to configure the size of requests sent to GCS.
+
+- ```gcsManagedLedgerOffloadMaxBlockSizeInBytes``` configures the maximum size of a "part" sent
+ during a multipart upload. This cannot be smaller than 5MB. Default is 64MB.
+- ```gcsManagedLedgerOffloadReadBufferSizeInBytes``` configures the block size for each individual
+ read when reading back data from GCS. Default is 1MB.
+
+In both cases, these should not be touched unless you know what you are doing.
+
+## Configuring offload to run automatically
+
+Namespace policies can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that the topic has stored on the pulsar cluster. Once the topic reaches the threshold, an offload operation will be triggered. Setting a negative value to the threshold will disable automatic offloading. Setting the threshold to 0 will cause the broker to offload data as soon as it possiby can.
+
+```bash
+$ bin/pulsar-admin namespaces set-offload-threshold --size 10M my-tenant/my-namespace
+```
+
+> Automatic offload runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, offload will not until the current segment is full.
+
+
+## Triggering offload manually
+
+Offloading can manually triggered through a REST endpoint on the Pulsar broker. We provide a CLI which will call this rest endpoint for you.
+
+When triggering offload, you must specify the maximum size, in bytes, of backlog which will be retained locally on the bookkeeper. The offload mechanism will offload segments from the start of the topic backlog until this condition is met.
+
+```bash
+$ bin/pulsar-admin topics offload --size-threshold 10M my-tenant/my-namespace/topic1
+Offload triggered for persistent://my-tenant/my-namespace/topic1 for messages before 2:0:-1
+```
+
+The command to triggers an offload will not wait until the offload operation has completed. To check the status of the offload, use offload-status.
+
+```bash
+$ bin/pulsar-admin topics offload-status my-tenant/my-namespace/topic1
+Offload is currently running
+```
+
+To wait for offload to complete, add the -w flag.
+
+```bash
+$ bin/pulsar-admin topics offload-status -w my-tenant/my-namespace/topic1
+Offload was a success
+```
+
+If there is an error offloading, the error will be propagated to the offload-status command.
+
+```bash
+$ bin/pulsar-admin topics offload-status persistent://public/default/topic1
+Error in offload
+null
+
+Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads. Please authenticate. (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 798758DE3F1776DF; S3 Extended Request ID: dhBFz/lZm1oiG/oBEepeNlhrtsDlzoOhocuYMpKihQGXe6EG8puRGOkK6UwqzVrMXTWBxxHcS+g=), S3 Extended Request ID: dhBFz/lZm1oiG/oBEepeNlhr [...]
+````
+
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md
new file mode 100644
index 0000000..4de56d9
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md
@@ -0,0 +1,357 @@
+---
+id: version-2.1.1-incubating-deploy-bare-metal
+title: Deploying a cluster on bare metal
+sidebar_label: Bare metal
+original_id: deploy-bare-metal
+---
+
+
+> ### Tips
+>
+> 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with
+> Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance,
+> however, see the guide [here](deploy-bare-metal-multi-cluster.md).
+>
+> 2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors`
+> package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you
+> have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md).
+
+Deploying a Pulsar cluster involves doing the following (in order):
+
+* Deploying a [ZooKeeper](#deploying-a-zookeeper-cluster) cluster (optional)
+* Initializing [cluster metadata](#initializing-cluster-metadata)
+* Deploying a [BookKeeper](#deploying-a-bookkeeper-cluster) cluster
+* Deploying one or more Pulsar [brokers](#deploying-pulsar-brokers)
+
+## Preparation
+
+### Requirements
+
+> If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines
+> for running ZooKeeper.
+
+To run Pulsar on bare metal, you will need:
+
+* At least 6 Linux machines or VMs
+ * 3 running [ZooKeeper](https://zookeeper.apache.org)
+ * 3 running a Pulsar broker, and a [BookKeeper](https://bookkeeper.apache.org) bookie
+* A single [DNS](https://en.wikipedia.org/wiki/Domain_Name_System) name covering all of the Pulsar broker hosts
+
+Each machine in your cluster will need to have [Java 8](http://www.oracle.com/technetwork/java/javase/downloads/index.html) or higher installed.
+
+Here's a diagram showing the basic setup:
+
+
+
+In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case `pulsar-cluster.acme.com`, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.
+
+### Hardware considerations
+
+When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.
+
+#### ZooKeeper
+
+For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, *not* for basic operations. If you're running Pulsar on [Amazon Web Services](https://aws.amazon.com/) (AWS), for example, a [t2.small](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html) instance would likely suffice.
+
+#### Bookies & Brokers
+
+For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, [i3.4xlarge](https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/) instances may be appropriate. On those machines we also recommend:
+
+* Fast CPUs and 10Gbps [NIC](https://en.wikipedia.org/wiki/Network_interface_controller) (for Pulsar brokers)
+* Small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache (for BookKeeper bookies)
+
+## Installing the Pulsar binary package
+
+> You'll need to install the Pulsar binary package on *each machine in the cluster*, including machines running [ZooKeeper](#deploying-a-zookeeper-cluster) and [BookKeeper](#deploying-a-bookkeeper-cluster).
+
+To get started deploying a Pulsar cluster on bare metal, you'll need to download a binary tarball release in one of the following ways:
+
+* By clicking on the link directly below, which will automatically trigger a download:
+ * <a href="pulsar:binary_release_url" download>Pulsar {{pulsar:version}} binary release</a>
+* From the Pulsar [downloads page](pulsar:download_page_url)
+* From the Pulsar [releases page](https://github.com/apache/incubator-pulsar/releases/latest) on [GitHub](https://github.com)
+* Using [wget](https://www.gnu.org/software/wget):
+
+```bash
+$ wget pulsar:binary_release_url
+```
+
+Once you've downloaded the tarball, untar it and `cd` into the resulting directory:
+
+```bash
+$ tar xvzf apache-pulsar-{{pulsar:version}}-bin.tar.gz
+$ cd apache-pulsar-{{pulsar:version}}
+```
+
+The untarred directory contains the following subdirectories:
+
+Directory | Contains
+:---------|:--------
+`bin` | Pulsar's [command-line tools](reference-cli-tools.md), such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md)
+`conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more
+`data` | The data storage directory used by ZooKeeper and BookKeeper.
+`lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar.
+`logs` | Logs created by the installation.
+
+## Installing Builtin Connectors (optional)
+
+> Since release `2.1.0-incubating`, Pulsar releases a separate binary distribution, containing all the `builtin` connectors.
+> If you would like to enable those `builtin` connectors, you can follow the instructions as below; otherwise you can
+> skip this section for now.
+
+To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in
+one of the following ways:
+
+* by clicking the link below and downloading the release from an Apache mirror:
+
+ * <a href="pulsar:connector_release_url" download>Pulsar IO Connectors {{pulsar:version}} release</a>
+
+* from the Pulsar [downloads page](pulsar:download_page_url)
+* from the Pulsar [releases page](https://github.com/apache/incubator-pulsar/releases/latest)
+* using [wget](https://www.gnu.org/software/wget):
+
+ ```shell
+ $ wget pulsar:connector_release_url
+ ```
+
+Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as `connectors`
+in the pulsar directory:
+
+```bash
+$ tar xvfz apache-pulsar-io-connectors-{{pulsar:version}}-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-{{pulsar:version}}` in the pulsar directory
+// then copy the connectors
+
+$ mv apache-pulsar-io-connectors-{{pulsar:version}}/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar:version}}.nar
+pulsar-io-cassandra-{{pulsar:version}}.nar
+pulsar-io-kafka-{{pulsar:version}}.nar
+pulsar-io-kinesis-{{pulsar:version}}.nar
+pulsar-io-rabbitmq-{{pulsar:version}}.nar
+pulsar-io-twitter-{{pulsar:version}}.nar
+...
+```
+
+## Deploying a ZooKeeper cluster
+
+> If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.
+
+[ZooKeeper](https://zookeeper.apache.org) manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.
+
+To begin, add all ZooKeeper servers to the configuration specified in [`conf/zookeeper.conf`](reference-configuration.md#zookeeper) (in the Pulsar directory you created [above](#installing-the-pulsar-binary-package)). Here's an example:
+
+```properties
+server.1=zk1.us-west.example.com:2888:3888
+server.2=zk2.us-west.example.com:2888:3888
+server.3=zk3.us-west.example.com:2888:3888
+```
+
+On each host, you need to specify the ID of the node in each node's `myid` file, which is in each server's `data/zookeeper` folder by default (this can be changed via the [`dataDir`](reference-configuration.md#zookeeper-dataDir) parameter).
+
+> See the [Multi-server setup guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup) in the ZooKeeper documentation for detailed info on `myid` and more.
+
+On a ZooKeeper server at `zk1.us-west.example.com`, for example, you could set the `myid` value like this:
+
+```bash
+$ mkdir -p data/zookeeper
+$ echo 1 > data/zookeeper/myid
+```
+
+On `zk2.us-west.example.com` the command would be `echo 2 > data/zookeeper/myid` and so on.
+
+Once each server has been added to the `zookeeper.conf` configuration and has the appropriate `myid` entry, you can start ZooKeeper on all hosts (in the background, using nohup) with the [`pulsar-daemon`](reference-cli-tools.md#pulsar-daemon) CLI tool:
+
+```bash
+$ bin/pulsar-daemon start zookeeper
+```
+
+## Initializing cluster metadata
+
+Once you've deployed ZooKeeper for your cluster, there is some metadata that needs to be written to ZooKeeper for each cluster in your instance. It only needs to be written **once**.
+
+You can initialize this metadata using the [`initialize-cluster-metadata`](reference-cli-tools.md#pulsar-initialize-cluster-metadata) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool. This command can be run on any machine in your ZooKeeper cluster. Here's an example:
+
+```shell
+$ bin/pulsar initialize-cluster-metadata \
+ --cluster pulsar-cluster-1 \
+ --zookeeper zk1.us-west.example.com:2181 \
+ --configuration-store zk1.us-west.example.com:2181 \
+ --web-service-url http://pulsar.us-west.example.com:8080 \
+ --web-service-url-tls https://pulsar.us-west.example.com:8443 \
+ --broker-service-url pulsar://pulsar.us-west.example.com:6650 \
+ --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651
+```
+
+As you can see from the example above, the following needs to be specified:
+
+Flag | Description
+:----|:-----------
+`--cluster` | A name for the cluster
+`--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster.
+`--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster.
+`--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port).
+`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port).
+`--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port).
+`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port).
+
+## Deploying a BookKeeper cluster
+
+[BookKeeper](https://bookkeeper.apache.org) handles all persistent data storage in Pulsar. You will need to deploy a cluster of BookKeeper bookies to use Pulsar. We recommend running a **3-bookie BookKeeper cluster**.
+
+BookKeeper bookies can be configured using the [`conf/bookkeeper.conf`](reference-configuration.md#bookkeeper) configuration file. The most important step in configuring bookies for our purposes here is ensuring that the [`zkServers`](reference-configuration.md#bookkeeper-zkServers) is set to the connection string for the ZooKeeper cluster. Here's an example:
+
+```properties
+zkServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
+```
+
+Once you've appropriately modified the `zkServers` parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters [here](reference-configuration.md#bookkeeper), although we would recommend consulting the [BookKeeper documentation](http://bookkeeper.apache.org/docs/latest/reference/config/) for a more in-depth guide.
+
+> ##### NOTES
+>
+> Since Pulsar 2.1.0 release, Pulsar introduces [stateful function](functions-state.md) for Pulsar Functions. If you would like to enable that feature,
+> you need to enable table service on BookKeeper by setting following setting in `conf/bookkeeper.conf` file.
+>
+> ```conf
+> extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
+> ```
+
+Once you've applied the desired configuration in `conf/bookkeeper.conf`, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using [nohup](https://en.wikipedia.org/wiki/Nohup), or in the foreground.
+
+To start the bookie in the background, use the [`pulsar-daemon`](reference-cli-tools.md#pulsar-daemon) CLI tool:
+
+```bash
+$ bin/pulsar-daemon start bookie
+```
+
+To start the bookie in the foreground:
+
+```bash
+$ bin/bookkeeper bookie
+```
+
+You can verify that a bookie is working properly by running the `bookiesanity` command for the [BookKeeper shell](reference-cli-tools.md#shell) on it:
+
+```bash
+$ bin/bookkeeper shell bookiesanity
+```
+
+This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.
+
+After you have started all the bookies, you can use `simpletest` command for [BookKeeper shell](reference-cli-tools.md#shell) on any bookie node, to
+verify all the bookies in the cluster are up running.
+
+```bash
+$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+```
+
+This command will create a `num-bookies` sized ledger on the cluster, write a few entries, and finally delete the ledger.
+
+
+## Deploying Pulsar brokers
+
+Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running **3 brokers**, one for each machine that's already running a BookKeeper bookie.
+
+### Configuring Brokers
+
+The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the [`zookeeperServers`](reference-configuration.md#broker-zookeeperServers) and [`configurationStoreServers`](reference-configuration.md#broker-configurationStoreServers) parameters. In this case, since we only have 1 cluster and no configuration store setup, the `configurationStoreServers` will point to the same `zookeeperServers`.
+
+```properties
+zookeeperServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
+configurationStoreServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
+```
+
+You also need to specify the cluster name (matching the name that you provided when [initializing the cluster's metadata](#initializing-cluster-metadata):
+
+```properties
+clusterName=pulsar-cluster-1
+```
+
+### Enabling Pulsar Functions (optional)
+
+If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below:
+
+1. Edit `conf/broker.conf` to enable function worker, by setting `functionsWorkerEnabled` to `true`.
+
+ ```conf
+ functionsWorkerEnabled=true
+ ```
+
+2. Edit `conf/functions_worker.yml` and set `pulsarFunctionsCluster` to the cluster name that you provided when [initializing the cluster's metadata](#initializing-cluster-metadata).
+
+ ```conf
+ pulsarFunctionsCluster=pulsar-cluster-1
+ ```
+
+### Starting Brokers
+
+You can then provide any other configuration changes that you'd like in the [`conf/broker.conf`](reference-configuration.md#broker) file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.
+
+You can start a broker in the foreground using the [`pulsar broker`](reference-cli-tools.md#pulsar-broker) command:
+
+```bash
+$ bin/pulsar broker
+```
+
+You can start a broker in the background using the [`pulsar-daemon`](reference-cli-tools.md#pulsar-daemon) CLI tool:
+
+```bash
+$ bin/pulsar-daemon start broker
+```
+
+Once you've succesfully started up all the brokers you intend to use, your Pulsar cluster should be ready to go!
+
+## Connecting to the running cluster
+
+Once your Pulsar cluster is up and running, you should be able to connect with it using Pulsar clients. One such client is the [`pulsar-client`](reference-cli-tools.md#pulsar-client) tool, which is included with the Pulsar binary package. The `pulsar-client` tool can publish messages to and consume messages from Pulsar topics and thus provides a simple way to make sure that your cluster is runnning properly.
+
+To use the `pulsar-client` tool, first modify the client configuration file in [`conf/client.conf`](reference-configuration.md#client) in your binary package. You'll need to change the values for `webServiceUrl` and `brokerServiceUrl`, substituting `localhost` (which is the default), with the DNS name that you've assigned to your broker/bookie hosts. Here's an example:
+
+```properties
+webServiceUrl=http://us-west.example.com:8080/
+brokerServiceurl=pulsar://us-west.example.com:6650/
+```
+
+Once you've done that, you can publish a message to Pulsar topic:
+
+```bash
+$ bin/pulsar-client produce \
+ persistent://public/default/test \
+ -n 1 \
+ -m "Hello, Pulsar"
+```
+
+> You may need to use a different cluster name in the topic if you specified a cluster name different from `pulsar-cluster-1`.
+
+This will publish a single message to the Pulsar topic.
+
+## Running Functions
+
+> If you have [enabled](#enabling-pulsar-functions-optional) Pulsar Functions, you can also tryout pulsar functions now.
+
+Create a ExclamationFunction `exclamation`.
+
+```bash
+bin/pulsar-admin functions create \
+ --jar examples/api-examples.jar \
+ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+```
+
+Check if the function is running as expected by [triggering](functions-deploying.md#triggering-pulsar-functions) the function.
+
+```bash
+bin/pulsar-admin functions trigger --name exclamation --trigger-value "hello world"
+```
+
+You will see output as below:
+
+```shell
+hello world!
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-api.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-api.md
new file mode 100644
index 0000000..197a1fb
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-api.md
@@ -0,0 +1,721 @@
+---
+id: version-2.1.1-incubating-functions-api
+title: The Pulsar Functions API
+sidebar_label: API
+original_id: functions-api
+---
+
+[Pulsar Functions](functions-overview.md) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine.
+
+> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview.md).
+
+## Core programming model
+
+Pulsar Functions provide a wide range of functionality but are based on a very simple programming model. You can think of Pulsar Functions as lightweight processes that
+
+* consume messages from one or more Pulsar topics and then
+* apply some user-defined processing logic to each incoming message. That processing logic could be just about anything you want, including
+ * producing the resulting, processed message on another Pulsar topic, or
+ * doing something else with the message, such as writing results to an external database.
+
+You could use Pulsar Functions, for example, to set up the following processing chain:
+
+* A [Python](#functions-for-python) function listens on the `raw-sentences` topic and "[sanitizes](#example-function)" incoming strings (removing extraneous whitespace and converting all characters to lower case) and then publishes the results to a `sanitized-sentences` topic
+* A [Java](#functions-for-java) function listens on the `sanitized-sentences` topic, counts the number of times each word appears within a specified time window, and publishes the results to a `results` topic
+* Finally, a Python function listens on the `results` topic and writes the results to a MySQL table
+
+### Example function
+
+Here's an example "input sanitizer" function written in Python and stored in a `sanitizer.py` file:
+
+```python
+def clean_string(s):
+ return s.strip().lower()
+
+def process(input):
+ return clean_string(input)
+```
+
+Some things to note about this Pulsar Function:
+
+* There is no client, producer, or consumer object involved. All message "plumbing" is already taken care of for you, enabling you to worry only about processing logic.
+* No topics, subscription types, tenants, or namespaces are specified in the function logic itself. Instead, topics are specified upon [deployment](#example-deployment). This means that you can use and re-use Pulsar Functions across topics, tenants, and namespaces without needing to hard-code those attributes.
+
+### Example deployment
+
+Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+ --py sanitizer.py \ # The Python file with the function's code
+ --classname sanitizer \ # The class or function holding the processing logic
+ --tenant public \ # The function's tenant (derived from the topic name by default)
+ --namespace default \ # The function's namespace (derived from the topic name by default)
+ --name sanitizer-function \ # The name of the function (the class name by default)
+ --inputs dirty-strings-in \ # The input topic(s) for the function
+ --output clean-strings-out \ # The output topic for the function
+ --log-topic sanitizer-logs # The topic to which all functions logs are published
+```
+
+For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying.md) guide.
+
+### Available APIs
+
+In both Java and Python, you have two options for writing Pulsar Functions:
+
+Interface | Description | Use cases
+:---------|:------------|:---------
+Language-native interface | No Pulsar-specific libraries or special dependencies required (only core libraries from Java/Python) | Functions that don't require access to the function's [context](#context)
+Pulsar Function SDK for Java/Python | Pulsar-specific libraries that provide a range of functionality not provided by "native" interfaces | Functions that require access to the function's [context](#context)
+
+In Python, for example, this language-native function, which adds an exclamation point to all incoming strings and publishes the resulting string to a topic, would have no external dependencies:
+
+```python
+def process(input):
+ return "{}!".format(input)
+```
+
+This function, however, would use the Pulsar Functions [SDK for Python](#python-sdk-functions):
+
+```python
+from pulsar import Function
+
+class DisplayFunctionName(Function):
+ def process(self, input, context):
+ function_name = context.function_name()
+ return "The function processing this message has the name {0}".format(function_name)
+```
+
+### Functions, Messages and Message Types
+
+Pulsar Functions can take byte arrays as inputs and spit out byte arrays as output. However in languages that support typed interfaces(just Java at the moment) one can write typed Functions as well. In this scenario, there are two ways one can bind messages to types.
+* [Schema Registry](#Schema-Registry)
+* [SerDe](#SerDe)
+
+### Schema Registry
+Pulsar has a built in [Schema Registry](concepts-schema-registry) and comes bundled with a variety of popular schema types(avro, json and protobuf). Pulsar Functions can leverage existing schema information from input topics to derive the input type. The same applies for output topic as well.
+
+### SerDe
+
+SerDe stands for **Ser**ialization and **De**serialization. All Pulsar Functions use SerDe for message handling. How SerDe works by default depends on the language you're using for a particular function:
+
+* In [Python](#python-serde), the default SerDe is identity, meaning that the type is serialized as whatever type the producer function returns
+* In [Java](#java-serde), a number of commonly used types (`String`s, `Integer`s, etc.) are supported by default
+
+In both languages, however, you can write your own custom SerDe logic for more complex, application-specific types. See the docs for [Java](#java-serde) and [Python](#python-serde) for language-specific instructions.
+
+### Context
+
+Both the [Java](#java-sdk-functions) and [Python](#python-sdk-functions) SDKs provide access to a **context object** that can be used by the function. This context object provides a wide variety of information and functionality to the function:
+
+* The name and ID of the Pulsar Function
+* The message ID of each message. Each Pulsar message is automatically assigned an ID.
+* The name of the topic on which the message was sent
+* The names of all input topics as well as the output topic associated with the function
+* The name of the class used for [SerDe](#serialization-and-deserialization-serde)
+* The [tenant](reference-terminology.md#tenant) and namespace associated with the function
+* The ID of the Pulsar Functions instance running the function
+* The version of the function
+* The [logger object](functions-overview.md#logging) used by the function, which can be used to create function log messages
+* Access to arbitrary [user config](#user-config) values supplied via the CLI
+* An interface for recording [metrics](functions-metrics.md)
+* An interface for storing and retrieving state in [state storage](functions-overview.md#state-storage)
+
+### User config
+
+When you run or update Pulsar Functions created using the [SDK](#available-apis), you can pass arbitrary key/values to them via the command line with the `--userConfig` flag. Key/values must be specified as JSON. Here's an example of a function creation command that passes a user config key/value to a function:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --name word-filter \
+ # Other function configs
+ --user-config '{"forbidden-word":"rosebud"}'
+```
+
+If the function were a Python function, that config value could be accessed like this:
+
+```python
+from pulsar import Function
+
+class WordFilter(Function):
+ def process(self, context, input):
+ forbidden_word = context.user_config()["forbidden-word"]
+
+ # Don't publish the message if it contains the user-supplied
+ # forbidden word
+ if forbidden_word in input:
+ pass
+ # Otherwise publish the message
+ else:
+ return input
+```
+
+## Functions for Java
+
+Writing Pulsar Functions in Java involves implementing one of two interfaces:
+
+* The [`java.util.Function`](https://docs.oracle.com/javase/8/docs/api/java/util/function/Function.html) interface
+* The {@inject: javadoc:Function:/pulsar-functions/org/apache/pulsar/functions/api/Function} interface. This interface works much like the `java.util.Function` interface, but with the important difference that it provides a {@inject: javadoc:Context:/pulsar-functions/org/apache/pulsar/functions/api/Context} object that you can use in a [variety of ways](#context)
+
+### Getting started
+
+In order to write Pulsar Functions in Java, you'll need to install the proper [dependencies](#dependencies) and package your function [as a JAR](#packaging).
+
+#### Dependencies
+
+How you get started writing Pulsar Functions in Java depends on which API you're using:
+
+* If you're writing a [Java native function](#java-native-functions), you won't need any external dependencies.
+* If you're writing a [Java SDK function](#java-sdk-functions), you'll need to import the `pulsar-functions-api` library.
+
+ Here's an example for a Maven `pom.xml` configuration file:
+
+ ```xml
+ <dependency>
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>pulsar-functions-api</artifactId>
+ <version>2.0.0-incubating-SNAPSHOT</version>
+ </dependency>
+ ```
+
+ Here's an example for a Gradle `build.gradle` configuration file:
+
+ ```groovy
+ dependencies {
+ compile group: 'org.apache.pulsar', name: 'pulsar-functions-api', version: '2.0.0-incubating-SNAPSHOT'
+ }
+ ```
+
+#### Packaging
+
+Whether you're writing Java Pulsar Functions using the [native](#java-native-functions) Java `java.util.Function` interface or using the [Java SDK](#java-sdk-functions), you'll need to package your function(s) as a "fat" JAR.
+
+> #### Starter repo
+> If you'd like to get up and running quickly, you can use [this repo](https://github.com/streamlio/pulsar-functions-java-starter), which contains the necessary Maven configuration to build a fat JAR as well as some example functions.
+
+### Java native functions
+
+If your function doesn't require access to its [context](#context), you can create a Pulsar Function by implementing the [`java.util.Function`](https://docs.oracle.com/javase/8/docs/api/java/util/function/Function.html) interface, which has this very simple, single-method signature:
+
+```java
+public interface Function<I, O> {
+ O apply(I input);
+}
+```
+
+Here's an example function that takes a string as its input, adds an exclamation point to the end of the string, and then publishes the resulting string:
+
+```java
+import java.util.Function;
+
+public class ExclamationFunction implements Function<String, String> {
+ @Override
+ public String process(String input) {
+ return String.format("%s!", input);
+ }
+}
+```
+
+In general, you should use native functions when you don't need access to the function's [context](#context). If you *do* need access to the function's context, then we recommend using the [Pulsar Functions Java SDK](#java-sdk-functions).
+
+#### Java native examples
+
+There is one example Java native function in this {@inject: github:folder:/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples}:
+
+* {@inject: github:`JavaNativeExclamationFunction`:/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/JavaNativeExclamationFunction.java}
+
+### Java SDK functions
+
+To get started developing Pulsar Functions using the Java SDK, you'll need to add a dependency on the `pulsar-functions-api` artifact to your project. Instructions can be found [above](#dependencies).
+
+> An easy way to get up and running with Pulsar Functions in Java is to clone the [`pulsar-functions-java-starter`](https://github.com/streamlio/pulsar-functions-java-starter) repo and follow the instructions there.
+
+
+#### Java SDK examples
+
+There are several example Java SDK functions in this {@inject: github:folder:/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples}:
+
+Function name | Description
+:-------------|:-----------
+[`ContextFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/ContextFunction.java) | Illustrates [context](#context)-specific functionality like [logging](#java-logging) and [metrics](#java-metrics)
+[`WordCountFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/WordCountFunction.java) | Illustrates usage of Pulsar Function [state-storage](functions-overview.md#state-storage)
+[`ExclamationFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/ExclamationFunction.java) | A basic string manipulation function for the Java SDK
+[`LoggingFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/LoggingFunction.java) | A function that shows how [logging](#java-logging) works for Java
+[`PublishFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/PublishFunction.java) | Publishes results to a topic specified in the function's [user config](#java-user-config) (rather than on the function's output topic)
+[`UserConfigFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/UserConfigFunction.java) | A function that consumes [user-supplied configuration](#java-user-config) values
+[`UserMetricFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/UserMetricFunction.java) | A function that records metrics
+[`VoidFunction`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/UserMetricFunction.java) | A simple [void function](#void-functions)
+
+### Java context object
+
+The {@inject: javadoc:Context:/client/org/apache/pulsar/functions/api/Context} interface provides a number of methods that you can use to access the function's [context](#context). The various method signatures for the `Context` interface are listed below:
+
+```java
+public interface Context {
+ Record<?> getCurrentRecord();
+ Collection<String> getInputTopics();
+ String getOutputTopic();
+ String getOutputSchemaType();
+ String getTenant();
+ String getNamespace();
+ String getFunctionName();
+ String getFunctionId();
+ String getInstanceId();
+ String getFunctionVersion();
+ Logger getLogger();
+ void incrCounter(String key, long amount);
+ long getCounter(String key);
+ void putState(String key, ByteBuffer value);
+ ByteBuffer getState(String key);
+ Map<String, Object> getUserConfigMap();
+ Optional<Object> getUserConfigValue(String key);
+ Object getUserConfigValueOrDefault(String key, Object defaultValue);
+ void recordMetric(String metricName, double value);
+ <O> CompletableFuture<Void> publish(String topicName, O object, String schemaOrSerdeClassName);
+ <O> CompletableFuture<Void> publish(String topicName, O object);
+}
+```
+
+Here's an example function that uses several methods available via the `Context` object:
+
+```java
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+import org.slf4j.Logger;
+
+import java.util.stream.Collectors;
+
+public class ContextFunction implements Function<String, Void> {
+ public Void process(String input, Context context) {
+ Logger LOG = context.getLogger();
+ String inputTopics = context.getInputTopics().stream().collect(Collectors.joining(", "));
+ String functionName = context.getFunctionName();
+
+ String logMessage = String.format("A message with a value of \"%s\" has arrived on one of the following topics: %s\n",
+ input,
+ inputTopics);
+
+ LOG.info(logMessage);
+
+ String metricName = String.format("function-%s-messages-received", functionName);
+ context.recordMetric(metricName, 1);
+
+ return null;
+ }
+}
+```
+
+### Void functions
+
+Pulsar Functions can publish results to an output topic, but this isn't required. You can also have functions that simply produce a log, write results to a database, etc. Here's a function that writes a simple log every time a message is received:
+
+```java
+import org.slf4j.Logger;
+
+public class LogFunction implements PulsarFunction<String, Void> {
+ public String apply(String input, Context context) {
+ Logger LOG = context.getLogger();
+ LOG.info("The following message was received: {}", input);
+ return null;
+ }
+}
+```
+
+> When using Java functions in which the output type is `Void`, the function must *always* return `null`.
+
+### Java SerDe
+
+Pulsar Functions use [SerDe](#serialization-and-deserialization-serde) when publishing data to and consuming data from Pulsar topics. When you're writing Pulsar Functions in Java, the following basic Java types are built in and supported by default:
+
+* `String`
+* `Double`
+* `Integer`
+* `Float`
+* `Long`
+* `Short`
+* `Byte`
+
+Built-in vs. custom. For custom, you need to implement this interface:
+
+```java
+public interface SerDe<T> {
+ T deserialize(byte[] input);
+ byte[] serialize(T input);
+}
+```
+
+#### Java SerDe example
+
+Imagine that you're writing Pulsar Functions in Java that are processing tweet objects. Here's a simple example `Tweet` class:
+
+```java
+public class Tweet {
+ private String username;
+ private String tweetContent;
+
+ public Tweet(String username, String tweetContent) {
+ this.username = username;
+ this.tweetContent = tweetContent;
+ }
+
+ // Standard setters and getters
+}
+```
+
+In order to be able to pass `Tweet` objects directly between Pulsar Functions, you'll need to provide a custom SerDe class. In the example below, `Tweet` objects are basically strings in which the username and tweet content are separated by a `|`.
+
+```java
+package com.example.serde;
+
+import org.apache.pulsar.functions.api.SerDe;
+
+import java.util.regex.Pattern;
+
+public class TweetSerde implements SerDe<Tweet> {
+ public Tweet deserialize(byte[] input) {
+ String s = new String(input);
+ String[] fields = s.split(Pattern.quote("|"));
+ return new Tweet(fields[0], fields[1]);
+ }
+
+ public byte[] serialize(Tweet input) {
+ return "%s|%s".format(input.getUsername(), input.getTweetContent()).getBytes();
+ }
+}
+```
+
+To apply this custom SerDe to a particular Pulsar Function, you would need to:
+
+* Package the `Tweet` and `TweetSerde` classes into a JAR
+* Specify a path to the JAR and SerDe class name when deploying the function
+
+Here's an example [`create`](reference-pulsar-admin.md#create-1) operation:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --jar /path/to/your.jar \
+ --output-serde-classname com.example.serde.TweetSerde \
+ # Other function attributes
+```
+
+> #### Custom SerDe classes must be packaged with your function JARs
+> Pulsar does not store your custom SerDe classes separately from your Pulsar Functions. That means that you'll need to always include your SerDe classes in your function JARs. If not, Pulsar will return an error.
+
+### Java logging
+
+Pulsar Functions that use the [Java SDK](#java-sdk-functions) have access to an [SLF4j](https://www.slf4j.org/) [`Logger`](https://www.slf4j.org/api/org/apache/log4j/Logger.html) object that can be used to produce logs at the chosen log level. Here's a simple example function that logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`:
+
+```java
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+import org.slf4j.Logger;
+
+public class LoggingFunction implements Function<String, Void> {
+ @Override
+ public void apply(String input, Context context) {
+ Logger LOG = context.getLogger();
+ String messageId = new String(context.getMessageId());
+
+ if (input.contains("danger")) {
+ LOG.warn("A warning was received in message {}", messageId);
+ } else {
+ LOG.info("Message {} received\nContent: {}", messageId, input);
+ }
+
+ return null;
+ }
+}
+```
+
+If you want your function to produce logs, you need to specify a log topic when creating or running the function. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --jar my-functions.jar \
+ --classname my.package.LoggingFunction \
+ --log-topic persistent://public/default/logging-function-logs \
+ # Other function configs
+```
+
+Now, all logs produced by the `LoggingFunction` above can be accessed via the `persistent://public/default/logging-function-logs` topic.
+
+### Java user config
+
+The Java SDK's [`Context`](#context) object enables you to access key/value pairs provided to the Pulsar Function via the command line (as JSON). Here's an example function creation command that passes a key/value pair:
+
+```bash
+$ bin/pulsar-admin functions create \
+ # Other function configs
+ --user-config '{"word-of-the-day":"verdure"}'
+```
+
+To access that value in a Java function:
+
+```java
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+import org.slf4j.Logger;
+
+import java.util.Optional;
+
+public class UserConfigFunction implements Function<String, Void> {
+ @Override
+ public void apply(String input, Context context) {
+ Logger LOG = context.getLogger();
+ Optional<String> wotd = context.getUserConfigValue("word-of-the-day");
+ if (wotd.isPresent()) {
+ LOG.info("The word of the day is {}", wotd);
+ } else {
+ LOG.warn("No word of the day provided");
+ }
+ return null;
+ }
+}
+```
+
+The `UserConfigFunction` function will log the string `"The word of the day is verdure"` every time the function is invoked (i.e. every time a message arrives). The `word-of-the-day` user config will be changed only when the function is updated with a new config value via the command line.
+
+You can also access the entire user config map or set a default value in case no value is present:
+
+```java
+// Get the whole config map
+Map<String, String> allConfigs = context.getUserConfigMap();
+
+// Get value or resort to default
+String wotd = context.getUserConfigValueOrDefault("word-of-the-day", "perspicacious");
+```
+
+> For all key/value pairs passed to Java Pulsar Functions, both the key *and* the value are `String`s. If you'd like the value to be of a different type, you will need to deserialize from the `String` type.
+
+### Java metrics
+
+You can record metrics using the [`Context`](#context) object on a per-key basis. You can, for example, set a metric for the key `process-count` and a different metric for the key `elevens-count` every time the function processes a message. Here's an example:
+
+```java
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+
+public class MetricRecorderFunction implements Function<Integer, Void> {
+ @Override
+ public void apply(Integer input, Context context) {
+ // Records the metric 1 every time a message arrives
+ context.recordMetric("hit-count", 1);
+
+ // Records the metric only if the arriving number equals 11
+ if (input == 11) {
+ context.recordMetric("elevens-count", 1);
+ }
+
+ return null;
+ }
+}
+```
+
+> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.
+
+
+## Functions for Python
+
+Writing Pulsar Functions in Python entails implementing one of two things:
+
+* A `process` function that takes an input (message data from the function's input topic(s)), applies some kind of logic to it, and either returns an object (to be published to the function's output topic) or `pass`es and thus doesn't produce a message
+* A `Function` class that has a `process` method that provides a message input to process and a [context](#context) object
+
+### Getting started
+
+Regardless of which [deployment mode](functions-deploying.md) you're using, 'pulsar-client' python library has to installed on any machine that's running Pulsar Functions written in Python.
+
+That could be your local machine for [local run mode](functions-deploying.md#local-run-mode) or a machine running a Pulsar [broker](reference-terminology.md#broker) for [cluster mode](functions-deploying.md#cluster-mode). To install those libraries using pip:
+
+```bash
+$ pip install pulsar-client
+```
+
+### Packaging
+
+At the moment, the code for Pulsar Functions written in Python must be contained within a single Python file. In the future, Pulsar Functions may support other packaging formats, such as [**P**ython **EX**ecutables](https://github.com/pantsbuild/pex) (PEXes).
+
+### Python native functions
+
+If your function doesn't require access to its [context](#context), you can create a Pulsar Function by implementing a `process` function, which provides a single input object that you can process however you wish. Here's an example function that takes a string as its input, adds an exclamation point at the end of the string, and then publishes the resulting string:
+
+```python
+def process(input):
+ return "{0}!".format(input)
+```
+
+In general, you should use native functions when you don't need access to the function's [context](#context). If you *do* need access to the function's context, then we recommend using the [Pulsar Functions Python SDK](#python-sdk-functions).
+
+#### Python native examples
+
+There is one example Python native function in this {@inject: github:folder:/pulsar-functions/python-examples}:
+
+* {@inject: github:`native_exclamation_function.py`:/pulsar-functions/python-examples/native_exclamation_function.py}
+
+### Python SDK functions
+
+To get started developing Pulsar Functions using the Python SDK, you'll need to install the [`pulsar-client`](/api/python) library using the instructions [above](#getting-started).
+
+#### Python SDK examples
+
+There are several example Python functions in this {@inject: github:folder:/pulsar-functions/python-examples}:
+
+Function file | Description
+:-------------|:-----------
+[`exclamation_function.py`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/python-examples/exclamation_function.py) | Adds an exclamation point at the end of each incoming string
+[`logging_function.py`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/python-examples/logging_function.py) | Logs each incoming message
+[`thumbnailer.py`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/python-examples/thumbnailer.py) | Takes image data as input and outputs a 128x128 thumbnail of each image
+
+#### Python context object
+
+The [`Context`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/python/pulsar/functions/context.py) class provides a number of methods that you can use to access the function's [context](#context). The various methods for the `Context` class are listed below:
+
+Method | What it provides
+:------|:----------------
+`get_message_id` | The message ID of the message being processed
+`get_current_message_topic_name` | The topic of the message being currently being processed
+`get_function_tenant` | The tenant under which the current Pulsar Function runs under
+`get_function_namespace` | The namespace under which the current Pulsar Function runs under
+`get_function_name` | The name of the current Pulsar Function
+`get_function_id` | The ID of the current Pulsar Function
+`get_instance_id` | The ID of the current Pulsar Functions instance
+`get_function_version` | The version of the current Pulsar Function
+`get_logger` | A logger object that can be used for [logging](#python-logging)
+`get_user_config_value` | Returns the value of a [user-defined config](#python-user-config) (or `None` if the config doesn't exist)
+`get_user_config_map` | Returns the entire user-defined config as a dict
+`record_metric` | Records a per-key [metric](#python-metrics)
+`publish` | Publishes a message to the specified Pulsar topic
+`get_output_serde_class_name` | The name of the output [SerDe](#python-serde) class
+`ack` | [Acks](reference-terminology.md#acknowledgment-ack) the message being processed to Pulsar
+
+### Python SerDe
+
+Pulsar Functions use [SerDe](#serialization-and-deserialization-serde) when publishing data to and consuming data from Pulsar topics (this is true of both [native](#python-native-functions) functions and [SDK](#python-sdk-functions) functions). You can specify the SerDe when [creating](functions-deploying.md#cluster-mode) or [running](functions-deploying.md#local-run-mode) functions. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --tenant public \
+ --namespace default \
+ --name my_function \
+ --py my_function.py \
+ --classname my_function.MyFunction \
+ --custom-serde-inputs '{"input-topic-1":"Serde1","input-topic-2":"Serde2"}' \
+ --output-serde-classname Serde3 \
+ --output output-topic-1
+```
+
+In this case, there are two input topics, `input-topic-1` and `input-topic-2`, each of which is mapped to a different SerDe class (the map must be specified as a JSON string). The output topic, `output-topic-1`, uses the `Serde3` class for SerDe. At the moment, all Pulsar Function logic, include processing function and SerDe classes, must be contained within a single Python file.
+
+When using Pulsar Functions for Python, you essentially have three SerDe options:
+
+1. You can use the [`IdentitySerde`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/python/pulsar/functions/serde.py#L70), which leaves the data unchanged. The `IdentitySerDe` is the **default**. Creating or running a function without explicitly specifying SerDe will mean that this option is used.
+2. You can use the [`PickeSerDe`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/python/pulsar/functions/serde.py#L62), which uses Python's [`pickle`](https://docs.python.org/3/library/pickle.html) for SerDe.
+3. You can create a custom SerDe class by implementing the baseline [`SerDe`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/python/pulsar/functions/serde.py#L50) class, which has just two methods: [`serialize`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/python/pulsar/functions/serde.py#L53) for converting the object into bytes, and [`deserialize`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/python/pulsa [...]
+
+The table below shows when you should use each SerDe:
+
+SerDe option | When to use
+:------------|:-----------
+`IdentitySerde` | When you're working with simple types like strings, Booleans, integers, and the like
+`PickleSerDe` | When you're working with complex, application-specific types and are comfortable with `pickle`'s "best effort" approach
+Custom SerDe | When you require explicit control over SerDe, potentially for performance or data compatibility purposes
+
+#### Python SerDe example
+
+Imagine that you're writing Pulsar Functions in Python that are processing tweet objects. Here's a simple `Tweet` class:
+
+```python
+class Tweet(object):
+ def __init__(self, username, tweet_content):
+ self.username = username
+ self.tweet_content = tweet_content
+```
+
+In order to use this class in Pulsar Functions, you'd have two options:
+
+1. You could specify `PickleSerDe`, which would apply the [`pickle`](https://docs.python.org/3/library/pickle.html) library's SerDe
+1. You could create your own SerDe class. Here's a simple example:
+
+ ```python
+ from pulsar import SerDe
+
+ class TweetSerDe(SerDe):
+ def __init__(self, tweet):
+ self.tweet = tweet
+
+ def serialize(self, input):
+ return bytes("{0}|{1}".format(self.tweet.username, self.tweet.tweet_content))
+
+ def deserialize(self, input_bytes):
+ tweet_components = str(input_bytes).split('|')
+ return Tweet(tweet_components[0], tweet_componentsp[1])
+ ```
+
+### Python logging
+
+Pulsar Functions that use the [Python SDK](#python-sdk-functions) have access to a logging object that can be used to produce logs at the chosen log level. Here's a simple example function that logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`:
+
+```python
+from pulsar import Function
+
+class LoggingFunction(Function):
+ def process(self, input, context):
+ logger = context.get_logger()
+ msg_id = context.get_message_id()
+ if 'danger' in input:
+ logger.warn("A warning was received in message {0}".format(context.get_message_id()))
+ else:
+ logger.info("Message {0} received\nContent: {1}".format(msg_id, input))
+```
+
+If you want your function to produce logs on a Pulsar topic, you need to specify a **log topic** when creating or running the function. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --py logging_function.py \
+ --classname logging_function.LoggingFunction \
+ --log-topic logging-function-logs \
+ # Other function configs
+```
+
+Now, all logs produced by the `LoggingFunction` above can be accessed via the `logging-function-logs` topic.
+
+### Python user config
+
+The Python SDK's [`Context`](#context) object enables you to access key/value pairs provided to the Pulsar Function via the command line (as JSON). Here's an example function creation command that passes a key/value pair:
+
+```bash
+$ bin/pulsar-admin functions create \
+ # Other function configs \
+ --user-config '{"word-of-the-day":"verdure"}'
+```
+
+To access that value in a Python function:
+
+```python
+from pulsar import Function
+
+class UserConfigFunction(Function):
+ def process(self, input, context):
+ logger = context.get_logger()
+ wotd = context.get_user_config_value('word-of-the-day')
+ if wotd is None:
+ logger.warn('No word of the day provided')
+ else:
+ logger.info("The word of the day is {0}".format(wotd))
+```
+
+### Python metrics
+
+You can record metrics using the [`Context`](#context) object on a per-key basis. You can, for example, set a metric for the key `process-count` and a different metric for the key `elevens-count` every time the function processes a message. Here's an example:
+
+```python
+from pulsar import Function
+
+class MetricRecorderFunction(Function):
+ def process(self, input, context):
+ context.record_metric('hit-count', 1)
+
+ if input == 11:
+ context.record_metric('elevens-count', 1)
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-deploying.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-deploying.md
new file mode 100644
index 0000000..a467727
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-deploying.md
@@ -0,0 +1,227 @@
+---
+id: version-2.1.1-incubating-functions-deploying
+title: Deploying and managing Pulsar Functions
+sidebar_label: Deploying functions
+original_id: functions-deploying
+---
+
+At the moment, there are two deployment modes available for Pulsar Functions:
+
+Mode | Description
+:----|:-----------
+Local run mode | The function runs in your local environment, for example on your laptop
+Cluster mode | The function runs *inside of* your Pulsar cluster, on the same machines as your Pulsar brokers
+
+> #### Contributing new deployment modes
+> The Pulsar Functions feature was designed, however, with extensibility in mind. Other deployment options will be available in the future. If you'd like to add a new deployment option, we recommend getting in touch with the Pulsar developer community at [dev@pulsar.incubator.apache.org](mailto:dev@pulsar.incubator.apache.org).
+
+## Requirements
+
+In order to deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this:
+
+* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine
+* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](deploy-dcos.md), and more
+
+If you're running a non-[standalone](reference-terminology.md#standalone) cluster, you'll need to obtain the service URL for the cluster. How you obtain the service URL will depend on how you deployed your Pulsar cluster.
+
+## Command-line interface
+
+Pulsar Functions are deployed and managed using the [`pulsar-admin functions`](reference-pulsar-admin.md#functions) interface, which contains commands such as [`create`](reference-pulsar-admin.md#functions-create) for deploying functions in [cluster mode](#cluster-mode), [`trigger`](reference-pulsar-admin.md#trigger) for [triggering](#triggering-pulsar-functions) functions, [`list`](reference-pulsar-admin.md#list-2) for listing deployed functions, and several others.
+
+### Fully Qualified Function Name (FQFN)
+
+Each Pulsar Function has a **Fully Qualified Function Name** (FQFN) that consists of three elements: the function's tenant, namespace, and function name. FQFN's look like this:
+
+```http
+tenant/namespace/name
+```
+
+FQFNs enable you to, for example, create multiple functions with the same name provided that they're in different namespaces.
+
+### Default arguments
+
+When managing Pulsar Functions, you'll need to specify a variety of information about those functions, including tenant, namespace, input and output topics, etc. There are some parameters, however, that have default values that will be supplied if omitted. The table below lists the defaults:
+
+Parameter | Default
+:---------|:-------
+Function name | Whichever value is specified for the class name (minus org, library, etc.). The flag `--classname org.example.MyFunction`, for example, would give the function a name of `MyFunction`.
+Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`.
+Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`.
+Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`.
+Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees.md), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied
+Processing guarantees | [`ATLEAST_ONCE`](functions-guarantees.md)
+Pulsar service URL | `pulsar://localhost:6650`
+
+#### Example use of defaults
+
+Take this `create` command:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --jar my-pulsar-functions.jar \
+ --classname org.example.MyFunction \
+ --inputs my-function-input-topic1,my-function-input-topic2
+```
+
+The created function would have default values supplied for the function name (`MyFunction`), tenant (`public`), namespace (`default`), subscription type (`SHARED`), processing guarantees (`ATLEAST_ONCE`), and Pulsar service URL (`pulsar://localhost:6650`).
+
+## Local run mode
+
+If you run a Pulsar Function in **local run** mode, it will run on the machine from which the command is run (this could be your laptop, an [AWS EC2](https://aws.amazon.com/ec2/) instance, etc.). Here's an example [`localrun`](reference-pulsar-admin.md#localrun) command:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+ --py myfunc.py \
+ --classname myfunc.SomeFunction \
+ --inputs persistent://public/default/input-1 \
+ --output persistent://public/default/output-1
+```
+
+By default, the function will connect to a Pulsar cluster running on the same machine, via a local [broker](reference-terminology.md#broker) service URL of `pulsar://localhost:6650`. If you'd like to use local run mode to run a function but connect it to a non-local Pulsar cluster, you can specify a different broker URL using the `--brokerServiceUrl` flag. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+ --broker-service-url pulsar://my-cluster-host:6650 \
+ # Other function parameters
+```
+
+## Cluster mode
+
+When you run a Pulsar Function in **cluster mode**, the function code will be uploaded to a Pulsar broker and run *alongside the broker* rather than in your [local environment](#local-run-mode). You can run a function in cluster mode using the [`create`](reference-pulsar-admin.md#create-1) command. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --py myfunc.py \
+ --classname myfunc.SomeFunction \
+ --inputs persistent://public/default/input-1 \
+ --output persistent://public/default/output-1
+```
+
+### Updating cluster mode functions
+
+You can use the [`update`](reference-pulsar-admin.md#update-1) command to update a Pulsar Function running in cluster mode. This command, for example, would update the function created in the section [above](#cluster-mode):
+
+```bash
+$ bin/pulsar-admin functions update \
+ --py myfunc.py \
+ --classname myfunc.SomeFunction \
+ --inputs persistent://public/default/new-input-topic \
+ --output persistent://public/default/new-output-topic
+```
+
+### Parallelism
+
+Pulsar Functions run as processes called **instances**. When you run a Pulsar Function, it runs as a single instance by default (and in [local run mode](#local-run-mode) you can *only* run a single instance of a function).
+
+You can also specify the *parallelism* of a function, i.e. the number of instances to run, when you create the function. You can set the parallelism factor using the `--parallelism` flag of the [`create`](reference-pulsar-admin.md#functions-create) command. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --parallelism 3 \
+ # Other function info
+```
+
+You can adjust the parallelism of an already created function using the [`update`](reference-pulsar-admin.md#update-1) interface.
+
+```bash
+$ bin/pulsar-admin functions update \
+ --parallelism 5 \
+ # Other function
+```
+
+If you're specifying a function's configuration via YAML, use the `parallelism` parameter. Here's an example config file:
+
+```yaml
+# function-config.yaml
+parallelism: 3
+inputs:
+- persistent://public/default/input-1
+output: persistent://public/default/output-1
+# other parameters
+```
+
+And here's the corresponding update command:
+
+```bash
+$ bin/pulsar-admin functions update \
+ --function-config-file function-config.yaml
+```
+
+### Function instance resources
+
+When you run Pulsar Functions in [cluster run](#cluster-mode) mode, you can specify the resources that are assigned to each function [instance](#parallelism):
+
+Resource | Specified as... | Runtimes
+:--------|:----------------|:--------
+CPU | The number of cores | Docker (coming soon)
+RAM | The number of bytes | Process, Docker
+Disk space | The number of bytes | Docker
+
+Here's an example function creation command that allocates 8 cores, 8 GB of RAM, and 10 GB of disk space to a function:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --jar target/my-functions.jar \
+ --classname org.example.functions.MyFunction \
+ --cpu 8 \
+ --ram 8589934592 \
+ --disk 10737418240
+```
+
+> #### Resources are *per instance*
+> The resources that you apply to a given Pulsar Function are applied to each [instance](#parallelism) of the function. If you apply 8 GB of RAM to a function with a paralellism of 5, for example, then you are applying 40 GB of RAM total for the function. You should always make sure to factor paralellism---i.e. the number of instances---into your resource calculations
+
+## Triggering Pulsar Functions
+
+If a Pulsar Function is running in [cluster mode](#cluster-mode), you can **trigger** it at any time using the command line. Triggering a function means that you send a message with a specific value to the function and get the function's output (if any) via the command line.
+
+> Triggering a function is ultimately no different from invoking a function by producing a message on one of the function's input topics. The [`pulsar-admin functions trigger`](reference-pulsar-admin.md#trigger) command is essentially a convenient mechanism for sending messages to functions without needing to use the [`pulsar-client`](reference-cli-tools.md#pulsar-client) tool or a language-specific client library.
+
+To show an example of function triggering, let's start with a simple [Python function](functions-api.md#functions-for-python) that returns a simple string based on the input:
+
+```python
+# myfunc.py
+def process(input):
+ return "This function has been triggered with a value of {0}".format(input)
+```
+
+Let's run that function in [local run mode](functions-deploying.md#local-run-mode):
+
+```bash
+$ bin/pulsar-admin functions create \
+ --tenant public \
+ --namespace default \
+ --name myfunc \
+ --py myfunc.py \
+ --classname myfunc \
+ --inputs persistent://public/default/in \
+ --output persistent://public/default/out
+```
+
+Now let's make a consumer listen on the output topic for messages coming from the `myfunc` function using the [`pulsar-client consume`](reference-cli-tools.md#consume) command:
+
+```bash
+$ bin/pulsar-client consume persistent://public/default/out \
+ --subscription-name my-subscription
+ --num-messages 0 # Listen indefinitely
+```
+
+Now let's trigger that function:
+
+```bash
+$ bin/pulsar-admin functions trigger \
+ --tenant public \
+ --namespace default \
+ --name myfunc \
+ --trigger-value "hello world"
+```
+
+The consumer listening on the output topic should then produce this in its logs:
+
+```
+----- got message -----
+This function has been triggered with a value of hello world
+```
+
+> #### Topic info not required
+> In the `trigger` command above, you may have noticed that you only need to specify basic information about the function (tenant, namespace, and name). To trigger the function, you didn't need to know the function's input topic(s).
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-guarantees.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-guarantees.md
new file mode 100644
index 0000000..d834f7a
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-guarantees.md
@@ -0,0 +1,42 @@
+---
+id: version-2.1.1-incubating-functions-guarantees
+title: Processing guarantees
+sidebar_label: Processing guarantees
+original_id: functions-guarantees
+---
+
+Pulsar Functions provides three different messaging semantics that you can apply to any function:
+
+Delivery semantics | Description
+:------------------|:-------
+**At-most-once** delivery | Each message that is sent to the function will most likely be processed but also may not be (hence the "at most")
+**At-least-once** delivery | Each message that is sent to the function could be processed more than once (hence the "at least")
+**Effectively-once** delivery | Each message that is sent to the function will have one output associated with it
+
+## Applying processing guarantees to a function
+
+You can set the processing guarantees for a Pulsar Function when you create the Function. This [`pulsar-function create`](reference-pulsar-admin.md#create-1) command, for example, would apply effectively-once guarantees to the Function:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --processing-guarantees EFFECTIVELY_ONCE \
+ # Other function configs
+```
+
+The available options are:
+
+* `ATMOST_ONCE`
+* `ATLEAST_ONCE`
+* `EFFECTIVELY_ONCE`
+
+> By default, Pulsar Functions provide at-most-once delivery guarantees. So if you create a function without supplying a value for the `--processingGuarantees` flag, then the function will provide at-most-once guarantees.
+
+## Updating the processing guarantees of a function
+
+You can change the processing guarantees applied to a function once it's already been created using the [`update`](reference-pulsar-admin.md#update-1) command. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions update \
+ --processing-guarantees ATMOST_ONCE \
+ # Other function configs
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-overview.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-overview.md
new file mode 100644
index 0000000..fe76f65
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-overview.md
@@ -0,0 +1,452 @@
+---
+id: version-2.1.1-incubating-functions-overview
+title: Pulsar Functions overview
+sidebar_label: Overview
+original_id: functions-overview
+---
+
+**Pulsar Functions** are lightweight compute processes that
+
+* consume messages from one or more Pulsar topics,
+* apply a user-supplied processing logic to each message,
+* publish the results of the computation to another topic
+
+Here's an example Pulsar Function for Java (using the [native interface](functions-api.md#java-native-functions)):
+
+```java
+import java.util.Function;
+
+public class ExclamationFunction implements Function<String, String> {
+ @Override
+ public String apply(String input) { return String.format("%s!", input); }
+}
+```
+
+Here's an equivalent function in Python (also using the [native interface](functions-api.md#python-native-functions)):
+
+```python
+def process(input):
+ return "{0}!".format(input)
+```
+
+Functions are executed each time a message is published to the input topic. If a function is listening on the topic `tweet-stream`, for example, then the function would be run each time a message is published to that topic.
+
+## Goals
+
+The core goal behind Pulsar Functions is to enable you to easily create processing logic of any level of complexity without needing to deploy a separate neighboring system (such as [Apache Storm](http://storm.apache.org/), [Apache Heron](https://apache.github.io/incubator-heron), [Apache Flink](https://flink.apache.org/), etc.). Pulsar Functions is essentially ready-made compute infrastructure at your disposal as part of your Pulsar messaging system. This core goal is tied to a series of [...]
+
+* Developer productivity ([language-native](#language-native-functions) vs. [Pulsar Functions SDK](#the-pulsar-functions-sdk) functions)
+* Easy troubleshooting
+* Operational simplicity (no need for an external processing system)
+
+## Inspirations
+
+The Pulsar Functions feature was inspired by (and takes cues from) several systems and paradigms:
+
+* Stream processing engines such as [Apache Storm](http://storm.apache.org/), [Apache Heron](https://apache.github.io/incubator-heron), and [Apache Flink](https://flink.apache.org)
+* "Serverless" and "Function as a Service" (FaaS) cloud platforms like [Amazon Web Services Lambda](https://aws.amazon.com/lambda/), [Google Cloud Functions](https://cloud.google.com/functions/), and [Azure Cloud Functions](https://azure.microsoft.com/en-us/services/functions/)
+
+Pulsar Functions could be described as
+
+* [Lambda](https://aws.amazon.com/lambda/)-style functions that are
+* specifically designed to use Pulsar as a message bus
+
+## Programming model
+
+The core programming model behind Pulsar Functions is very simple:
+
+* Functions receive messages from one or more **input [topics](reference-terminology.md#topic)**. Every time a message is received, the function can do a variety of things:
+ * Apply some processing logic to the input and write output to:
+ * An **output topic** in Pulsar
+ * [Apache BookKeeper](#state-storage)
+ * Write logs to a **log topic** (potentially for debugging purposes)
+ * Increment a [counter](#word-count-example)
+
+
+
+### Word count example
+
+If you were to implement the classic word count example using Pulsar Functions, it might look something like this:
+
+
+
+If you were writing the function in [Java](functions-api.md#functions-for-java) using the [Pulsar Functions SDK for Java](functions-api.md#java-sdk-functions), you could write the function like this...
+
+```java
+package org.example.functions;
+
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+
+import java.util.Arrays;
+
+public class WordCountFunction implements Function<String, Void> {
+ // This function is invoked every time a message is published to the input topic
+ @Override
+ public Void process(String input, Context context) {
+ Arrays.asList(input.split(" ")).forEach(word -> {
+ String counterKey = word.toLowerCase();
+ context.incrCounter(counterKey, 1)
+ });
+ return null;
+ }
+}
+```
+
+...and then [deploy it](#cluster-run-mode) in your Pulsar cluster using the [command line](#command-line-interface) like this:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --jar target/my-jar-with-dependencies.jar \
+ --classname org.example.functions.WordCountFunction \
+ --tenant public \
+ --namespace default \
+ --name word-count \
+ --inputs persistent://public/default/sentences \
+ --output persistent://public/default/count
+```
+
+### Content-based routing example
+
+The use cases for Pulsar Functions are essentially endless, but let's dig into a more sophisticated example that involves content-based routing.
+
+Imagine a function that takes items (strings) as input and publishes them to either a fruits or vegetables topic, depending on the item. Or, if an item is neither a fruit nor a vegetable, a warning is logged to a [log topic](#logging). Here's a visual representation:
+
+
+
+If you were implementing this routing functionality in Python, it might look something like this:
+
+```python
+from pulsar import Function
+
+class RoutingFunction(Function):
+ def __init__(self):
+ self.fruits_topic = "persistent://public/default/fruits"
+ self.vegetables_topic = "persistent://public/default/vegetables"
+
+ def is_fruit(item):
+ return item in ["apple", "orange", "pear", "other fruits..."]
+
+ def is_vegetable(item):
+ return item in ["carrot", "lettuce", "radish", "other vegetables..."]
+
+ def process(self, item, context):
+ if self.is_fruit(item):
+ context.publish(self.fruits_topic, item)
+ elif self.is_vegetable(item):
+ context.publish(self.vegetables_topic, item)
+ else:
+ warning = "The item {0} is neither a fruit nor a vegetable".format(item)
+ context.get_logger().warn(warning)
+```
+
+## Command-line interface
+
+Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode):
+
+```bash
+$ bin/pulsar-functions localrun \
+ --inputs persistent://public/default/test_src \
+ --output persistent://public/default/test_result \
+ --jar examples/api-examples.jar \
+ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction
+```
+
+## Fully Qualified Function Name (FQFN)
+
+Each Pulsar Function has a **Fully Qualified Function Name** (FQFN) that consists of three elements: the function's tenant, namespace, and function name. FQFN's look like this:
+
+```http
+tenant/namespace/name
+```
+
+FQFNs enable you to, for example, create multiple functions with the same name provided that they're in different namespaces.
+
+## Configuration
+
+Pulsar Functions can be configured in two ways:
+
+* Via [command-line arguments](#command-line-interface) passed to the [`pulsar-admin functions`](reference-pulsar-admin.md#functions) interface
+* Via [YAML](http://yaml.org/) configuration files
+
+If you're supplying a YAML configuration, you must specify a path to the file on the command line. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --function-config-file ./my-function.yaml
+```
+
+And here's an example `my-function.yaml` file:
+
+```yaml
+name: my-function
+tenant: public
+namespace: default
+jar: ./target/my-functions.jar
+className: org.example.pulsar.functions.MyFunction
+inputs:
+- persistent://public/default/test_src
+output: persistent://public/default/test_result
+```
+
+You can also mix and match configuration methods by specifying some function attributes via the CLI and others via YAML configuration.
+
+## Supported languages
+
+Pulsar Functions can currently be written in [Java](functions-api.md#functions-for-java) and [Python](functions-api.md#functions-for-python). Support for additional languages is coming soon.
+
+## The Pulsar Functions API
+
+The Pulsar Functions API enables you to create processing logic that is:
+
+* Type safe. Pulsar Functions can process raw bytes or more complex, application-specific types.
+* Based on SerDe (**Ser**ialization/**De**serialization). A variety of types are supported "out of the box" but you can also create your own custom SerDe logic.
+
+### Function context
+
+Each Pulsar Function created using the [Pulsar Functions SDK](#the-pulsar-functions-sdk) has access to a context object that both provides:
+
+1. A wide variety of information about the function, including:
+ * The name of the function
+ * The tenant and namespace of the function
+ * [User-supplied configuration](#user-configuration) values
+2. Special functionality, including:
+ * The ability to produce [logs](#logging) to a specified logging topic
+ * The ability to produce [metrics](#metrics)
+
+### Language-native functions
+
+Both Java and Python support writing "native" functions, i.e. Pulsar Functions with no dependencies.
+
+The benefit of native functions is that they don't have any dependencies beyond what's already available in Java/Python "out of the box." The downside is that they don't provide access to the function's [context](#function-context), which is necessary for a variety of functionality, including [logging](#logging), [user configuration](#user-configuration), and more.
+
+## The Pulsar Functions SDK
+
+If you'd like a Pulsar Function to have access to a [context object](#function-context), you can use the **Pulsar Functions SDK**, available for both [Java](functions-api.md#functions-for-java) and [Pythnon](functions-api.md#functions-for-python).
+
+### Java
+
+Here's an example Java function that uses information about its context:
+
+```java
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+import org.slf4j.Logger;
+
+public class ContextAwareFunction implements Function<String, Void> {
+ @Override
+ public Void process(String input, Context, context) {
+ Logger LOG = context.getLogger();
+ String functionTenant = context.getTenant();
+ String functionNamespace = context.getNamespace();
+ String functionName = context.getName();
+ LOG.info("Function tenant/namespace/name: {}/{}/{}", functionTenant, functionNamespace, functionName);
+ return null;
+ }
+}
+```
+
+### Python
+
+Here's an example Python function that uses information about its context:
+
+```python
+from pulsar import Function
+
+class ContextAwareFunction(Function):
+ def process(self, input, context):
+ log = context.get_logger()
+ function_tenant = context.get_function_tenant()
+ function_namespace = context.get_function_namespace()
+ function_name = context.get_function_name()
+ log.info("Function tenant/namespace/name: {0}/{1}/{2}".format(function_tenant, function_namespace, function_name))
+```
+
+## Deployment
+
+The Pulsar Functions feature was built to support a variety of deployment options. At the moment, there are two ways to run Pulsar Functions:
+
+Deployment mode | Description
+:---------------|:-----------
+[Local run mode](#local-run-mode) | The function runs in your local environment, for example on your laptop
+[Cluster mode](#cluster-run-mode) | The function runs *inside of* your Pulsar cluster, on the same machines as your Pulsar [brokers](reference-terminology.md#broker)
+
+### Local run mode
+
+If you run a Pulsar Function in **local run** mode, it will run on the machine from which the command is run (this could be your laptop, an [AWS EC2](https://aws.amazon.com/ec2/) instance, etc.). Here's an example [`localrun`](reference-pulsar-admin.md#localrun) command:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+ --py myfunc.py \
+ --classname myfunc.SomeFunction \
+ --inputs persistent://public/default/input-1 \
+ --output persistent://public/default/output-1
+```
+
+By default, the function will connect to a Pulsar cluster running on the same machine, via a local broker service URL of `pulsar://localhost:6650`. If you'd like to use local run mode to run a function but connect it to a non-local Pulsar cluster, you can specify a different broker URL using the `--brokerServiceUrl` flag. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+ --broker-service-url pulsar://my-cluster-host:6650 \
+ # Other function parameters
+```
+
+### Cluster run mode
+
+When you run a Pulsar Function in **cluster mode**, the function code will be uploaded to a Pulsar broker and run *alongside the broker* rather than in your [local environment](#local-run-mode). You can run a function in cluster mode using the [`create`](reference-pulsar-admin.md#create-1) command. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --py myfunc.py \
+ --classname myfunc.SomeFunction \
+ --inputs persistent://public/default/input-1 \
+ --output persistent://public/default/output-1
+```
+
+This command will upload `myfunc.py` to Pulsar, which will use the code to start one [or more](#parallelism) instances of the function.
+
+### Parallelism
+
+By default, only one **instance** of a Pulsar Function runs when you create and run it in [cluster run mode](#cluster-run-mode). You can also, however, run multiple instances in parallel. You can specify the number of instances when you create the function, or update an existing single-instance function with a new parallelism factor.
+
+This command, for example, would create and run a function with a parallelism of 5 (i.e. 5 instances):
+
+```bash
+$ bin/pulsar-admin functions create \
+ --name parallel-fun \
+ --tenant public \
+ --namespace default \
+ --py func.py \
+ --classname func.ParallelFunction \
+ --parallelism 5
+```
+
+### Function instance resources
+
+When you run Pulsar Functions in [cluster run](#cluster-run-mode) mode, you can specify the resources that are assigned to each function [instance](#parallelism):
+
+Resource | Specified as... | Runtimes
+:--------|:----------------|:--------
+CPU | The number of cores | Docker (coming soon)
+RAM | The number of bytes | Process, Docker
+Disk space | The number of bytes | Docker
+
+Here's an example function creation command that allocates 8 cores, 8 GB of RAM, and 10 GB of disk space to a function:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --jar target/my-functions.jar \
+ --classname org.example.functions.MyFunction \
+ --cpu 8 \
+ --ram 8589934592 \
+ --disk 10737418240
+```
+
+For more information on resources, see the [Deploying and Managing Pulsar Functions](functions-deploying.md#resources) documentation.
+
+### Logging
+
+Pulsar Functions created using the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can send logs to a log topic that you specify as part of the function's configuration. The function created using the command below, for example, would produce all logs on the `persistent://public/default/my-func-1-log` topic:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --name my-func-1 \
+ --log-topic persistent://public/default/my-func-1-log \
+ # Other configs
+```
+
+Here's an example [Java function](functions-api.md#java-logging) that logs at different log levels based on the function's input:
+
+```java
+public class LoggerFunction implements Function<String, Void> {
+ @Override
+ public Void process(String input, Context context) {
+ Logger LOG = context.getLogger();
+ if (input.length() <= 100) {
+ LOG.info("This string has a length of {}", input);
+ } else {
+ LOG.warn("This string is getting too long! It has {} characters", input);
+ }
+ }
+}
+```
+
+### User configuration
+
+Pulsar Functions can be passed arbitrary key-values via the command line (both keys and values must be strings). This set of key-values is called the functions **user configuration**. User configurations must consist of JSON strings.
+
+Here's an example of passing a user configuration to a function:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --user-config '{"key-1":"value-1","key-2","value-2"}' \
+ # Other configs
+```
+
+Here's an example of a function that accesses that config map:
+
+```java
+public class ConfigMapFunction implements Function<String, Void> {
+ @Override
+ public Void process(String input, Context context) {
+ String val1 = context.getUserConfigValue("key1").get();
+ String val2 = context.getUserConfigValue("key2").get();
+ context.getLogger().info("The user-supplied values are {} and {}", val1, val2);
+ return null;
+ }
+}
+```
+
+### Triggering Pulsar Functions
+
+Pulsar Functions running in [cluster mode](#cluster-run-mode) can be [triggered](functions-deploying.md#triggering-pulsar-functions) via the [command line](#command-line-interface). With triggering you can easily pass a specific value to a function and get the function's return value *without* needing to worry about creating a client, sending a message to the right input topic, etc. Triggering can be very useful for---but is by no means limited to---testing and debugging purposes.
+
+> Triggering a function is ultimately no different from invoking a function by producing a message on one of the function's input topics. The [`pulsar-admin functions trigger`](reference-pulsar-admin.md#trigger) command is essentially a convenient mechanism for sending messages to functions without needing to use the [`pulsar-client`](reference-cli-tools.md#pulsar-client) tool or a language-specific client library.
+
+Let's take an example Pulsar Function written in Python (using the [native interface](functions-api.md#python-native-functions)) that simply reverses string inputs:
+
+```python
+def process(input):
+ return input[::-1]
+```
+
+If that function were running in a Pulsar cluster, it could be triggered like this:
+
+```bash
+$ bin/pulsar-admin functions trigger \
+ --tenant public \
+ --namespace default \
+ --name reverse-func \
+ --trigger-value "snoitcnuf raslup ot emoclew"
+```
+
+That should return `welcome to pulsar functions` as the console output.
+
+> Instead of passing in a string via the CLI, you can also trigger a Pulsar Function with the contents of a file using the `--triggerFile` flag.
+
+## Processing guarantees
+
+The Pulsar Functions feature provides three different messaging semantics that you can apply to any function:
+
+Delivery semantics | Description
+:------------------|:-------
+**At-most-once** delivery | Each message that is sent to the function will most likely be processed but also may not be (hence the "at most")
+**At-least-once** delivery | Each message that is sent to the function could be processed more than once (hence the "at least")
+**Effectively-once** delivery | Each message that is sent to the function will have one output associated with it
+
+This command, for example, would run a function in [cluster mode](#cluster-run-mode) with effectively-once guarantees applied:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --name my-effectively-once-function \
+ --processing-guarantees EFFECTIVELY_ONCE \
+ # Other function configs
+```
+
+## Metrics
+
+Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics.md).
+
+## State storage
+
+Pulsar Functions use [Apache BookKeeper](https://bookkeeper.apache.org) as a state storage interface. All Pulsar installations, including local standalone installations, include a deployment of BookKeeper bookies.
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-quickstart.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-quickstart.md
new file mode 100644
index 0000000..10d7e2a
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-quickstart.md
@@ -0,0 +1,266 @@
+---
+id: version-2.1.1-incubating-functions-quickstart
+title: Getting started with Pulsar Functions
+sidebar_label: Getting started
+original_id: functions-quickstart
+---
+
+This tutorial will walk you through running a [standalone](reference-terminology.md#standalone) Pulsar [cluster](reference-terminology.md#cluster) on your machine and then running your first Pulsar Functions using that cluster. The first function will run in local run mode (outside your Pulsar [cluster](reference-terminology.md#cluster)), while the second will run in cluster mode (inside your cluster).
+
+> In local run mode, your Pulsar Function will communicate with your Pulsar cluster but will run outside of the cluster.
+
+## Prerequisites
+
+In order to follow along with this tutorial, you'll need to have [Maven](https://maven.apache.org/download.cgi) installed on your machine.
+
+## Run a standalone Pulsar cluster
+
+In order to run our Pulsar Functions, we'll need to run a Pulsar cluster locally first. The easiest way to do that is to run Pulsar in [standalone](reference-terminology.md#standalone) mode. Follow these steps to start up a standalone cluster:
+
+```bash
+$ wget pulsar:binary_release_url
+$ tar xvfz apache-pulsar-{{pulsar:version}}-bin.tar.gz
+$ cd apache-pulsar-{{pulsar:version}}
+$ bin/pulsar standalone \
+ --advertised-address 127.0.0.1
+```
+
+When running Pulsar in standalone mode, the `public` tenant and `default` namespace will be created automatically for you. That tenant and namespace will be used throughout this tutorial.
+
+## Run a Pulsar Function in local run mode
+
+Let's start with a simple function that takes a string as input from a Pulsar topic, adds an exclamation point to the end of the string, and then publishes that new string to another Pulsar topic. Here's the code for the function:
+
+```java
+package org.apache.pulsar.functions.api.examples;
+
+import java.util.function.Function;
+
+public class ExclamationFunction implements Function<String, String> {
+ @Override
+ public String apply(String input) {
+ return String.format("%s!", input);
+ }
+}
+```
+
+A JAR file containing this and several other functions (written in Java) is included with the binary distribution you downloaded above (in the `examples` folder). To run the function in local mode, i.e. on our laptop but outside our Pulsar cluster:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+ --jar examples/api-examples.jar \
+ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --name exclamation
+```
+
+> #### Multiple input topics allowed
+>
+> In the example above, a single topic was specified using the `--inputs` flag. You can also specify multiple input topics as a comma-separated list using the same flag. Here's an example:
+>
+> ```bash
+> --inputs topic1,topic2
+> ```
+
+We can open up another shell and use the [`pulsar-client`](reference-cli-tools.md#pulsar-client) tool to listen for messages on the output topic:
+
+```bash
+$ bin/pulsar-client consume persistent://public/default/exclamation-output \
+ --subscription-name my-subscription \
+ --num-messages 0
+```
+
+> Setting the `--num-messages` flag to 0 means that the consumer will listen on the topic indefinitely (rather than only accepting a certain number of messages).
+
+With a listener up and running, we can open up another shell and produce a message on the input topic that we specified:
+
+```bash
+$ bin/pulsar-client produce persistent://public/default/exclamation-input \
+ --num-produce 1 \
+ --messages "Hello world"
+```
+
+In the output, you should see the following:
+
+```
+----- got message -----
+Hello world!
+```
+
+Success! As you can see, the message has been successfully processed by the exclamation function. To shut down the function, simply hit **Ctrl+C**.
+
+Here's what happened:
+
+* The `Hello world` message that we published to the input topic (`persistent://public/default/exclamation-input`) was passed to the exclamation function that we ran on our machine
+* The exclamation function processed the message (providing a result of `Hello world!`) and published the result to the output topic (`persistent://public/default/exclamation-output`).
+* If our exclamation function *hadn't* been running, Pulsar would have durably stored the message data published to the input topic in [Apache BookKeeper](https://bookkeeper.apache.org) until a consumer consumed and acknowledged the message
+
+## Run a Pulsar Function in cluster mode
+
+[Local run mode](#run-a-pulsar-function-in-local-run-mode) is useful for development and experimentation, but if you want to use Pulsar Functions in a real Pulsar deployment, you'll want to run them in **cluster mode**. In this mode, Pulsar Functions run *inside* your Pulsar cluster and are managed using the same [`pulsar-admin functions`](reference-pulsar-admin.md#functions) interface that we've been using thus far.
+
+This command, for example, would deploy the same exclamation function we ran locally above *in our Pulsar cluster* (rather than outside it):
+
+```bash
+$ bin/pulsar-admin functions create \
+ --jar examples/api-examples.jar \
+ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --name exclamation
+```
+
+You should see `Created successfully` in the output. Now, let's see a list of functions running in our cluster:
+
+```bash
+$ bin/pulsar-admin functions list \
+ --tenant public \
+ --namespace default
+```
+
+We should see just the `exclamation` function listed there. We can also check the status of our deployed function using the `getstatus` command:
+
+```bash
+$ bin/pulsar-admin functions getstatus \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+```
+
+You should see this JSON output:
+
+```json
+{
+ "functionStatusList": [
+ {
+ "running": true,
+ "instanceId": "0"
+ }
+ ]
+}
+```
+
+As we can see, (a) the instance is currently running and (b) there is one instance, with an ID of 0, running. We can get other information about the function (topics, tenant, namespace, etc.) using the `get` command instead of `getstatus`:
+
+```bash
+$ bin/pulsar-admin functions get \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+```
+
+You should see this JSON output:
+
+```json
+{
+ "tenant": "public",
+ "namespace": "default",
+ "name": "exclamation",
+ "className": "org.apache.pulsar.functions.api.examples.ExclamationFunction",
+ "output": "persistent://public/default/exclamation-output",
+ "autoAck": true,
+ "inputs": [
+ "persistent://public/default/exclamation-input"
+ ],
+ "parallelism": 1
+}
+```
+
+As we can see, the parallelism of the function is 1, meaning that only one instance of the function is running in our cluster. Let's update our function to a parallelism of 3 using the `update` command:
+
+```bash
+$ bin/pulsar-admin functions update \
+ --jar examples/api-examples.jar \
+ --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation \
+ --parallelism 3
+```
+
+You should see `Updated successfully` in the output. If you run the `get` command from above for the function, you can see that the parallelism has increased to 3, meaning that there are now three instances of the function running in our cluster:
+
+```json
+{
+ "tenant": "public",
+ "namespace": "default",
+ "name": "exclamation",
+ "className": "org.apache.pulsar.functions.api.examples.ExclamationFunction",
+ "output": "persistent://public/default/exclamation-output",
+ "autoAck": true,
+ "inputs": [
+ "persistent://public/default/exclamation-input"
+ ],
+ "parallelism": 3
+}
+```
+
+Finally, we can shut down our running function using the `delete` command:
+
+```bash
+$ bin/pulsar-admin functions delete \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+```
+
+If you see `Deleted successfully` in the output, then you've succesfully run, updated, and shut down a Pulsar Function running in cluster mode. Congrats! Now, let's go even further and run a brand new function in the next section.
+
+## Writing and running a new function
+
+> In order to write and run the [Python](functions-api.md#functions-for-python) function below, you'll need to install a few dependencies:
+> ```bash
+> $ pip install pulsar-client
+> ```
+
+In the above examples, we ran and managed a pre-written Pulsar Function and saw how it worked. To really get our hands dirty, let's write and our own function from scratch, using the Python API. This simple function will also take a string as input but it will reverse the string and publish the resulting, reversed string to the specified topic.
+
+First, create a new Python file:
+
+```bash
+$ touch reverse.py
+```
+
+In that file, add the following:
+
+```python
+def process(input):
+ return input[::-1]
+```
+
+Here, the `process` method defines the processing logic of the Pulsar Function. It simply uses some Python slice magic to reverse each incoming string. Now, we can deploy the function using `create`:
+
+```bash
+$ bin/pulsar-admin functions create \
+ --py reverse.py \
+ --class-name reverse \
+ --inputs persistent://public/default/backwards \
+ --output persistent://public/default/forwards \
+ --tenant public \
+ --namespace default \
+ --name reverse
+```
+
+If you see `Created successfully`, the function is ready to accept incoming messages. Because the function is running in cluster mode, we can **trigger** the function using the [`trigger`](reference-pulsar-admin.md#trigger) command. This command will send a message that we specify to the function and also give us the function's output. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions trigger \
+ --name reverse \
+ --tenant public \
+ --namespace default \
+ --trigger-value "sdrawrof won si tub sdrawkcab saw gnirts sihT"
+```
+
+You should get this output:
+
+```
+This string was backwards but is now forwards
+```
+
+Once again, success! We created a brand new Pulsar Function, deployed it in our Pulsar standalone cluster in [cluster mode](#run-a-pulsar-function-in-cluster-mode) and successfully triggered the function. If you're ready for more, check out one of these docs:
+
+* [The Pulsar Functions API](functions-api.md)
+* [Deploying Pulsar Functions](functions-deploying.md)
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-clients.md b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-clients.md
new file mode 100644
index 0000000..c5aa174
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-clients.md
@@ -0,0 +1,58 @@
+---
+id: version-2.1.1-incubating-client-libraries
+title: Pulsar client libraries
+sidebar_label: Client libraries
+original_id: client-libraries
+---
+
+Pulsar currently has client libraries available for following languages:
+
+* [Java](#java-client)
+* [Go](#go-client)
+* [Python](#python-client)
+* [C++](#c-client)
+
+## Java client
+
+For a tutorial on using the Pulsar Java client to produce and consume messages, see [The Pulsar Java client](client-libraries-java.md).
+
+There are also two independent sets of Javadoc API docs available:
+
+Library | Purpose
+:-------|:-------
+[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java.md) for producing and consuming messages on Pulsar topics
+[`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md)
+
+
+## Go client
+
+For a tutorial on using the Pulsar Go client, see [The Pulsar Go client](client-libraries-go.md).
+
+
+## Python client
+
+For a tutorial on using the Pulsar Python client, see [The Pulsar Python client](client-libraries-python.md).
+
+There are also [pdoc](https://github.com/BurntSushi/pdoc)-generated API docs for the Python client [here](/api/python).
+
+## C++ client
+
+For a tutorial on using the Pulsar C++ clent, see [The Pulsar C++ client](client-libraries-cpp.md).
+
+There are also [Doxygen](http://www.stack.nl/~dimitri/doxygen/)-generated API docs for the C++ client [here](/api/cpp).
+
+## Feature Matrix
+
+This matrix listing all the features among different languages in Pulsar master can be found [here](https://github.com/apache/incubator-pulsar/wiki/Client-Features-Matrix).
+
+## Thirdparty Clients
+
+Besides the official released clients, there are also multiple projects on developing a Pulsar client in different languages.
+
+> if you have developed a Pulsar client, but it doesn't show up here. Feel free to submit a pull request to add your client to the list below.
+
+| Language | Project | Maintainer | License | Description |
+|----------|---------|------------|---------|-------------|
+| Go | [pulsar-client-go](https://github.com/Comcast/pulsar-client-go) | [Comcast](https://github.com/Comcast) | [](https://opensource.org/licenses/Apache-2.0) | A native golang client |
+| Go | [go-pulsar](https://github.com/t2y/go-pulsar) | [t2y](https://github.com/t2y) | [](https://opensource.org/licenses/Apache-2.0) | |
+| Scala | [pulsar4s](https://github.com/sksamuel/pulsar4s) | [sksamuel](https://github.com/sksamuel) | [](https://opensource.org/licenses/Apache-2.0) | Idomatic, typesafe, and reactive Scala client for Apache Pulsar |
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-standalone.md b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-standalone.md
new file mode 100644
index 0000000..8bc1834
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-standalone.md
@@ -0,0 +1,198 @@
+---
+id: version-2.1.1-incubating-standalone
+title: Setting up a local standalone cluster
+sidebar_label: Run Pulsar locally
+original_id: standalone
+---
+
+For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process.
+
+> #### Pulsar in production?
+> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide.
+
+## Run Pulsar Standalone Manually
+
+### System requirements
+
+Pulsar is currently available for **MacOS** and **Linux**. In order to use Pulsar, you'll need to install [Java 8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html).
+
+
+### Installing Pulsar
+
+To get started running Pulsar, download a binary tarball release in one of the following ways:
+
+* by clicking the link below and downloading the release from an Apache mirror:
+
+ * <a href="pulsar:binary_release_url" download>Pulsar {{pulsar:version}} binary release</a>
+
+* from the Pulsar [downloads page](pulsar:download_page_url)
+* from the Pulsar [releases page](https://github.com/apache/incubator-pulsar/releases/latest)
+* using [wget](https://www.gnu.org/software/wget):
+
+ ```shell
+ $ wget pulsar:binary_release_url
+ ```
+
+Once the tarball is downloaded, untar it and `cd` into the resulting directory:
+
+```bash
+$ tar xvfz apache-pulsar-{{pulsar:version}}-bin.tar.gz
+$ cd apache-pulsar-{{pulsar:version}}
+```
+
+### What your package contains
+
+The Pulsar binary package initially contains the following directories:
+
+Directory | Contains
+:---------|:--------
+`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md)
+`conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more
+`examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md)
+`lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar
+`licenses` | License files, in `.txt` form, for various components of the Pulsar [codebase](developing-codebase.md)
+
+These directories will be created once you begin running Pulsar:
+
+Directory | Contains
+:---------|:--------
+`data` | The data storage directory used by ZooKeeper and BookKeeper
+`instances` | Artifacts created for [Pulsar Functions](functions-overview.md)
+`logs` | Logs created by the installation
+
+
+### Installing Builtin Connectors
+
+Since release `2.1.0-incubating`, Pulsar releases a separate binary distribution, containing all the `builtin` connectors.
+If you would like to enable those `builtin` connectors, you can download the connectors tarball release in one of the following ways:
+
+* by clicking the link below and downloading the release from an Apache mirror:
+
+ * <a href="pulsar:connector_release_url" download>Pulsar IO Connectors {{pulsar:version}} release</a>
+
+* from the Pulsar [downloads page](pulsar:download_page_url)
+* from the Pulsar [releases page](https://github.com/apache/incubator-pulsar/releases/latest)
+* using [wget](https://www.gnu.org/software/wget):
+
+ ```shell
+ $ wget pulsar:connector_release_url
+ ```
+
+Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as `connectors`
+in the pulsar directory:
+
+```bash
+$ tar xvfz /path/to/apache-pulsar-io-connectors-{{pulsar:version}}-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-{{pulsar:version}}` in the pulsar directory
+// then copy the connectors
+
+$ cd apache-pulsar-io-connectors-{{pulsar:version}}/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar:version}}.nar
+pulsar-io-cassandra-{{pulsar:version}}.nar
+pulsar-io-kafka-{{pulsar:version}}.nar
+pulsar-io-kinesis-{{pulsar:version}}.nar
+pulsar-io-rabbitmq-{{pulsar:version}}.nar
+pulsar-io-twitter-{{pulsar:version}}.nar
+...
+```
+
+> #### NOTES
+>
+> If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory
+> (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions).
+>
+> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DCOS](deploy-dcos.md)),
+> you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors).
+
+### Starting the cluster
+
+Once you have an up-to-date local copy of the release, you can start up a local cluster using the [`pulsar`](reference-cli-tools.md#pulsar) command, which is stored in the `bin` directory, and specifying that you want to start up Pulsar in standalone mode:
+
+```bash
+$ bin/pulsar standalone
+```
+
+If Pulsar has been successfully started, you should see `INFO`-level log messages like this:
+
+```bash
+2017-06-01 14:46:29,192 - INFO - [main:WebSocketService@95] - Global Zookeeper cache started
+2017-06-01 14:46:29,192 - INFO - [main:AuthenticationService@61] - Authentication is disabled
+2017-06-01 14:46:29,192 - INFO - [main:WebSocketService@108] - Pulsar WebSocket Service started
+```
+
+> #### Automatically created namespace
+> When you start a local standalone cluster, Pulsar will automatically create a `public/default` [namespace](concepts-messaging.md#namespaces) that you can use for development purposes. All Pulsar topics are managed within namespaces. For more info, see [Topics](concepts-messaging.md#topics).
+
+## Run Pulsar Standalone in Docker
+
+Alternatively, you can run pulsar standalone locally in docker.
+
+```bash
+docker run -it -p 80:80 -p 8080:8080 -p 6650:6650 apachepulsar/pulsar-standalone
+```
+
+The command forwards following port to localhost:
+
+- 80: the port for pulsar dashboard
+- 8080: the http service url for pulsar service
+- 6650: the binary protocol service url for pulsar service
+
+After the docker container is running, you can access the dashboard under http://localhost .
+
+## Testing your cluster setup
+
+Pulsar provides a CLI tool called [`pulsar-client`](reference-cli-tools.md#pulsar-client) that enables you to do things like send messages to a Pulsar topic in a running cluster. This command will send a simple message saying `hello-pulsar` to the `my-topic` topic:
+
+```bash
+$ bin/pulsar-client produce my-topic --messages "hello-pulsar"
+```
+
+If the message has been successfully published to the topic, you should see a confirmation like this in the `pulsar-client` logs:
+
+```
+13:09:39.356 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced
+```
+
+
+> #### No need to explicitly create new topics
+> You may have noticed that we did not explicitly create the `my-topic` topic to which we sent the `hello-pulsar` message. If you attempt to write a message to a topic that does not yet exist, Pulsar will automatically create that topic for you.
+
+## Using Pulsar clients locally
+
+Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster:
+
+* `http://localhost:8080`
+* `pulsar://localhost:6650`
+
+Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client:
+
+```java
+String localClusterUrl = "pulsar://localhost:6650";
+
+PulsarClient client = PulsarClient.builder().serviceURL(localClusterUrl).build();
+Producer<byte[]> producer = client.newProducer().topic("my-topic").create();
+```
+
+Here's an example [Python](client-libraries-python.md) producer:
+
+```python
+import pulsar
+
+client = pulsar.Client('pulsar://localhost:6650')
+producer = client.create_producer('my-topic')
+```
+
+Finally, here's an example [C++](client-libraries-cpp.md) producer:
+
+```cpp
+Client client("pulsar://localhost:6650");
+Producer producer;
+Result result = client.createProducer("my-topic", producer);
+if (result != ResultOk) {
+ LOG_ERROR("Error creating producer: " << result);
+ return -1;
+}
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/io-managing.md b/site2/website/versioned_docs/version-2.1.1-incubating/io-managing.md
new file mode 100644
index 0000000..fb40833
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/io-managing.md
@@ -0,0 +1,162 @@
+---
+id: version-2.1.1-incubating-io-managing
+title: Managing Connectors
+sidebar_label: Managing Connectors
+original_id: io-managing
+---
+
+This section describes how to manage Pulsar IO connectors in a Pulsar cluster. You will learn how to:
+
+- Deploy builtin connectors
+- Monitor and update running connectors with Pulsar Admin CLI
+- Deploy customized connectors
+- Upgrade a connector
+
+## Using Builtin Connectors
+
+Pulsar bundles several [builtin connectors](io-overview.md#working-with-connectors) that should be used for moving data in and out
+of commonly used systems such as databases, messaging systems. Getting set up to use these builtin connectors is simple. You can follow
+the [instructions](getting-started-standalone.md#installing-builtin-connectors) on installing builtin connectors. After setup, all
+the builtin connectors will be automatically discovered by Pulsar brokers (or function-workers), so no additional installation steps are
+required.
+
+## Configuring Connectors
+
+Configuring Pulsar IO connectors is straightforward. What you need to do is to provide a yaml configuration file when your [run connectors](#running-connectors).
+The yaml configuration file basically tells Pulsar where to locate the sources and sinks and how to connect those sources and sinks with Pulsar topics.
+
+Below is an example yaml configuration file for Cassandra Sink:
+
+```shell
+tenant: public
+namespace: default
+name: cassandra-test-sink
+...
+# cassandra specific config
+configs:
+ roots: "localhost:9042"
+ keyspace: "pulsar_test_keyspace"
+ columnFamily: "pulsar_test_table"
+ keyname: "key"
+ columnName: "col"
+```
+
+The example yaml basically tells Pulsar which Cassandra cluster to connect, what is the `keyspace` and `columnFamily` to be used in Cassandra for collecting data,
+and how to map a Pulsar message into Cassandra table key and columns.
+
+For details, consult the documentation for [individual connectors](io-overview.md#working-with-connectors).
+
+## Running Connectors
+
+Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool.
+
+### Running sources
+
+You can submit a source to be run in an existing Pulsar cluster using a command of this form:
+
+```bash
+$ ./bin/pulsar-admin source create --classname <classname> --archive <jar-location> --tenant <tenant> --namespace <namespace> --name <source-name> --destination-topic-name <output-topic>
+```
+
+Here’s an example command:
+
+```bash
+bin/pulsar-admin source create --classname org.apache.pulsar.io.twitter.TwitterFireHose --archive ~/application.jar --tenant test --namespace ns1 --name twitter-source --destination-topic-name twitter_data
+```
+
+Instead of submitting a source to run on an existing Pulsar cluster, you alternatively can run a source as a process on your local machine:
+
+```bash
+bin/pulsar-admin source localrun --classname org.apache.pulsar.io.twitter.TwitterFireHose --archive ~/application.jar --tenant test --namespace ns1 --name twitter-source --destination-topic-name twitter_data
+```
+
+If you are submitting a built-in source, you don't need to specify `--classname` and `--archive`.
+You can simply specify the source type `--source-type`. The command to submit a built-in source is
+in following form:
+
+```bash
+./bin/pulsar-admin source create \
+ --tenant <tenant> \
+ --namespace <namespace> \
+ --name <source-name> \
+ --destination-topic-name <input-topics> \
+ --source-type <source-type>
+```
+
+Here's an example to submit a Kafka source:
+
+```bash
+./bin/pulsar-admin source create \
+ --tenant test-tenant \
+ --namespace test-namespace \
+ --name test-kafka-source \
+ --destination-topic-name pulsar_sink_topic \
+ --source-type kafka
+```
+
+### Running Sinks
+
+You can submit a sink to be run in an existing Pulsar cluster using a command of this form:
+
+```bash
+./bin/pulsar-admin sink create --classname <classname> --archive <jar-location> --tenant test --namespace <namespace> --name <sink-name> --inputs <input-topics>
+```
+
+Here’s an example command:
+
+```bash
+./bin/pulsar-admin sink create --classname org.apache.pulsar.io.cassandra --archive ~/application.jar --tenant test --namespace ns1 --name cassandra-sink --inputs test_topic
+```
+
+Instead of submitting a sink to run on an existing Pulsar cluster, you alternatively can run a sink as a process on your local machine:
+
+```bash
+./bin/pulsar-admin sink localrun --classname org.apache.pulsar.io.cassandra --archive ~/application.jar --tenant test --namespace ns1 --name cassandra-sink --inputs test_topic
+```
+
+If you are submitting a built-in sink, you don't need to specify `--classname` and `--archive`.
+You can simply specify the sink type `--sink-type`. The command to submit a built-in sink is
+in following form:
+
+```bash
+./bin/pulsar-admin sink create \
+ --tenant <tenant> \
+ --namespace <namespace> \
+ --name <sink-name> \
+ --inputs <input-topics> \
+ --sink-type <sink-type>
+```
+
+Here's an example to submit a Cassandra sink:
+
+```bash
+./bin/pulsar-admin sink create \
+ --tenant test-tenant \
+ --namespace test-namespace \
+ --name test-cassandra-sink \
+ --inputs pulsar_input_topic \
+ --sink-type cassandra
+```
+
+## Monitoring Connectors
+
+Since Pulsar IO connectors are running as [Pulsar Functions](functions-overiew.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands
+available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool.
+
+### Retrieve Connector Metadata
+
+```
+bin/pulsar-admin functions get \
+ --tenant <tenant> \
+ --namespace <namespace> \
+ --name <connector-name>
+```
+
+### Retrieve Connector Running Status
+
+```
+bin/pulsar-admin functions getstatus \
+ --tenant <tenant> \
+ --namespace <namespace> \
+ --name <connector-name>
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/io-quickstart.md b/site2/website/versioned_docs/version-2.1.1-incubating/io-quickstart.md
new file mode 100644
index 0000000..3aa5c5b
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/io-quickstart.md
@@ -0,0 +1,400 @@
+---
+id: version-2.1.1-incubating-io-quickstart
+title: Tutorial: Connecting Pulsar with Apache Cassandra
+sidebar_label: Getting started
+original_id: io-quickstart
+---
+
+This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code.
+It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide
+to gain a deeper understanding. At the end of this tutorial, you will be able to:
+
+- Connect your Pulsar cluster with your Cassandra cluster
+
+> #### Tip
+>
+> 1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all
+> the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes.
+>
+> 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution.
+
+## Installing Pulsar
+
+To get started running Pulsar, download a binary tarball release in one of the following ways:
+
+* by clicking the link below and downloading the release from an Apache mirror:
+
+ * <a href="pulsar:binary_release_url" download>Pulsar {{pulsar:version}} binary release</a>
+
+* from the Pulsar [downloads page](pulsar:download_page_url)
+* from the Pulsar [releases page](https://github.com/apache/incubator-pulsar/releases/latest)
+* using [wget](https://www.gnu.org/software/wget):
+
+ ```shell
+ $ wget pulsar:binary_release_url
+ ```
+
+Once the tarball is downloaded, untar it and `cd` into the resulting directory:
+
+```bash
+$ tar xvfz apache-pulsar-{{pulsar:version}}-bin.tar.gz
+$ cd apache-pulsar-{{pulsar:version}}
+```
+
+## Installing Builtin Connectors
+
+Since release `2.1.0-incubating`, Pulsar releases a separate binary distribution, containing all the `builtin` connectors.
+If you would like to enable those `builtin` connectors, you can download the connectors tarball release in one of the following ways:
+
+* by clicking the link below and downloading the release from an Apache mirror:
+
+ * <a href="pulsar:connector_release_url" download>Pulsar IO Connectors {{pulsar:version}} release</a>
+
+* from the Pulsar [downloads page](pulsar:download_page_url)
+* from the Pulsar [releases page](https://github.com/apache/incubator-pulsar/releases/latest)
+* using [wget](https://www.gnu.org/software/wget):
+
+ ```shell
+ $ wget pulsar:connector_release_url
+ ```
+
+Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as `connectors`
+in the pulsar directory:
+
+```bash
+$ tar xvfz /path/to/apache-pulsar-io-connectors-{{pulsar:version}}-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-{{pulsar:version}}` in the pulsar directory
+// then copy the connectors
+
+$ cp -r apache-pulsar-io-connectors-{{pulsar:version}}/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar:version}}.nar
+pulsar-io-cassandra-{{pulsar:version}}.nar
+pulsar-io-kafka-{{pulsar:version}}.nar
+pulsar-io-kinesis-{{pulsar:version}}.nar
+pulsar-io-rabbitmq-{{pulsar:version}}.nar
+pulsar-io-twitter-{{pulsar:version}}.nar
+...
+```
+
+
+## Start Pulsar Service
+
+```bash
+bin/pulsar standalone
+```
+
+All the components of a Pulsar service will start in order. You can curl those pulsar service endpoints to make sure Pulsar service is up running correctly.
+
+1. Check pulsar binary protocol port.
+
+```bash
+telnet localhost 6650
+```
+
+2. Check pulsar function cluster
+
+```bash
+curl -s http://localhost:8080/admin/v2/functions/cluster
+```
+
+Example output:
+```shell
+[{"workerId":"c-standalone-fw-localhost-6750","workerHostname":"localhost","port":6750}]
+```
+
+3. Make sure public tenant and default namespace exist
+
+```bash
+curl -s http://localhost:8080/admin/v2/namespaces/public
+```
+
+Example outoupt:
+```shell
+["public/default","public/functions"]
+```
+
+4. All builtin connectors should be listed as available.
+
+```bash
+curl -s http://localhost:8080/admin/v2/functions/connectors
+```
+
+Example output:
+```json
+[{"name":"aerospike","description":"Aerospike database sink","sinkClass":"org.apache.pulsar.io.aerospike.AerospikeStringSink"},{"name":"cassandra","description":"Writes data into Cassandra","sinkClass":"org.apache.pulsar.io.cassandra.CassandraStringSink"},{"name":"kafka","description":"Kafka source and sink connector","sourceClass":"org.apache.pulsar.io.kafka.KafkaStringSource","sinkClass":"org.apache.pulsar.io.kafka.KafkaStringSink"},{"name":"kinesis","description":"Kinesis sink connect [...]
+```
+
+If an error occurred while starting Pulsar service, you may be able to seen exception at the terminal you are running `pulsar/standalone`,
+or you can navigate the `logs` directory under the Pulsar directory to view the logs.
+
+## Connect Pulsar to Apache Cassandra
+
+> Make sure you have docker available at your laptop. If you don't have docker installed, you can follow the [instructions](https://docs.docker.com/docker-for-mac/install/).
+
+We are using `cassandra` docker image to start a single-node cassandra cluster in Docker.
+
+### Setup the Cassandra Cluster
+
+#### Start a Cassandra Cluster
+
+```bash
+docker run -d --rm --name=cassandra -p 9042:9042 cassandra
+```
+
+Before moving to next steps, make sure the cassandra cluster is up running.
+
+1. Make sure the docker process is running.
+
+```bash
+docker ps
+```
+
+2. Check the cassandra logs to make sure cassandra process is running as expected.
+
+```bash
+docker logs cassandra
+```
+
+3. Check the cluster status
+
+```bash
+docker exec cassandra nodetool status
+```
+
+Example output:
+```
+Datacenter: datacenter1
+=======================
+Status=Up/Down
+|/ State=Normal/Leaving/Joining/Moving
+-- Address Load Tokens Owns (effective) Host ID Rack
+UN 172.17.0.2 103.67 KiB 256 100.0% af0e4b2f-84e0-4f0b-bb14-bd5f9070ff26 rack1
+```
+
+#### Create keyspace and table
+
+We are using `cqlsh` to connect to the cassandra cluster to create keyspace and table.
+
+```bash
+$ docker exec -ti cassandra cqlsh localhost
+Connected to Test Cluster at localhost:9042.
+[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
+Use HELP for help.
+cqlsh>
+```
+
+All the following commands are executed in `cqlsh`.
+
+##### Create keyspace `pulsar_test_keyspace`
+
+```bash
+cqlsh> CREATE KEYSPACE pulsar_test_keyspace WITH replication = {'class':'SimpleStrategy', 'replication_factor':1};
+```
+
+#### Create table `pulsar_test_table`
+
+```bash
+cqlsh> USE pulsar_test_keyspace;
+cqlsh:pulsar_test_keyspace> CREATE TABLE pulsar_test_table (key text PRIMARY KEY, col text);
+```
+
+### Configure a Cassandra Sink
+
+Now that we have a Cassandra cluster running locally. In this section, we will configure a Cassandra sink connector.
+The Cassandra sink connector will read messages from a Pulsar topic and write the messages into a Cassandra table.
+
+In order to run a Cassandra sink connector, you need to prepare a yaml config file including informations that Pulsar IO
+runtime needs to know. For example, how Pulsar IO can find the cassandra cluster, what is the keyspace and table that
+Pulsar IO will be using for writing Pulsar messages to.
+
+Create a file `examples/cassandra-sink.yml` and edit it to fill in following content:
+
+```
+configs:
+ roots: "localhost:9042"
+ keyspace: "pulsar_test_keyspace"
+ columnFamily: "pulsar_test_table"
+ keyname: "key"
+ columnName: "col"
+```
+
+To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md).
+
+### Submit a Cassandra Sink
+
+Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors.
+
+We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`.
+
+```shell
+bin/pulsar-admin sink create \
+ --tenant public \
+ --namespace default \
+ --name cassandra-test-sink \
+ --sink-type cassandra \
+ --sink-config-file examples/cassandra-sink.yml \
+ --inputs test_cassandra
+```
+
+Once the command is executed, Pulsar will create a sink connector named `cassandra-test-sink` and the sink connector will be running
+as a Pulsar Function and write the messages produced in topic `test_cassandra` to Cassandra table `pulsar_test_table`.
+
+### Inspect the Cassandra Sink
+
+Since an IO connector is running as [Pulsar Functions](functions-overview.md), you can use [functions CLI](reference-pulsar-admin.md#functions)
+for inspecting and managing the IO connectors.
+
+#### Retrieve Sink Info
+
+```bash
+bin/pulsar-admin functions get \
+ --tenant public \
+ --namespace default \
+ --name cassandra-test-sink
+```
+
+Example output:
+
+```shell
+{
+ "tenant": "public",
+ "namespace": "default",
+ "name": "cassandra-test-sink",
+ "className": "org.apache.pulsar.functions.api.utils.IdentityFunction",
+ "autoAck": true,
+ "parallelism": 1,
+ "source": {
+ "topicsToSerDeClassName": {
+ "test_cassandra": ""
+ }
+ },
+ "sink": {
+ "configs": "{\"roots\":\"cassandra\",\"keyspace\":\"pulsar_test_keyspace\",\"columnFamily\":\"pulsar_test_table\",\"keyname\":\"key\",\"columnName\":\"col\"}",
+ "builtin": "cassandra"
+ },
+ "resources": {}
+}
+```
+
+#### Check Sink Running Status
+
+```bash
+bin/pulsar-admin functions getstatus \
+ --tenant public \
+ --namespace default \
+ --name cassandra-test-sink
+```
+
+Example output:
+
+```shell
+{
+ "functionStatusList": [
+ {
+ "running": true,
+ "instanceId": "0",
+ "metrics": {
+ "metrics": {
+ "__total_processed__": {},
+ "__total_successfully_processed__": {},
+ "__total_system_exceptions__": {},
+ "__total_user_exceptions__": {},
+ "__total_serialization_exceptions__": {},
+ "__avg_latency_ms__": {}
+ }
+ },
+ "workerId": "c-standalone-fw-localhost-6750"
+ }
+ ]
+}
+```
+
+### Verify the Cassandra Sink
+
+Now lets produce some messages to the input topic of the Cassandra sink `test_cassandra`.
+
+```bash
+for i in {0..9}; do bin/pulsar-client produce -m "key-$i" -n 1 test_cassandra; done
+```
+
+Inspect the sink running status again. You should be able to see 10 messages are processed by the Cassandra sink.
+
+```bash
+bin/pulsar-admin functions getstatus \
+ --tenant public \
+ --namespace default \
+ --name cassandra-test-sink
+```
+
+Example output:
+
+```shell
+{
+ "functionStatusList": [
+ {
+ "running": true,
+ "numProcessed": "11",
+ "numSuccessfullyProcessed": "11",
+ "lastInvocationTime": "1532031040117",
+ "instanceId": "0",
+ "metrics": {
+ "metrics": {
+ "__total_processed__": {
+ "count": 5.0,
+ "sum": 5.0,
+ "max": 5.0
+ },
+ "__total_successfully_processed__": {
+ "count": 5.0,
+ "sum": 5.0,
+ "max": 5.0
+ },
+ "__total_system_exceptions__": {},
+ "__total_user_exceptions__": {},
+ "__total_serialization_exceptions__": {},
+ "__avg_latency_ms__": {}
+ }
+ },
+ "workerId": "c-standalone-fw-localhost-6750"
+ }
+ ]
+}
+```
+
+Finally, lets inspect the results in Cassandra using `cqlsh`
+
+```bash
+docker exec -ti cassandra cqlsh localhost
+```
+
+Select the rows from the Cassandra table `pulsar_test_table`:
+
+```bash
+cqlsh> use pulsar_test_keyspace;
+cqlsh:pulsar_test_keyspace> select * from pulsar_test_table;
+
+ key | col
+--------+--------
+ key-5 | key-5
+ key-0 | key-0
+ key-9 | key-9
+ key-2 | key-2
+ key-1 | key-1
+ key-3 | key-3
+ key-6 | key-6
+ key-7 | key-7
+ key-4 | key-4
+ key-8 | key-8
+```
+
+### Delete the Cassandra Sink
+
+```shell
+bin/pulsar-admin sink delete \
+ --tenant public \
+ --namespace default \
+ --name cassandra-test-sink
+```
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/reference-configuration.md b/site2/website/versioned_docs/version-2.1.1-incubating/reference-configuration.md
new file mode 100644
index 0000000..a2d88c5
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/reference-configuration.md
@@ -0,0 +1,468 @@
+---
+id: version-2.1.1-incubating-reference-configuration
+title: Pulsar configuration
+sidebar_label: Pulsar configuration
+original_id: reference-configuration
+---
+
+<style type="text/css">
+ table{
+ font-size: 80%;
+ }
+</style>
+
+
+Pulsar configuration can be managed either via a series of configuration files contained in the [`conf`](https://github.com/apache/incubator-pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md)
+
+* [BookKeeper](#bookkeeper)
+* [Broker](#broker)
+* [Client](#client)
+* [Service discovery](#service-discovery)
+* [Log4j](#log4j)
+* [Log4j shell](#log4j-shell)
+* [Standalone](#standalone)
+* [WebSocket](#websocket)
+* [ZooKeeper](#zookeeper)
+
+## BookKeeper
+
+BookKeeper is a replicated log storage system that Pulsar uses for durable storage of all messages.
+
+
+|Name|Description|Default|
+|---|---|---|
+|bookiePort|The port on which the bookie server listens.|3181|
+|allowLoopback|Whether the bookie is allowed to use a loopback interface as its primary interface (i.e. the interface used to establish its identity). By default, loopback interfaces are not allowed as the primary interface. Using a loopback interface as the primary interface usually indicates a configuration error. For example, it’s fairly common in some VPS setups to not configure a hostname or to have the hostname resolve to `127.0.0.1`. If this is the case, then all bookies in the cl [...]
+|listeningInterface|The network interface on which the bookie listens. If not set, the bookie will listen on all interfaces.|eth0|
+|journalDirectory|The directory where Bookkeeper outputs its write-ahead log (WAL)|data/bookkeeper/journal|
+|ledgerDirectories|The directory where Bookkeeper outputs ledger snapshots. This could define multiple directories to store snapshots separated by comma, for example `ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data`. Ideally, ledger dirs and the journal dir are each in a different device, which reduces the contention between random I/O and sequential write. It is possible to run with a single disk, but performance will be significantly lower.|data/bookkeeper/ledgers|
+|ledgerManagerType|The type of ledger manager used to manage how ledgers are stored, managed, and garbage collected. See [BookKeeper Internals](http://bookkeeper.apache.org/docs/latest/getting-started/concepts) for more info.|hierarchical|
+|zkLedgersRootPath|The root ZooKeeper path used to store ledger metadata. This parameter is used by the ZooKeeper-based ledger manager as a root znode to store all ledgers.|/ledgers|
+|ledgerStorageClass|Ledger storage implementation class|org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage|
+|entryLogFilePreallocationEnabled|Enable or disable entry logger preallocation|true|
+|logSizeLimit|Max file size of the entry logger, in bytes. A new entry log file will be created when the old one reaches the file size limitation.|2147483648|
+|minorCompactionThreshold|Threshold of minor compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a minor compaction. If set to less than zero, the minor compaction is disabled.|0.2|
+|minorCompactionInterval|Time interval to run minor compaction, in seconds. If set to less than zero, the minor compaction is disabled.|3600|
+|majorCompactionThreshold|The threshold of major compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a major compaction. Those entry log files whose remaining size percentage is still higher than the threshold will never be compacted. If set to less than zero, the minor compaction is disabled.|0.5|
+|majorCompactionInterval|The time interval to run major compaction, in seconds. If set to less than zero, the major compaction is disabled.|86400|
+|compactionMaxOutstandingRequests|Sets the maximum number of entries that can be compacted without flushing. When compacting, the entries are written to the entrylog and the new offsets are cached in memory. Once the entrylog is flushed the index is updated with the new offsets. This parameter controls the number of entries added to the entrylog before a flush is forced. A higher value for this parameter means more memory will be used for offsets. Each offset consists of 3 longs. This pa [...]
+|compactionRate|The rate at which compaction will read entries, in adds per second.|1000|
+|isThrottleByBytes|Throttle compaction by bytes or by entries.|false|
+|compactionRateByEntries|The rate at which compaction will read entries, in adds per second.|1000|
+|compactionRateByBytes|Set the rate at which compaction will readd entries. The unit is bytes added per second.|1000000|
+|journalMaxSizeMB|Max file size of journal file, in megabytes. A new journal file will be created when the old one reaches the file size limitation.|2048|
+|journalMaxBackups|The max number of old journal filse to keep. Keeping a number of old journal files would help data recovery in special cases.|5|
+|journalPreAllocSizeMB|How space to pre-allocate at a time in the journal.|16|
+|journalWriteBufferSizeKB|The of the write buffers used for the journal.|64|
+|journalRemoveFromPageCache|Whether pages should be removed from the page cache after force write.|true|
+|journalAdaptiveGroupWrites|Whether to group journal force writes, which optimizes group commit for higher throughput.|true|
+|journalMaxGroupWaitMSec|The maximum latency to impose on a journal write to achieve grouping.|1|
+|journalAlignmentSize|All the journal writes and commits should be aligned to given size|4096|
+|journalBufferedWritesThreshold|Maximum writes to buffer to achieve grouping|524288|
+|journalFlushWhenQueueEmpty|If we should flush the journal when journal queue is empty|false|
+|numJournalCallbackThreads|The number of threads that should handle journal callbacks|8|
+|rereplicationEntryBatchSize|The number of max entries to keep in fragment for re-replication|5000|
+|gcWaitTime|How long the interval to trigger next garbage collection, in milliseconds. Since garbage collection is running in background, too frequent gc will heart performance. It is better to give a higher number of gc interval if there is enough disk capacity.|900000|
+|gcOverreplicatedLedgerWaitTime|How long the interval to trigger next garbage collection of overreplicated ledgers, in milliseconds. This should not be run very frequently since we read the metadata for all the ledgers on the bookie from zk.|86400000|
+|flushInterval|How long the interval to flush ledger index pages to disk, in milliseconds. Flushing index files will introduce much random disk I/O. If separating journal dir and ledger dirs each on different devices, flushing would not affect performance. But if putting journal dir and ledger dirs on same device, performance degrade significantly on too frequent flushing. You can consider increment flush interval to get better performance, but you need to pay more time on bookie server [...]
+|bookieDeathWatchInterval|Interval to watch whether bookie is dead or not, in milliseconds|1000|
+|zkServers|A list of one of more servers on which zookeeper is running. The server list can be comma separated values, for example: zkServers=zk1:2181,zk2:2181,zk3:2181.|localhost:2181|
+|zkTimeout|ZooKeeper client session timeout in milliseconds Bookie server will exit if it received SESSION_EXPIRED because it was partitioned off from ZooKeeper for more than the session timeout JVM garbage collection, disk I/O will cause SESSION_EXPIRED. Increment this value could help avoiding this issue|30000|
+|serverTcpNoDelay|This settings is used to enabled/disabled Nagle’s algorithm, which is a means of improving the efficiency of TCP/IP networks by reducing the number of packets that need to be sent over the network. If you are sending many small messages, such that more than one can fit in a single IP packet, setting server.tcpnodelay to false to enable Nagle algorithm can provide better performance.|true|
+|openFileLimit|Max number of ledger index files could be opened in bookie server If number of ledger index files reaches this limitation, bookie server started to swap some ledgers from memory to disk. Too frequent swap will affect performance. You can tune this number to gain performance according your requirements.|0|
+|pageSize|Size of a index page in ledger cache, in bytes A larger index page can improve performance writing page to disk, which is efficent when you have small number of ledgers and these ledgers have similar number of entries. If you have large number of ledgers and each ledger has fewer entries, smaller index page would improve memory usage.|8192|
+|pageLimit|How many index pages provided in ledger cache If number of index pages reaches this limitation, bookie server starts to swap some ledgers from memory to disk. You can increment this value when you found swap became more frequent. But make sure pageLimit*pageSize should not more than JVM max memory limitation, otherwise you would got OutOfMemoryException. In general, incrementing pageLimit, using smaller index page would gain bettern performance in lager number of ledgers with [...]
+|readOnlyModeEnabled|If all ledger directories configured are full, then support only read requests for clients. If “readOnlyModeEnabled=true” then on all ledger disks full, bookie will be converted to read-only mode and serve only read requests. Otherwise the bookie will be shutdown. By default this will be disabled.|true|
+|diskUsageThreshold|For each ledger dir, maximum disk space which can be used. Default is 0.95f. i.e. 95% of disk can be used at most after which nothing will be written to that partition. If all ledger dir partions are full, then bookie will turn to readonly mode if ‘readOnlyModeEnabled=true’ is set, else it will shutdown. Valid values should be in between 0 and 1 (exclusive).|0.95|
+|diskCheckInterval|Disk check interval in milli seconds, interval to check the ledger dirs usage.|10000|
+|auditorPeriodicCheckInterval|Interval at which the auditor will do a check of all ledgers in the cluster. By default this runs once a week. The interval is set in seconds. To disable the periodic check completely, set this to 0. Note that periodic checking will put extra load on the cluster, so it should not be run more frequently than once a day.|604800|
+|auditorPeriodicBookieCheckInterval|The interval between auditor bookie checks. The auditor bookie check, checks ledger metadata to see which bookies should contain entries for each ledger. If a bookie which should contain entries is unavailable, thea the ledger containing that entry is marked for recovery. Setting this to 0 disabled the periodic check. Bookie checks will still run when a bookie fails. The interval is specified in seconds.|86400|
+|numAddWorkerThreads|number of threads that should handle write requests. if zero, the writes would be handled by netty threads directly.|0|
+|numReadWorkerThreads|number of threads that should handle read requests. if zero, the reads would be handled by netty threads directly.|8|
+|maxPendingReadRequestsPerThread|If read workers threads are enabled, limit the number of pending requests, to avoid the executor queue to grow indefinitely.|2500|
+|readBufferSizeBytes|The number of bytes we should use as capacity for BufferedReadChannel.|4096|
+|writeBufferSizeBytes|The number of bytes used as capacity for the write buffer|65536|
+|useHostNameAsBookieID|Whether the bookie should use its hostname to register with the coordination service (e.g.: zookeeper service). When false, bookie will use its ipaddress for the registration.|false|
+|statsProviderClass||org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider|
+|prometheusStatsHttpPort||8000|
+|dbStorage_writeCacheMaxSizeMb|Size of Write Cache. Memory is allocated from JVM direct memory. Write cache is used to buffer entries before flushing into the entry log For good performance, it should be big enough to hold a sub|512|
+|dbStorage_readAheadCacheMaxSizeMb|Size of Read cache. Memory is allocated from JVM direct memory. This read cache is pre-filled doing read-ahead whenever a cache miss happens|256|
+|dbStorage_readAheadCacheBatchSize|How many entries to pre-fill in cache after a read cache miss|1000|
+|dbStorage_rocksDB_blockCacheSize|Size of RocksDB block-cache. For best performance, this cache should be big enough to hold a significant portion of the index database which can reach ~2GB in some cases|268435456|
+|dbStorage_rocksDB_writeBufferSizeMB||64|
+|dbStorage_rocksDB_sstSizeInMB||64|
+|dbStorage_rocksDB_blockSize||65536|
+|dbStorage_rocksDB_bloomFilterBitsPerKey||10|
+|dbStorage_rocksDB_numLevels||-1|
+|dbStorage_rocksDB_numFilesInLevel0||4|
+|dbStorage_rocksDB_maxSizeInLevel1MB||256|
+
+
+
+## Broker
+
+Pulsar brokers are responsible for handling incoming messages from producers, dispatching messages to consumers, replicating data between clusters, and more.
+
+|Name|Description|Default|
+|---|---|---|
+|enablePersistentTopics| Whether persistent topics are enabled on the broker |true|
+|enableNonPersistentTopics| Whether non-persistent topics are enabled on the broker |true|
+|functionsWorkerEnabled| Whether the Pulsar Functions worker service is enabled in the broker |false|
+|zookeeperServers| Zookeeper quorum connection string ||
+|globalZookeeperServers| Global Zookeeper quorum connection string ||
+|brokerServicePort| Broker data port |6650|
+|brokerServicePortTls| Broker data port for TLS |6651|
+|webServicePort| Port to use to server HTTP request |8080|
+|webServicePortTls| Port to use to server HTTPS request |8443|
+|webSocketServiceEnabled| Enable the WebSocket API service in broker |false|
+|bindAddress| Hostname or IP address the service binds on, default is 0.0.0.0. |0.0.0.0|
+|advertisedAddress| Hostname or IP address the service advertises to the outside world. If not set, the value of `InetAddress.getLocalHost().getHostName()` is used. ||
+|clusterName| Name of the cluster to which this broker belongs to ||
+|brokerDeduplicationEnabled| Sets the default behavior for message deduplication in the broker. If enabled, the broker will reject messages that were already stored in the topic. This setting can be overridden on a per-namespace basis. |false|
+|brokerDeduplicationMaxNumberOfProducers| The maximum number of producers for which information will be stored for deduplication purposes. |10000|
+|brokerDeduplicationEntriesInterval| The number of entries after which a deduplication informational snapshot is taken. A larger interval will lead to fewer snapshots being taken, though this would also lengthen the topic recovery time (the time required for entries published after the snapshot to be replayed). |1000|
+|brokerDeduplicationProducerInactivityTimeoutMinutes| The time of inactivity (in minutes) after which the broker will discard deduplication information related to a disconnected producer. |360|
+|zooKeeperSessionTimeoutMillis| Zookeeper session timeout in milliseconds |30000|
+|brokerShutdownTimeoutMs| Time to wait for broker graceful shutdown. After this time elapses, the process will be killed |60000|
+|backlogQuotaCheckEnabled| Enable backlog quota check. Enforces action on topic when the quota is reached |true|
+|backlogQuotaCheckIntervalInSeconds| How often to check for topics that have reached the quota |60|
+|backlogQuotaDefaultLimitGB| Default per-topic backlog quota limit |10|
+|brokerDeleteInactiveTopicsEnabled| Enable the deletion of inactive topics |true|
+|brokerDeleteInactiveTopicsFrequencySeconds| How often to check for inactive topics |60|
+|messageExpiryCheckIntervalInMinutes| How frequently to proactively check and purge expired messages |5|
+|brokerServiceCompactionMonitorIntervalInSeconds| Interval between checks to see if topics with compaction policies need to be compacted |60|
+|activeConsumerFailoverDelayTimeMillis| How long to delay rewinding cursor and dispatching messages when active consumer is changed. |1000|
+|clientLibraryVersionCheckEnabled| Enable check for minimum allowed client library version |false|
+|clientLibraryVersionCheckAllowUnversioned| Allow client libraries with no version information |true|
+|statusFilePath| Path for the file used to determine the rotation status for the broker when responding to service discovery health checks ||
+|preferLaterVersions| If true, (and ModularLoadManagerImpl is being used), the load manager will attempt to use only brokers running the latest software version (to minimize impact to bundles) |false|
+|tlsEnabled| Enable TLS |false|
+|tlsCertificateFilePath| Path for the TLS certificate file ||
+|tlsKeyFilePath| Path for the TLS private key file ||
+|tlsTrustCertsFilePath| Path for the trusted TLS certificate file ||
+|tlsAllowInsecureConnection| Accept untrusted TLS certificate from client |false|
+|maxUnackedMessagesPerConsumer| Max number of unacknowledged messages allowed to receive messages by a consumer on a shared subscription. Broker will stop sending messages to consumer once, this limit reaches until consumer starts acknowledging messages back. Using a value of 0, is disabling unackeMessage limit check and consumer can receive messages without any restriction |50000|
+|maxUnackedMessagesPerSubscription| Max number of unacknowledged messages allowed per shared subscription. Broker will stop dispatching messages to all consumers of the subscription once this limit reaches until consumer starts acknowledging messages back and unack count reaches to limit/2. Using a value of 0, is disabling unackedMessage-limit check and dispatcher can dispatch messages without any restriction |200000|
+|maxConcurrentLookupRequest| Max number of concurrent lookup request broker allows to throttle heavy incoming lookup traffic |50000|
+|maxConcurrentTopicLoadRequest| Max number of concurrent topic loading request broker allows to control number of zk-operations |5000|
+|authenticationEnabled| Enable authentication |false|
+|authenticationProviders| Autentication provider name list, which is comma separated list of class names ||
+|authorizationEnabled| Enforce authorization |false|
+|superUserRoles| Role names that are treated as “super-user”, meaning they will be able to do all admin operations and publish/consume from all topics ||
+|brokerClientAuthenticationPlugin| Authentication settings of the broker itself. Used when the broker connects to other brokers, either in same or other clusters ||
+|brokerClientAuthenticationParameters|||
+|athenzDomainNames| Supported Athenz provider domain names(comma separated) for authentication ||
+|bookkeeperClientAuthenticationPlugin| Authentication plugin to use when connecting to bookies ||
+|bookkeeperClientAuthenticationParametersName| BookKeeper auth plugin implementatation specifics parameters name and values ||
+|bookkeeperClientAuthenticationParameters|||
+|bookkeeperClientTimeoutInSeconds| Timeout for BK add / read operations |30|
+|bookkeeperClientSpeculativeReadTimeoutInMillis| Speculative reads are initiated if a read request doesn’t complete within a certain time Using a value of 0, is disabling the speculative reads |0|
+|bookkeeperClientHealthCheckEnabled| Enable bookies health check. Bookies that have more than the configured number of failure within the interval will be quarantined for some time. During this period, new ledgers won’t be created on these bookies |true|
+|bookkeeperClientHealthCheckIntervalSeconds||60|
+|bookkeeperClientHealthCheckErrorThresholdPerInterval||5|
+|bookkeeperClientHealthCheckQuarantineTimeInSeconds ||1800|
+|bookkeeperClientRackawarePolicyEnabled| Enable rack-aware bookie selection policy. BK will chose bookies from different racks when forming a new bookie ensemble |true|
+|bookkeeperClientIsolationGroups| Enable bookie isolation by specifying a list of bookie groups to choose from. Any bookie outside the specified groups will not be used by the broker ||
+|managedLedgerDefaultEnsembleSize| Number of bookies to use when creating a ledger |2|
+|managedLedgerDefaultWriteQuorum| Number of copies to store for each message |2|
+|managedLedgerDefaultAckQuorum| Number of guaranteed copies (acks to wait before write is complete) |2|
+|managedLedgerCacheSizeMB| Amount of memory to use for caching data payload in managed ledger. This memory is allocated from JVM direct memory and it’s shared across all the topics running in the same broker |1024|
+|managedLedgerCacheEvictionWatermark| Threshold to which bring down the cache level when eviction is triggered |0.9|
+|managedLedgerDefaultMarkDeleteRateLimit| Rate limit the amount of writes per second generated by consumer acking the messages |1.0|
+|managedLedgerMaxEntriesPerLedger| Max number of entries to append to a ledger before triggering a rollover. A ledger rollover is triggered on these conditions: <ul><li>Either the max rollover time has been reached</li><li>or max entries have been written to the ledged and at least min-time has passed</li></ul>|50000|
+|managedLedgerMinLedgerRolloverTimeMinutes| Minimum time between ledger rollover for a topic |10|
+|managedLedgerMaxLedgerRolloverTimeMinutes| Maximum time before forcing a ledger rollover for a topic |240|
+|managedLedgerCursorMaxEntriesPerLedger| Max number of entries to append to a cursor ledger |50000|
+|managedLedgerCursorRolloverTimeInSeconds| Max time before triggering a rollover on a cursor ledger |14400|
+|managedLedgerMaxUnackedRangesToPersist| Max number of “acknowledgment holes” that are going to be persistently stored. When acknowledging out of order, a consumer will leave holes that are supposed to be quickly filled by acking all the messages. The information of which messages are acknowledged is persisted by compressing in “ranges” of messages that were acknowledged. After the max number of ranges is reached, the information will only be tracked in memory and messages will be redel [...]
+|autoSkipNonRecoverableData| Skip reading non-recoverable/unreadable data-ledger under managed-ledger’s list.It helps when data-ledgers gets corrupted at bookkeeper and managed-cursor is stuck at that ledger. |false|
+|loadBalancerEnabled| Enable load balancer |true|
+|loadBalancerPlacementStrategy| Strategy to assign a new bundle weightedRandomSelection ||
+|loadBalancerReportUpdateThresholdPercentage| Percentage of change to trigger load report update |10|
+|loadBalancerReportUpdateMaxIntervalMinutes| maximum interval to update load report |15|
+|loadBalancerHostUsageCheckIntervalMinutes| Frequency of report to collect |1|
+|loadBalancerSheddingIntervalMinutes| Load shedding interval. Broker periodically checks whether some traffic should be offload from some over-loaded broker to other under-loaded brokers |30|
+|loadBalancerSheddingGracePeriodMinutes| Prevent the same topics to be shed and moved to other broker more that once within this timeframe |30|
+|loadBalancerBrokerMaxTopics| Usage threshold to allocate max number of topics to broker |50000|
+|loadBalancerBrokerUnderloadedThresholdPercentage| Usage threshold to determine a broker as under-loaded |1|
+|loadBalancerBrokerOverloadedThresholdPercentage| Usage threshold to determine a broker as over-loaded |85|
+|loadBalancerResourceQuotaUpdateIntervalMinutes| Interval to update namespace bundle resource quotat |15|
+|loadBalancerBrokerComfortLoadLevelPercentage| Usage threshold to determine a broker is having just right level of load |65|
+|loadBalancerAutoBundleSplitEnabled| enable/disable namespace bundle auto split |false|
+|loadBalancerNamespaceBundleMaxTopics| maximum topics in a bundle, otherwise bundle split will be triggered |1000|
+|loadBalancerNamespaceBundleMaxSessions| maximum sessions (producers + consumers) in a bundle, otherwise bundle split will be triggered |1000|
+|loadBalancerNamespaceBundleMaxMsgRate| maximum msgRate (in + out) in a bundle, otherwise bundle split will be triggered |1000|
+|loadBalancerNamespaceBundleMaxBandwidthMbytes| maximum bandwidth (in + out) in a bundle, otherwise bundle split will be triggered |100|
+|loadBalancerNamespaceMaximumBundles| maximum number of bundles in a namespace |128|
+|replicationMetricsEnabled| Enable replication metrics |true|
+|replicationConnectionsPerBroker| Max number of connections to open for each broker in a remote cluster More connections host-to-host lead to better throughput over high-latency links. |16|
+|replicationProducerQueueSize| Replicator producer queue size |1000|
+|replicatorPrefix| Replicator prefix used for replicator producer name and cursor name pulsar.repl||
+|replicationTlsEnabled| Enable TLS when talking with other clusters to replicate messages |false|
+|defaultRetentionTimeInMinutes| Default message retention time ||
+|defaultRetentionSizeInMB| Default retention size |0|
+|keepAliveIntervalSeconds| How often to check whether the connections are still alive |30|
+|brokerServicePurgeInactiveFrequencyInSeconds| How often broker checks for inactive topics to be deleted (topics with no subscriptions and no one connected) |60|
+|loadManagerClassName| Name of load manager to use |org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl|
+|managedLedgerOffloadDriver| Driver to use to offload old data to long term storage (Possible values: S3) ||
+|managedLedgerOffloadMaxThreads| Maximum number of thread pool threads for ledger offloading |2|
+|s3ManagedLedgerOffloadRegion| For Amazon S3 ledger offload, AWS region ||
+|s3ManagedLedgerOffloadBucket| For Amazon S3 ledger offload, Bucket to place offloaded ledger into ||
+|s3ManagedLedgerOffloadServiceEndpoint| For Amazon S3 ledger offload, Alternative endpoint to connect to (useful for testing) ||
+|s3ManagedLedgerOffloadMaxBlockSizeInBytes| For Amazon S3 ledger offload, Max block size in bytes. (64MB by default, 5MB minimum) |67108864|
+|s3ManagedLedgerOffloadReadBufferSizeInBytes| For Amazon S3 ledger offload, Read buffer size in bytes (1MB by default) |1048576|
+
+
+
+
+## Client
+
+The [`pulsar-client`](reference-cli-tools.md#pulsar-client) CLI tool can be used to publish messages to Pulsar and consume messages from Pulsar topics. This tool can be used in lieu of a client library.
+
+|Name|Description|Default|
+|---|---|---|
+|webServiceUrl| The web URL for the cluster. |http://localhost:8080/|
+|brokerServiceUrl| The Pulsar protocol URL for the cluster. |pulsar://localhost:6650/|
+|authPlugin| The authentication plugin. ||
+|authParams| The authentication parameters for the cluster, as a comma-separated string. ||
+|useTls| Whether or not TLS authentication will be enforced in the cluster. |false|
+|tlsAllowInsecureConnection|||
+|tlsTrustCertsFilePath|||
+
+
+## Service discovery
+
+|Name|Description|Default|
+|---|---|---|
+|zookeeperServers| Zookeeper quorum connection string (comma-separated) ||
+|globalZookeeperServers| Global zookeeper quorum connection string (comma-separated) ||
+|zookeeperSessionTimeoutMs| ZooKeeper session timeout |30000|
+|servicePort| Port to use to server binary-proto request |6650|
+|servicePortTls| Port to use to server binary-proto-tls request |6651|
+|webServicePort| Port that discovery service listen on |8080|
+|webServicePortTls| Port to use to server HTTPS request |8443|
+|bindOnLocalhost| Control whether to bind directly on localhost rather than on normal hostname |false|
+|authenticationEnabled| Enable authentication |false|
+|authenticationProviders| Authentication provider name list, which is comma separated list of class names (comma-separated) ||
+|authorizationEnabled| Enforce authorization |false|
+|superUserRoles| Role names that are treated as “super-user”, meaning they will be able to do all admin operations and publish/consume from all topics (comma-separated) ||
+|tlsEnabled| Enable TLS |false|
+|tlsCertificateFilePath| Path for the TLS certificate file ||
+|tlsKeyFilePath| Path for the TLS private key file ||
+
+
+
+## Log4j
+
+
+|Name|Default|
+|---|---|
+|pulsar.root.logger| WARN,CONSOLE|
+|pulsar.log.dir| logs|
+|pulsar.log.file| pulsar.log|
+|log4j.rootLogger| ${pulsar.root.logger}|
+|log4j.appender.CONSOLE| org.apache.log4j.ConsoleAppender|
+|log4j.appender.CONSOLE.Threshold| DEBUG|
+|log4j.appender.CONSOLE.layout| org.apache.log4j.PatternLayout|
+|log4j.appender.CONSOLE.layout.ConversionPattern| %d{ISO8601} - %-5p - [%t:%C{1}@%L] - %m%n|
+|log4j.appender.ROLLINGFILE| org.apache.log4j.DailyRollingFileAppender|
+|log4j.appender.ROLLINGFILE.Threshold| DEBUG|
+|log4j.appender.ROLLINGFILE.File| ${pulsar.log.dir}/${pulsar.log.file}|
+|log4j.appender.ROLLINGFILE.layout| org.apache.log4j.PatternLayout|
+|log4j.appender.ROLLINGFILE.layout.ConversionPattern| %d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n|
+|log4j.appender.TRACEFILE| org.apache.log4j.FileAppender|
+|log4j.appender.TRACEFILE.Threshold| TRACE|
+|log4j.appender.TRACEFILE.File| pulsar-trace.log|
+|log4j.appender.TRACEFILE.layout| org.apache.log4j.PatternLayout|
+|log4j.appender.TRACEFILE.layout.ConversionPattern| %d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n|
+
+
+## Log4j shell
+
+|Name|Default|
+|---|---|
+|bookkeeper.root.logger| ERROR,CONSOLE|
+|log4j.rootLogger| ${bookkeeper.root.logger}|
+|log4j.appender.CONSOLE| org.apache.log4j.ConsoleAppender|
+|log4j.appender.CONSOLE.Threshold| DEBUG|
+|log4j.appender.CONSOLE.layout| org.apache.log4j.PatternLayout|
+|log4j.appender.CONSOLE.layout.ConversionPattern| %d{ABSOLUTE} %-5p %m%n|
+|log4j.logger.org.apache.zookeeper| ERROR|
+|log4j.logger.org.apache.bookkeeper| ERROR|
+|log4j.logger.org.apache.bookkeeper.bookie.BookieShell| INFO|
+
+
+## Standalone
+
+|Name|Description|Default|
+|---|---|---|
+|zookeeperServers| The quorum connection string for local ZooKeeper ||
+|globalZookeeperServers| The quorum connection string for global ZooKeeper ||
+|brokerServicePort| The port on which the standalone broker listens for connections |6650|
+|webServicePort| THe port used by the standalone broker for HTTP requests |8080|
+|bindAddress| The hostname or IP address on which the standalone service binds |0.0.0.0|
+|advertisedAddress| The hostname or IP address that the standalone service advertises to the outside world. If not set, the value of `InetAddress.getLocalHost().getHostName()` is used. ||
+|clusterName| The name of the cluster that this broker belongs to. |standalone|
+|zooKeeperSessionTimeoutMillis| The ZooKeeper session timeout, in milliseconds. |30000|
+|brokerShutdownTimeoutMs| The time to wait for graceful broker shutdown. After this time elapses, the process will be killed. |60000|
+|backlogQuotaCheckEnabled| Enable the backlog quota check, which enforces a specified action when the quota is reached. |true|
+|backlogQuotaCheckIntervalInSeconds| How often to check for topics that have reached the backlog quota. |60|
+|backlogQuotaDefaultLimitGB| The default per-topic backlog quota limit. |10|
+|brokerDeleteInactiveTopicsEnabled| Enable the deletion of inactive topics. |true|
+|brokerDeleteInactiveTopicsFrequencySeconds| How often to check for inactive topics, in seconds. |60|
+|messageExpiryCheckIntervalInMinutes| How often to proactively check and purged expired messages. |5|
+|activeConsumerFailoverDelayTimeMillis| How long to delay rewinding cursor and dispatching messages when active consumer is changed. |1000|
+|clientLibraryVersionCheckEnabled| Enable checks for minimum allowed client library version. |false|
+|clientLibraryVersionCheckAllowUnversioned| Allow client libraries with no version information |true|
+|statusFilePath| The path for the file used to determine the rotation status for the broker when responding to service discovery health checks |/usr/local/apache/htdocs|
+|maxUnackedMessagesPerConsumer| The maximum number of unacknowledged messages allowed to be received by consumers on a shared subscription. The broker will stop sending messages to a consumer once this limit is reached or until the consumer begins acknowledging messages. A value of 0 disables the unacked message limit check and thus allows consumers to receive messages without any restrictions. |50000|
+|maxUnackedMessagesPerSubscription| The same as above, except per subscription rather than per consumer. |200000|
+|authenticationEnabled| Enable authentication for the broker. |false|
+|authenticationProviders| A comma-separated list of class names for authentication providers. |false|
+|authorizationEnabled| Enforce authorization in brokers. |false|
+|superUserRoles| Role names that are treated as “superusers.” Superusers are authorized to perform all admin tasks. ||
+|brokerClientAuthenticationPlugin| The authentication settings of the broker itself. Used when the broker connects to other brokers either in the same cluster or from other clusters. ||
+|brokerClientAuthenticationParameters| The parameters that go along with the plugin specified using brokerClientAuthenticationPlugin. ||
+|athenzDomainNames| Supported Athenz authentication provider domain names as a comma-separated list. ||
+|bookkeeperClientAuthenticationPlugin| Authentication plugin to be used when connecting to bookies (BookKeeper servers). ||
+|bookkeeperClientAuthenticationParametersName| BookKeeper authentication plugin implementation parameters and values. ||
+|bookkeeperClientAuthenticationParameters| Parameters associated with the bookkeeperClientAuthenticationParametersName ||
+|bookkeeperClientTimeoutInSeconds| Timeout for BookKeeper add and read operations. |30|
+|bookkeeperClientSpeculativeReadTimeoutInMillis| Speculative reads are initiated if a read request doesn’t complete within a certain time. A value of 0 disables speculative reads. |0|
+|bookkeeperClientHealthCheckEnabled| Enable bookie health checks. |true|
+|bookkeeperClientHealthCheckIntervalSeconds| The time interval, in seconds, at which health checks are performed. New ledgers are not created during health checks. |60|
+|bookkeeperClientHealthCheckErrorThresholdPerInterval| Error threshold for health checks. |5|
+|bookkeeperClientHealthCheckQuarantineTimeInSeconds| If bookies have more than the allowed number of failures within the time interval specified by bookkeeperClientHealthCheckIntervalSeconds |1800|
+|bookkeeperClientRackawarePolicyEnabled| |true|
+|bookkeeperClientIsolationGroups|||
+|managedLedgerDefaultEnsembleSize| |1|
+|managedLedgerDefaultWriteQuorum| |1|
+|managedLedgerDefaultAckQuorum| |1|
+|managedLedgerCacheSizeMB| |1024|
+|managedLedgerCacheEvictionWatermark| |0.9|
+|managedLedgerDefaultMarkDeleteRateLimit| |0.1|
+|managedLedgerMaxEntriesPerLedger| |50000|
+|managedLedgerMinLedgerRolloverTimeMinutes| |10|
+|managedLedgerMaxLedgerRolloverTimeMinutes| |240|
+|managedLedgerCursorMaxEntriesPerLedger| |50000|
+|managedLedgerCursorRolloverTimeInSeconds| |14400|
+|autoSkipNonRecoverableData| |false|
+|loadBalancerEnabled| |false|
+|loadBalancerPlacementStrategy| |weightedRandomSelection|
+|loadBalancerReportUpdateThresholdPercentage| |10|
+|loadBalancerReportUpdateMaxIntervalMinutes| |15|
+|loadBalancerHostUsageCheckIntervalMinutes| |1|
+|loadBalancerSheddingIntervalMinutes| |30|
+|loadBalancerSheddingGracePeriodMinutes| |30|
+|loadBalancerBrokerMaxTopics| |50000|
+|loadBalancerBrokerUnderloadedThresholdPercentage| |1|
+|loadBalancerBrokerOverloadedThresholdPercentage| |85|
+|loadBalancerResourceQuotaUpdateIntervalMinutes| |15|
+|loadBalancerBrokerComfortLoadLevelPercentage| |65|
+|loadBalancerAutoBundleSplitEnabled| |false|
+|loadBalancerNamespaceBundleMaxTopics| |1000|
+|loadBalancerNamespaceBundleMaxSessions| |1000|
+|loadBalancerNamespaceBundleMaxMsgRate| |1000|
+|loadBalancerNamespaceBundleMaxBandwidthMbytes| |100|
+|loadBalancerNamespaceMaximumBundles| |128|
+|replicationMetricsEnabled| |true|
+|replicationConnectionsPerBroker| |16|
+|replicationProducerQueueSize| |1000|
+|defaultRetentionTimeInMinutes| |0|
+|defaultRetentionSizeInMB| |0|
+|keepAliveIntervalSeconds| |30|
+|brokerServicePurgeInactiveFrequencyInSeconds| |60|
+
+
+
+
+
+## WebSocket
+
+|Name|Description|Default|
+|---|---|---|
+|globalZookeeperServers |||
+|zooKeeperSessionTimeoutMillis| |30000|
+|serviceUrl|||
+|serviceUrlTls|||
+|brokerServiceUrl|||
+|brokerServiceUrlTls|||
+|webServicePort||8080|
+|webServicePortTls||8443|
+|bindAddress||0.0.0.0|
+|clusterName |||
+|authenticationEnabled||false|
+|authenticationProviders|||
+|authorizationEnabled||false|
+|superUserRoles |||
+|brokerClientAuthenticationPlugin|||
+|brokerClientAuthenticationParameters|||
+|tlsEnabled||false|
+|tlsAllowInsecureConnection||false|
+|tlsCertificateFilePath|||
+|tlsKeyFilePath |||
+|tlsTrustCertsFilePath|||
+
+
+## Pulsar proxy
+
+The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) can be configured in the `conf/proxy.conf` file.
+
+
+|Name|Description|Default|
+|---|---|---|
+|zookeeperServers| The ZooKeeper quorum connection string (as a comma-separated list) ||
+|configurationStoreServers| Configuration store connection string (as a comma-separated list) ||
+|zookeeperSessionTimeoutMs| ZooKeeper session timeout (in milliseconds) |30000|
+|servicePort| The port to use for server binary Protobuf requests |6650|
+|servicePortTls| The port to use to server binary Protobuf TLS requests |6651|
+|statusFilePath| Path for the file used to determine the rotation status for the proxy instance when responding to service discovery health checks ||
+|authenticationEnabled| Whether authentication is enabled for the Pulsar proxy |false|
+|authenticationProviders| Authentication provider name list (a comma-separated list of class names) ||
+|authorizationEnabled| Whether authorization is enforced by the Pulsar proxy |false|
+|authorizationProvider| Authorization provider as a fully qualified class name |org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider|
+|brokerClientAuthenticationPlugin| The authentication plugin used by the Pulsar proxy to authenticate with Pulsar brokers ||
+|brokerClientAuthenticationParameters| The authentication parameters used by the Pulsar proxy to authenticate with Pulsar brokers ||
+|brokerClientTrustCertsFilePath| The path to trusted certificates used by the Pulsar proxy to authenticate with Pulsar brokers ||
+|superUserRoles| Role names that are treated as “super-users,” meaning that they will be able to perform all admin ||
+|forwardAuthorizationCredentials| Whether client authorization credentials are forwared to the broker for re-authorization. Authentication must be enabled via authenticationEnabled=true for this to take effect. |false|
+|maxConcurrentInboundConnections| Max concurrent inbound connections. The proxy will reject requests beyond that. |10000|
+|maxConcurrentLookupRequests| Max concurrent outbound connections. The proxy will error out requests beyond that. |50000|
+|tlsEnabledInProxy| Whether TLS is enabled for the proxy |false|
+|tlsEnabledWithBroker| Whether TLS is enabled when communicating with Pulsar brokers |false|
+|tlsCertificateFilePath| Path for the TLS certificate file ||
+|tlsKeyFilePath| Path for the TLS private key file ||
+|tlsTrustCertsFilePath| Path for the trusted TLS certificate pem file ||
+|tlsHostnameVerificationEnabled| Whether the hostname is validated when the proxy creates a TLS connection with brokers |false|
+|tlsRequireTrustedClientCertOnConnect| Whether client certificates are required for TLS. Connections are rejected if the client certificate isn’t trusted. |false|
+
+
+## ZooKeeper
+
+ZooKeeper handles a broad range of essential configuration- and coordination-related tasks for Pulsar. The default configuration file for ZooKeeper is in the `conf/zookeeper.conf` file in your Pulsar installation. The following parameters are available:
+
+
+|Name|Description|Default|
+|---|---|---|
+|tickTime| The tick is the basic unit of time in ZooKeeper, measured in milliseconds and used to regulate things like heartbeats and timeouts. tickTime is the length of a single tick. |2000|
+|initLimit| The maximum time, in ticks, that the leader ZooKeeper server allows follower ZooKeeper servers to successfully connect and sync. The tick time is set in milliseconds using the tickTime parameter. |10|
+|syncLimit| The maximum time, in ticks, that a follower ZooKeeper server is allowed to sync with other ZooKeeper servers. The tick time is set in milliseconds using the tickTime parameter. |5|
+|dataDir| The location where ZooKeeper will store in-memory database snapshots as well as the transaction log of updates to the database. |data/zookeeper|
+|clientPort| The port on which the ZooKeeper server will listen for connections. |2181|
+|autopurge.snapRetainCount| In ZooKeeper, auto purge determines how many recent snapshots of the database stored in dataDir to retain within the time interval specified by autopurge.purgeInterval (while deleting the rest). |3|
+|autopurge.purgeInterval| The time interval, in hours, by which the ZooKeeper database purge task is triggered. Setting to a non-zero number will enable auto purge; setting to 0 will disable. Read this guide before enabling auto purge. |1|
+|maxClientCnxns| The maximum number of client connections. Increase this if you need to handle more ZooKeeper clients. |60|
+
+
+
+
+In addition to the parameters in the table above, configuring ZooKeeper for Pulsar involves adding
+a `server.N` line to the `conf/zookeeper.conf` file for each node in the ZooKeeper cluster, where `N` is the number of the ZooKeeper node. Here's an example for a three-node ZooKeeper cluster:
+
+```properties
+server.1=zk1.us-west.example.com:2888:3888
+server.2=zk2.us-west.example.com:2888:3888
+server.3=zk3.us-west.example.com:2888:3888
+```
+
+> We strongly recommend consulting the [ZooKeeper Administrator's Guide](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html) for a more thorough and comprehensive introduction to ZooKeeper configuration
diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/reference-pulsar-admin.md b/site2/website/versioned_docs/version-2.1.1-incubating/reference-pulsar-admin.md
new file mode 100644
index 0000000..f8cd205
--- /dev/null
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/reference-pulsar-admin.md
@@ -0,0 +1,1795 @@
+---
+id: version-2.1.1-incubating-pulsar-admin
+title: Pulsar admin CLI
+sidebar_label: Pulsar Admin CLI
+original_id: pulsar-admin
+---
+
+The `pulsar-admin` tool enables you to manage Pulsar installations, including clusters, brokers, namespaces, tenants, and more.
+
+Usage
+```bash
+$ pulsar-admin command
+```
+
+Commands
+* `broker-stats`
+* `brokers`
+* `clusters`
+* `functions`
+* `namespaces`
+* `ns-isolation-policy`
+* `sink`
+* `source`
+* `topics`
+* `tenants`
+* `resource-quotas`
+* `schemas`
+
+## `broker-stats`
+
+Operations to collect broker statistics
+
+```bash
+$ pulsar-admin broker-stats subcommand
+```
+
+Subcommands
+* `allocator-stats`
+* `destinations`
+* `mbeans`
+* `monitoring-metrics`
+* `topics`
+
+
+### `allocator-stats`
+
+Dump allocator stats
+
+Usage
+```bash
+$ pulsar-admin broker-stats allocator-stats allocator-name
+```
+
+### `desinations`
+
+Dump topic stats
+
+Usage
+```bash
+$ pulsar-admin broker-stats destinations options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-i`, `--indent`|Indent JSON output|false|
+
+### `mbeans`
+
+Dump Mbean stats
+
+Usage
+```bash
+$ pulsar-admin broker-stats mbeans options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-i`, `--indent`|Indent JSON output|false|
+
+
+### `monitoring-metrics`
+
+Dump metrics for monitoring
+
+Usage
+```bash
+$ pulsar-admin broker-stats monitoring-metrics options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-i`, `--indent`|Indent JSON output|false|
+
+
+### `topics`
+
+Dump topic stats
+
+Usage
+```bash
+$ pulsar-admin broker-stats topics options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-i`, `--indent`|Indent JSON output|false|
+
+
+## `brokers`
+
+Operations about brokers
+
+```bash
+$ pulsar-admin brokers subcommand
+```
+
+Subcommands
+* `list`
+* `namespaces`
+* `update-dynamic-config`
+* `list-dynamic-config`
+* `get-all-dynamic-config`
+* `get-internal-config`
+
+### `list`
+List active brokers of the cluster
+
+Usage
+```bash
+$ pulsar-admin brokers list cluster-name
+```
+
+### `namespaces`
+List namespaces owned by the broker
+
+Usage
+```bash
+$ pulsar-admin brokers namespaces cluster-name options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--url`|The URL for the broker||
+
+
+### `update-dynamic-config`
+Update a broker's dynamic service configuration
+
+Usage
+```bash
+$ pulsar-admin brokers update-dynamic-config options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--config`|Service configuration parameter name||
+|`--value`|Value for the configuration parameter value specified using the `--config` flag||
+
+
+### `list-dynamic-config`
+Get list of updatable configuration name
+
+Usage
+```bash
+$ pulsar-admin brokers list-dynamic-config
+```
+
+### `get-all-dynamic-config`
+Get all overridden dynamic-configuration values
+
+Usage
+```bash
+$ pulsar-admin brokers get-all-dynamic-config
+```
+
+### `get-internal-config`
+Get internal configuration information
+
+Usage
+```bash
+$ pulsar-admin brokers get-internal-config
+```
+
+
+## `clusters`
+Operations about clusters
+
+Usage
+```bash
+$ pulsar-admin clusters subcommand
+```
+
+Subcommands
+* `get`
+* `create`
+* `update`
+* `delete`
+* `list`
+* `update-peer-clusters`
+
+
+### `get`
+Get the configuration data for the specified cluster
+
+Usage
+```bash
+$ pulsar-admin clusters get cluster-name
+```
+
+### `create`
+Provisions a new cluster. This operation requires Pulsar super-user privileges.
+
+Usage
+```bash
+$ pulsar-admin clusters create cluster-name options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--broker-url`|The URL for the broker service.||
+|`--broker-url-secure`|The broker service URL for a secure connection||
+|`--url`|service-url||
+|`--url-secure`|service-url for secure connection||
+
+
+### `update`
+Update the configuration for a cluster
+
+Usage
+```bash
+$ pulsar-admin clusters update cluster-name options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--broker-url`|The URL for the broker service.||
+|`--broker-url-secure`|The broker service URL for a secure connection||
+|`--url`|service-url||
+|`--url-secure`|service-url for secure connection||
+
+
+### `delete`
+Deletes an existing cluster
+
+Usage
+```bash
+$ pulsar-admin clusters delete cluster-name
+```
+
+### `list`
+List the existing clusters
+
+Usage
+```bash
+$ pulsar-admin clusters list
+```
+
+### `update-peer-clusters`
+Update peer cluster names
+
+Usage
+```bash
+$ pulsar-admin clusters update-peer-clusters peer-cluster-names
+```
+
+## `functions`
+
+A command-line interface for Pulsar Functions
+
+Usage
+```bash
+$ pulsar-admin functions subcommand
+```
+
+Subcommands
+* `localrun`
+* `create`
+* `delete`
+* `update`
+* `get`
+* `getstatus`
+* `list`
+* `querystate`
+* `trigger`
+
+
+### `localrun`
+Run a Pulsar Function locally
+
+
+Usage
+```bash
+$ pulsar-admin functions localrun options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--cpu`|The CPU to allocate to each function instance (in number of cores)||
+|`--ram`|The RAM to allocate to each function instance (in bytes)||
+|`--disk`|The disk space to allocate to each function instance (in bytes)||
+|`--auto-ack`|Let the functions framework manage acking||
+|`--subs-name`|Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer||
+|`--broker-service-url `|The URL of the Pulsar broker||
+|`--classname`|The name of the function’s class||
+|`--custom-serde-inputs`|A map of the input topic to SerDe name||
+|`--custom-schema-inputs`|A map of the input topic to Schema class name||
+|`--client-auth-params`|Client Authentication Params||
+|`--function-config-file`|The path of the YAML config file used to configure the function||
+|`--hostname-verification-enabled`|Enable Hostname verification||
+|`--instance-id-offset`|Instance ids will be assigned starting from this offset||
+|`--inputs`|The input topics for the function (as a comma-separated list if more than one topic is desired)||
+|`--log-topic`|The topic to which logs from this function are published||
+|`--jar`|A path to the JAR file for the function (if the function is written in Java)||
+|`--name`|The name of the function||
+|`--namespace`|The function’s namespace||
+|`--output`|The name of the topic to which the function publishes its output (if any)||
+|`--output-serde-classname`|The SerDe class used for the function’s output||
+|`--parallelism`|The function’s parallelism factor, i.e. the number of instances of the function to run|1|
+|`--processing-guarantees`|The processing guarantees applied to the function. Can be one of: ATLEAST_ONCE, ATMOST_ONCE, or EFFECTIVELY_ONCE|ATLEAST_ONCE|
+|`--py`|The path of the Python file containing the function’s processing logic (if the function is written in Python)||
+|`--schema-type`|Schema Type to be used for storing output messages||
+|`--sliding-interval-count`|Number of messages after which the window ends||
+|`--sliding-interval-duration-ms`|The time duration after which the window slides||
+|`--state-storage-service-url`|The service URL for the function’s state storage (if the function uses a storage system different from the Apache BookKeeper cluster used by Pulsar)||
+|`--subscription-type`|The subscription type used by the function when consuming messages on the input topic(s). Can be either SHARED or EXCLUSIVE|SHARED|
+|`--tenant`|The function’s tenant||
+|`--topics-pattern`|The topic pattern to consume from list of topics under a namespace that match the pattern||
+|`--user-config`|A user-supplied config value, set as a key/value pair. You can set multiple user config values.||
+|`--window-length-count`|The number of messages per window.||
+|`--window-length-duration-ms`|The time duration of the window in milliseconds.||
+
+
+### `create`
+Creates a new Pulsar Function on the target infrastructure
+
+Usage
+```
+$ pulsar-admin functions create options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--cpu`|The CPU to allocate to each function instance (in number of cores)||
+|`--ram`|The RAM to allocate to each function instance (in bytes)||
+|`--disk`|The disk space to allocate to each function instance (in bytes)||
+|`--auto-ack`|Let the functions framework manage acking||
+|`--subs-name`|Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer||
+|`--classname`|The name of the function’s class||
+|`--custom-serde-inputs`|A map of the input topic to SerDe name||
+|`--custom-schema-inputs`|A map of the input topic to Schema class name||
+|`--function-config-file`|The path of the YAML config file used to configure the function||
+|`--inputs`|The input topics for the function (as a comma-separated list if more than one topic is desired)||
+|`--log-topic`|The topic to which logs from this function are published||
+|`--jar`|A path to the JAR file for the function (if the function is written in Java)||
+|`--name`|The name of the function||
+|`--namespace`|The function’s namespace||
+|`--output`|The name of the topic to which the function publishes its output (if any)||
+|`--output-serde-classname`|The SerDe class used for the function’s output||
+|`--parallelism`|The function’s parallelism factor, i.e. the number of instances of the function to run|1|
+|`--processing-guarantees`|The processing guarantees applied to the function. Can be one of: ATLEAST_ONCE, ATMOST_ONCE, or EFFECTIVELY_ONCE|ATLEAST_ONCE|
+|`--py`|The path of the Python file containing the function’s processing logic (if the function is written in Python)||
+|`--schema-type`|Schema Type to be used for storing output messages||
+|`--sliding-interval-count`|Number of messages after which the window ends||
+|`--sliding-interval-duration-ms`|The time duration after which the window slides||
+|`--subscription-type`|The subscription type used by the function when consuming messages on the input topic(s). Can be either SHARED or EXCLUSIVE|SHARED|
+|`--tenant`|The function’s tenant||
+|`--topics-pattern`|The topic pattern to consume from list of topics under a namespace that match the pattern||
+|`--user-config`|A user-supplied config value, set as a key/value pair. You can set multiple user config values.||
+|`--window-length-count`|The number of messages per window.||
+|`--window-length-duration-ms`|The time duration of the window in milliseconds.||
+
+
+### `delete`
+Deletes an existing Pulsar Function
+
+Usage
+```bash
+$ pulsar-admin functions delete options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the function to delete||
+|`--namespace`|The namespace of the function to delete||
+|`--tenant`|The tenant of the function to delete||
+
+
+### `update`
+Updates an existing Pulsar Function
+
+Usage
+```bash
+$ pulsar-admin functions update options
+```
+
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--cpu`|The CPU to allocate to each function instance (in number of cores)||
+|`--ram`|The RAM to allocate to each function instance (in bytes)||
+|`--disk`|The disk space to allocate to each function instance (in bytes)||
+|`--auto-ack`|Let the functions framework manage acking||
+|`--subs-name`|Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer||
+|`--classname`|The name of the function’s class||
+|`--custom-serde-inputs`|A map of the input topic to SerDe name||
+|`--custom-schema-inputs`|A map of the input topic to Schema class name||
+|`--function-config-file`|The path of the YAML config file used to configure the function||
+|`--inputs`|The input topics for the function (as a comma-separated list if more than one topic is desired)||
+|`--log-topic`|The topic to which logs from this function are published||
+|`--jar`|A path to the JAR file for the function (if the function is written in Java)||
+|`--name`|The name of the function||
+|`--namespace`|The function’s namespace||
+|`--output`|The name of the topic to which the function publishes its output (if any)||
+|`--output-serde-classname`|The SerDe class used for the function’s output||
+|`--parallelism`|The function’s parallelism factor, i.e. the number of instances of the function to run|1|
+|`--processing-guarantees`|The processing guarantees applied to the function. Can be one of: ATLEAST_ONCE, ATMOST_ONCE, or EFFECTIVELY_ONCE|ATLEAST_ONCE|
+|`--py`|The path of the Python file containing the function’s processing logic (if the function is written in Python)||
+|`--schema-type`|Schema Type to be used for storing output messages||
+|`--sliding-interval-count`|Number of messages after which the window ends||
+|`--sliding-interval-duration-ms`|The time duration after which the window slides||
+|`--subscription-type`|The subscription type used by the function when consuming messages on the input topic(s). Can be either SHARED or EXCLUSIVE|SHARED|
+|`--tenant`|The function’s tenant||
+|`--topics-pattern`|The topic pattern to consume from list of topics under a namespace that match the pattern||
+|`--user-config`|A user-supplied config value, set as a key/value pair. You can set multiple user config values.||
+|`--window-length-count`|The number of messages per window.||
+|`--window-length-duration-ms`|The time duration of the window in milliseconds.||
+
+
+### `get`
+Fetch information about an existing Pulsar Function
+
+Usage
+```bash
+$ pulsar-admin functions get options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the function||
+|`--namespace`|The namespace of the function||
+|`--tenant`|The tenant of the function||
+
+
+### `restart`
+Restarts either all instances or one particular instance of a function
+
+Usage
+```bash
+$ pulsar-admin functions restart options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the function||
+|`--namespace`|The namespace of the function||
+|`--tenant`|The tenant of the function||
+|`--instance-id`|The function instanceId; restart all instances if instance-id is not provided||
+
+
+### `stop`
+Temporary stops function instance. (If worker restarts then it reassigns and starts functiona again)
+
+Usage
+```bash
+$ pulsar-admin functions stop options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the function||
+|`--namespace`|The namespace of the function||
+|`--tenant`|The tenant of the function||
+|`--instance-id`|The function instanceId; stop all instances if instance-id is not provided||
+
+
+### `getstatus`
+Get the status of an existing Pulsar Function
+
+Usage
+```bash
+$ pulsar-admin functions getstatus options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the function||
+|`--namespace`|The namespace of the function||
+|`--tenant`|The tenant of the function||
+|`--instance-id`|The function instanceId; get status of all instances if instance-id is not provided||
+
+### `list`
+List all Pulsar Functions for a specific tenant and namespace
+
+Usage
+```bash
+$ pulsar-admin functions list options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--namespace`|The namespace of the function||
+|`--tenant`|The tenant of the function||
+
+
+### `querystate`
+Retrieve the current state of a Pulsar Function by key
+
+Usage
+```bash
+$ pulsar-admin functions querystate options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-k`, `--key`|The key for the state you want to fetch||
+|`--name`|The name of the function whose state you want to query||
+|`--namespace`|The namespace of the function whose state you want to query||
+|`--tenant`|The tenant of the function whose state you want to query||
+|`-u`, `--storage-service-url`|The service URL for the function’s state storage (if the function uses a storage system different from the Apache BookKeeper cluster used by Pulsar)||
+|`-w`, `--watch`|If set, watching for state changes is enabled|false|
+
+
+### `trigger`
+Triggers the specified Pulsar Function with a supplied value or file data
+
+Usage
+```bash
+$ pulsar-admin functions trigger options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the Pulsar Function to trigger||
+|`--namespace`|The namespace of the Pulsar Function to trigger||
+|`--tenant`|The tenant of the Pulsar Function to trigger||
+|`--trigger-file`|The path to the file containing the data with which the Pulsar Function is to be triggered||
+|`--trigger-value`|The value with which the Pulsar Function is to be triggered||
+
+
+## `namespaces`
+
+Operations for managing namespaces
+
+
+```bash
+$ pulsar-admin namespaces subcommand
+```
+
+Subcommands
+* `list`
+* `list-cluster`
+* `destinations`
+* `policies`
+* `create`
+* `delete`
+* `set-deduplication`
+* `permissions`
+* `grant-permission`
+* `revoke-permission`
+* `set-clusters`
+* `get-clusters`
+* `get-backlog-quotas`
+* `set-backlog-quota`
+* `remove-backlog-quota`
+* `get-persistence`
+* `set-persistence`
+* `get-message-ttl`
+* `set-message-ttl`
+* `get-retention`
+* `set-retention`
+* `unload`
+* `clear-backlog`
+* `unsubscribe`
+* `get-compaction-threshold`
+* `set-compaction-threshold`
+* `get-offload-threshold`
+* `set-offload-threshold`
+
+
+### `list`
+Get the namespaces for a tenant
+
+Usage
+```bash
+$ pulsar-admin namespaces list tenant-name
+```
+
+### `list-cluster`
+Get the namespaces for a tenant in the cluster
+
+Usage
+```bash
+$ pulsar-admin namespaces list-cluster tenant/cluster
+```
+
+### `destinations`
+Get the destinations for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces destinations tenant/cluster/namespace
+```
+
+### `policies`
+Get the policies of a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces policies tenant/cluster/namespace
+```
+
+### `create`
+Create a new namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces create tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-b` , `--bundles`|The number of bundles to activate|0|
+
+
+### `delete`
+Deletes a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces delete tenant/cluster/namespace
+```
+
+### `set-deduplication`
+Enable or disable message deduplication on a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces set-deduplication tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--enable`, `-e`|Enable message deduplication on the specified namespace|false|
+|`--disable`, `-d`|Disable message deduplication on the specified namespace|false|
+
+
+### `permissions`
+Get the permissions on a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces permissions tenant/cluster/namespace
+```
+
+### `grant-permission`
+Grant permissions on a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces grant-permission tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--actions`|Actions to be granted (`produce` or `consume`)||
+|`--role`|The client role to which to grant the permissions||
+
+
+### `revoke-permission`
+Revoke permissions on a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces revoke-permission tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--role`|The client role to which to grant the permissions||
+
+
+### `set-clusters`
+Set replication clusters for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces set-clusters tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-c`, `--clusters`|Replication clusters ID list (comma-separated values)||
+
+
+### `get-clusters`
+Get replication clusters for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces get-clusters tenant/cluster/namespace
+```
+
+### `get-backlog-quotas`
+Get the backlog quota policies for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces get-backlog-quotas tenant/cluster/namespace
+```
+
+### `set-backlog-quota`
+Set a backlog quota for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces set-backlog-quota tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-l`, `--limit`|The backlog size limit (for example `10M` or `16G`)||
+|`-p`, `--policy`|The retention policy to enforce when the limit is reached. The valid options are: `producer_request_hold`, `producer_exception` or `consumer_backlog_eviction`|
+
+Example
+```bash
+$ pulsar-admin namespaces set-backlog-quota my-prop/my-cluster/my-ns \
+--limit 2G \
+--policy producer_request_hold
+```
+
+### `remove-backlog-quota`
+Remove a backlog quota policy from a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces remove-backlog-quota tenant/cluster/namespace
+```
+
+### `get-persistence`
+Get the persistence policies for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces get-persistence tenant/cluster/namespace
+```
+
+### `set-persistence`
+Set the persistence policies for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces set-persistence tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-a`, `--bookkeeper-ack-quorom`|The number of acks (guaranteed copies) to wait for each entry|0|
+|`-e`, `--bookkeeper-ensemble`|The number of bookies to use for a topic|0|
+|`-w`, `--bookkeeper-write-quorum`|How many writes to make of each entry|0|
+|`-r`, `--ml-mark-delete-max-rate`|Throttling rate of mark-delete operation (0 means no throttle)||
+
+
+### `get-message-ttl`
+Get the message TTL for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces get-message-ttl tenant/cluster/namespace
+```
+
+### `set-message-ttl`
+Set the message TTL for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces set-message-ttl options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-ttl`, `--messageTTL`|Message TTL in seconds|0|
+
+
+### `get-retention`
+Get the retention policy for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces get-retention tenant/cluster/namespace
+```
+
+### `set-retention`
+Set the retention policy for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces set-retention tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-s`, `--size`|The retention size limits (for example 10M, 16G or 3T). 0 means no retention and -1 means infinite size retention||
+|`-t`, `--time`|The retention time in minutes, hours, days, or weeks. Examples: 100m, 13h, 2d, 5w. 0 means no retention and -1 means infinite time retention||
+
+
+### `unload`
+Unload a namespace or namespace bundle from the current serving broker.
+
+Usage
+```bash
+$ pulsar-admin namespaces unload tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-b`, `--bundle`|||
+
+
+### `clear-backlog`
+Clear the backlog for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces clear-backlog tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-b`, `--bundle`|||
+|`-f`, `--force`|Whether to force a clear backlog without prompt|false|
+|`-s`, `--sub`|The subscription name||
+
+
+### `unsubscribe`
+Unsubscribe the given subscription on all destinations on a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces unsubscribe tenant/cluster/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-b`, `--bundle`|||
+|`-s`, `--sub`|The subscription name||
+
+
+### `get-compaction-threshold`
+Get compactionThreshold for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces get-compaction-threshold tenant/namespace
+```
+
+### `set-compaction-threshold`
+Set compactionThreshold for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces set-compaction-threshold tenant/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-t`, `--threshold`|Maximum number of bytes in a topic backlog before compaction is triggered (eg: 10M, 16G, 3T). 0 disables automatic compaction|0|
+
+
+### `get-offload-threshold`
+Get offloadThreshold for a namespace
+
+Usage
+```bash
+$ pulsar-admin namespaces get-offload-threshold tenant/namespace
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-s`, `--size`|Maximum number of bytes stored in the pulsar cluster for a topic before data will start being automatically offloaded to longterm storage (eg: 10M, 16G, 3T, 100). Negative values disable automatic offload. 0 triggers offloading as soon as possible.|-1|
+
+
+
+## `ns-isolation-policy`
+Operations for managing namespace isolation policies.
+
+Usage
+```bash
+$ pulsar-admin ns-isolation-policy subcommand
+```
+
+Subcommands
+* `set`
+* `get`
+* `list`
+* `delete`
+
+### `set`
+Create/update a namespace isolation policy for a cluster. This operation requires Pulsar superuser privileges.
+
+Usage
+```bash
+$ pulsar-admin ns-isolation-policy set cluster-name policy-name options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--auto-failover-policy-params`|Comma-separated name=value auto failover policy parameters|[]|
+|`--auto-failover-policy-type`|Auto failover policy type name. Currently available options: min_available.|[]|
+|`--namespaces`|Comma-separated namespaces regex list|[]|
+|`--primary`|Comma-separated primary broker regex list|[]|
+|`--secondary`|Comma-separated secondary broker regex list|[]|
+
+
+### `get`
+Get the namespace isolation policy of a cluster. This operation requires Pulsar superuser privileges.
+
+Usage
+```bash
+$ pulsar-admin ns-isolation-policy get cluster-name policy-name
+```
+
+### `list`
+List all namespace isolation policies of a cluster. This operation requires Pulsar superuser privileges.
+
+Usage
+```bash
+$ pulsar-admin ns-isolation-policy list cluster-name
+```
+
+### `delete`
+Delete namespace isolation policy of a cluster. This operation requires superuser privileges.
+
+Usage
+```bash
+$ pulsar-admin ns-isolation-policy delete
+```
+
+
+## `sink`
+
+An interface for managing Pulsar IO sinks (egress data from Pulsar)
+
+Usage
+```bash
+$ pulsar-admin sink subcommand
+```
+
+Subcommands
+* `create`
+* `update`
+* `delete`
+* `localrun`
+* `available-sinks`
+
+
+### `create`
+Submit a Pulsar IO sink connector to run in a Pulsar cluster
+
+Usage
+```bash
+$ pulsar-admin sink create options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--classname`|The sink’s Java class name||
+|`--cpu`|The CPU (in cores) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--custom-serde-inputs`|The map of input topics to SerDe class names (as a JSON string)||
+|`--custom-schema-inputs`|The map of input topics to Schema types or class names (as a JSON string)||
+|`--disk`|The disk (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--inputs`|The sink’s input topic(s) (multiple topics can be specified as a comma-separated list)||
+|`--archive`|Path to the archive file for the sink||
+|`--name`|The sink’s name||
+|`--namespace`|The sink’s namespace||
+|`--parallelism`|“The sink’s parallelism factor (i.e. the number of sink instances to run).”||
+|`--processing-guarantees`|“The processing guarantees (aka delivery semantics) applied to the sink. Available values: ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE.”||
+|`--ram`|The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--sink-config`|Sink config key/values||
+|`--sink-config-file`|The path to a YAML config file specifying the sink’s configuration||
+|`--sink-type`|The built-in sinks's connector provider||
+|`--topics-pattern`|TopicsPattern to consume from list of topics under a namespace that match the pattern.||
+|`--tenant`|The sink’s tenant||
+
+
+### `update`
+Submit a Pulsar IO sink connector to run in a Pulsar cluster
+
+Usage
+```bash
+$ pulsar-admin sink update options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--classname`|The sink’s Java class name||
+|`--cpu`|The CPU (in cores) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--custom-serde-inputs`|The map of input topics to SerDe class names (as a JSON string)||
+|`--custom-schema-inputs`|The map of input topics to Schema types or class names (as a JSON string)||
+|`--disk`|The disk (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--inputs`|The sink’s input topic(s) (multiple topics can be specified as a comma-separated list)||
+|`--archive`|Path to the archive file for the sink||
+|`--name`|The sink’s name||
+|`--namespace`|The sink’s namespace||
+|`--parallelism`|“The sink’s parallelism factor (i.e. the number of sink instances to run).”||
+|`--processing-guarantees`|“The processing guarantees (aka delivery semantics) applied to the sink. Available values: ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE.”||
+|`--ram`|The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--sink-config`|Sink config key/values||
+|`--sink-config-file`|The path to a YAML config file specifying the sink’s configuration||
+|`--sink-type`|The built-in sinks's connector provider||
+|`--topics-pattern`|TopicsPattern to consume from list of topics under a namespace that match the pattern.||
+|`--tenant`|The sink’s tenant||
+
+
+### `delete`
+Stops a Pulsar IO sink
+
+Usage
+```bash
+$ pulsar-admin sink delete options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the function to delete||
+|`--namespace`|The namespace of the function to delete||
+|`--tenant`|The tenant of the function to delete||
+
+
+### `localrun`
+Run the Pulsar sink locally (rather than in the Pulsar cluster)
+
+Usage
+```bash
+$ pulsar-admin sink localrun options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--broker-service-url`|The URL for the Pulsar broker||
+|`--classname`|The sink’s Java class name||
+|`--cpu`|The CPU (in cores) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--custom-serde-inputs`|The map of input topics to SerDe class names (as a JSON string)||
+|`--custom-schema-inputs`|The map of input topics to Schema types or class names (as a JSON string)||
+|`--disk`|The disk (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--inputs`|The sink’s input topic(s) (multiple topics can be specified as a comma-separated list)||
+|`--archive`|Path to the archive file for the sink||
+|`--name`|The sink’s name||
+|`--namespace`|The sink’s namespace||
+|`--parallelism`|“The sink’s parallelism factor (i.e. the number of sink instances to run).”||
+|`--processing-guarantees`|“The processing guarantees (aka delivery semantics) applied to the sink. Available values: ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE.”||
+|`--ram`|The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime)||
+|`--sink-config`|Sink config key/values||
+|`--sink-config-file`|The path to a YAML config file specifying the sink’s configuration||
+|`--sink-type`|The built-in sinks's connector provider||
+|`--topics-pattern`|TopicsPattern to consume from list of topics under a namespace that match the pattern.||
+|`--tenant`|The sink’s tenant||
+
+
+### `available-sinks`
+Get a list of all built-in sink connectors
+
+Usage
+```bash
+$ pulsar-admin sink available-sinks
+```
+
+
+## `source`
+An interface for managing Pulsar IO sources (ingress data into Pulsar)
+
+Usage
+```bash
+$ pulsar-admin source subcommand
+```
+
+Subcommands
+* `create`
+* `update`
+* `delete`
+* `localrun`
+* `available-sources`
+
+
+### `create`
+Submit a Pulsar IO source connector to run in a Pulsar cluster
+
+Usage
+```bash
+$ pulsar-admin source create options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--classname`|The source’s Java class name||
+|`--cpu`|The CPU (in cores) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--deserialization-classname`|The SerDe classname for the source||
+|`--destination-topic-name`|The Pulsar topic to which data is sent||
+|`--disk`|The disk (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--archive`|The path to the NAR archive for the Source||
+|`--name`|The source’s name||
+|`--namespace`|The source’s namespace||
+|`--parallelism`|The source’s parallelism factor (i.e. the number of source instances to run).||
+|`--processing-guarantees`|“The processing guarantees (aka delivery semantics) applied to the source. Available values: ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE.”||
+|`--ram`|The RAM (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--schema-type`|The schema type (either a builtin schema like 'avro', 'json', etc, or custom Schema class name to be used to encode messages emitted from the source||
+|`--source-type`|One of the built-in source's connector provider||
+|`--source-config`|Source config key/values||
+|`--source-config-file`|The path to a YAML config file specifying the source’s configuration||
+|`--tenant`|The source’s tenant||
+
+
+### `update`
+Update a already submitted Pulsar IO source connector
+
+Usage
+```bash
+$ pulsar-admin source update options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--classname`|The source’s Java class name||
+|`--cpu`|The CPU (in cores) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--deserialization-classname`|The SerDe classname for the source||
+|`--destination-topic-name`|The Pulsar topic to which data is sent||
+|`--disk`|The disk (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--archive`|The path to the NAR archive for the Source||
+|`--name`|The source’s name||
+|`--namespace`|The source’s namespace||
+|`--parallelism`|The source’s parallelism factor (i.e. the number of source instances to run).||
+|`--processing-guarantees`|“The processing guarantees (aka delivery semantics) applied to the source. Available values: ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE.”||
+|`--ram`|The RAM (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--schema-type`|The schema type (either a builtin schema like 'avro', 'json', etc, or custom Schema class name to be used to encode messages emitted from the source||
+|`--source-type`|One of the built-in source's connector provider||
+|`--source-config`|Source config key/values||
+|`--source-config-file`|The path to a YAML config file specifying the source’s configuration||
+|`--tenant`|The source’s tenant||
+
+
+### `delete`
+Stops a Pulsar IO source
+
+Usage
+```bash
+$ pulsar-admin source delete options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--name`|The name of the function to delete||
+|`--namespace`|The namespace of the function to delete||
+|`--tenant`|The tenant of the function to delete||
+
+
+### `localrun`
+Run the Pulsar source locally (rather than in the Pulsar cluster)
+
+Usage
+```bash
+$ pulsar-admin source localrun options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--classname`|The source’s Java class name||
+|`--cpu`|The CPU (in cores) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--deserialization-classname`|The SerDe classname for the source||
+|`--destination-topic-name`|The Pulsar topic to which data is sent||
+|`--disk`|The disk (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--archive`|The path to the NAR archive for the Source||
+|`--name`|The source’s name||
+|`--namespace`|The source’s namespace||
+|`--parallelism`|The source’s parallelism factor (i.e. the number of source instances to run).||
+|`--processing-guarantees`|“The processing guarantees (aka delivery semantics) applied to the source. Available values: ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE.”||
+|`--ram`|The RAM (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime)||
+|`--schema-type`|The schema type (either a builtin schema like 'avro', 'json', etc, or custom Schema class name to be used to encode messages emitted from the source||
+|`--source-type`|One of the built-in source's connector provider||
+|`--source-config`|Source config key/values||
+|`--source-config-file`|The path to a YAML config file specifying the source’s configuration||
+|`--tenant`|The source’s tenant||
+
+
+### `available-sources`
+Get a list of all built-in source connectors
+
+Usage
+```bash
+$ pulsar-admin source available-sources
+```
+
+
+## `topics`
+Operations for managing Pulsar topics (both persistent and non persistent)
+
+Usage
+```bash
+$ pulsar-admin topics subcommand
+```
+
+Subcommands
+* `compact`
+* `compaction-status`
+* `offload`
+* `offload-status`
+* `create-partitioned-topic`
+* `delete-partitioned-topic`
+* `get-partitioned-topic-metadata`
+* `update-partitioned-topic`
+* `list`
+* `list-in-bundle`
+* `terminate`
+* `permissions`
+* `grant-permission`
+* `revoke-permission`
+* `lookup`
+* `bundle-range`
+* `delete`
+* `unload`
+* `subscriptions`
+* `unsubscribe`
+* `stats`
+* `stats-internal`
+* `info-internal`
+* `partitioned-stats`
+* `skip`
+* `skip-all`
+* `expire-messages`
+* `expire-messages-all-subscriptions`
+* `peek-messages`
+* `reset-cursor`
+
+
+### `compact`
+Run compaction on the specified topic (persistent topics only)
+
+Usage
+```
+$ pulsar-admin topics compact persistent://tenant/namespace/topic
+```
+
+### `compaction-status`
+Check the status of a topic compaction (persistent topics only)
+
+Usage
+```bash
+$ pulsar-admin topics compaction-status persistent://tenant/namespace/topic
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-w`, `--wait-complete`|Wait for compaction to complete|false|
+
+
+### `offload`
+Trigger offload of data from a topic to long-term storage (e.g. Amazon S3)
+
+Usage
+```bash
+$ pulsar-admin topics offload persistent://tenant/namespace/topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-s`, `--size-threshold`|The maximum amount of data to keep in BookKeeper for the specific topic||
+
+
+### `offload-status`
+Check the status of data offloading from a topic to long-term storage
+
+Usage
+```bash
+$ pulsar-admin topics offload-status persistent://tenant/namespace/topic op
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-w`, `--wait-complete`|Wait for compaction to complete|false|
+
+
+### `create-partitioned-topic`
+Create a partitioned topic. A partitioned topic must be created before producers can publish to it.
+
+Usage
+```bash
+$ pulsar-admin topics create-partitioned-topic {persistent|non-persistent}://tenant/namespace/topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-p`, `--partitions`|The number of partitions for the topic|0|
+
+
+### `delete-partitioned-topic`
+Delete a partitioned topic. This will also delete all the partitions of the topic if they exist.
+
+Usage
+```bash
+$ pulsar-admin topics delete-partitioned-topic {persistent|non-persistent}
+```
+
+### `get-partitioned-topic-metadata`
+Get the partitioned topic metadata. If the topic is not created or is a non-partitioned topic, this will return an empty topic with zero partitions.
+
+Usage
+```bash
+$ pulsar-admin topics get-partitioned-topic-metadata {persistent|non-persistent}://tenant/namespace/topic
+```
+
+### `update-partitioned-topic`
+Update existing non-global partitioned topic. New updating number of partitions must be greater than existing number of partitions.
+
+Usage
+```bash
+$ pulsar-admin topics update-partitioned-topic {persistent|non-persistent}://tenant/namespace/topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-p`, `--partitions`|The number of partitions for the topic|0|
+
+### `list`
+Get the list of topics under a namespace
+
+Usage
+```
+$ pulsar-admin topics list tenant/cluster/namespace
+```
+
+### `list-in-bundle`
+Get a list of non-persistent topics present under a namespace bundle
+
+Usage
+```
+$ pulsar-admin topics list-in-bundle tenant/namespace options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-b`, `--bundle`|The bundle range||
+
+
+### `terminate`
+Terminate a topic (disallow further messages from being published on the topic)
+
+Usage
+```bash
+$ pulsar-admin topics terminate {persistent|non-persistent}://tenant/namespace/topic
+```
+
+### `permissions`
+Get the permissions on a topic. Retrieve the effective permissions for a desination. These permissions are defined by the permissions set at the namespace level combined (union) with any eventual specific permissions set on the topic.
+
+Usage
+```bash
+$ pulsar-admin topics permissions topic
+```
+
+### `grant-permission`
+Grant a new permission to a client role on a single topic
+
+Usage
+```bash
+$ pulsar-admin topics grant-permission {persistent|non-persistent}://tenant/namespace/topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--actions`|Actions to be granted (`produce` or `consume`)||
+|`--role`|The client role to which to grant the permissions||
+
+
+### `revoke-permission`
+Revoke permissions to a client role on a single topic. If the permission was not set at the topic level, but rather at the namespace level, this operation will return an error (HTTP status code 412).
+
+Usage
+```bash
+$ pulsar-admin topics revoke-permission topic
+```
+
+### `lookup`
+Look up a topic from the current serving broker
+
+Usage
+```bash
+$ pulsar-admin topics lookup topic
+```
+
+### `bundle-range`
+Get the namespace bundle which contains the given topic
+
+Usage
+```bash
+$ pulsar-admin topics bundle-range topic
+```
+
+### `delete`
+Delete a topic. The topic cannot be deleted if there are any active subscriptions or producers connected to the topic.
+
+Usage
+```bash
+$ pulsar-admin topics delete topic
+```
+
+### `unload`
+Unload a topic
+
+Usage
+```bash
+$ pulsar-admin topics unload topic
+```
+
+### `subscriptions`
+Get the list of subscriptions on the topic
+
+Usage
+```bash
+$ pulsar-admin topics subscriptions topic
+```
+
+### `unsubscribe`
+Delete a durable subscriber from a topic
+
+Usage
+```bash
+$ pulsar-admin topics unsubscribe topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-s`, `--subscription`|The subscription to delete||
+
+
+### `stats`
+Get the stats for the topic and its connected producers and consumers. All rates are computed over a 1-minute window and are relative to the last completed 1-minute period.
+
+Usage
+```bash
+$ pulsar-admin topics stats topic
+```
+
+### `stats-internal`
+Get the internal stats for the topic
+
+Usage
+```bash
+$ pulsar-admin topics stats-internal topic
+```
+
+### `info-internal`
+Get the internal metadata info for the topic
+
+Usage
+```bash
+$ pulsar-admin topics info-internal topic
+```
+
+### `partitioned-stats`
+Get the stats for the partitioned topic and its connected producers and consumers. All rates are computed over a 1-minute window and are relative to the last completed 1-minute period.
+
+Usage
+```bash
+$ pulsar-admin topics partitioned-stats topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--per-partition`|Get per-partition stats|false|
+
+
+### `skip`
+Skip some messages for the subscription
+
+Usage
+```bash
+$ pulsar-admin topics skip topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-n`, `--count`|The number of messages to skip|0|
+|`-s`, `--subscription`|The subscription on which to skip messages||
+
+
+### `skip-all`
+Skip all the messages for the subscription
+
+Usage
+```bash
+$ pulsar-admin topics skip-all topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-s`, `--subscription`|The subscription to clear||
+
+
+### `expire-messages`
+Expire messages that are older than the given expiry time (in seconds) for the subscription.
+
+Usage
+```bash
+$ pulsar-admin topics expire-messages topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-t`, `--expireTime`|Expire messages older than the time (in seconds)|0|
+|`-s`, `--subscription`|The subscription to skip messages on||
+
+
+### `expire-messages-all-subscriptions`
+Expire messages older than the given expiry time (in seconds) for all subscriptions
+
+Usage
+```bash
+$ pulsar-admin topics expire-messages-all-subscriptions topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-t`, `--expireTime`|Expire messages older than the time (in seconds)|0|
+
+
+### `peek-messages`
+Peek some messages for the subscription.
+
+Usage
+```bash
+$ pulsar-admin topics peek-messages topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-n`, `--count`|The number of messages|0|
+|`-s`, `--subscription`|Subscription to get messages from||
+
+
+### `reset-cursor`
+Reset position for subscription to closest to timestamp
+
+Usage
+```bash
+$ pulsar-admin topics reset-cursor topic options
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`-s`, `--subscription`|Subscription to reset position on||
+|`-t`, `--time`|The time, in minutes, to reset back to (or minutes, hours, days, weeks, etc.). Examples: `100m`, `3h`, `2d`, `5w`.||
+
+
+
+## `tenants`
+Operations for managing tenants
+
+Usage
+```bash
+$ pulsar-admin tenants subcommand
+```
+
+Subcommands
+* `list`
+* `get`
+* `create`
+* `update`
+* `delete`
+
+### `list`
+List the existing tenants
+
+Usage
+```bash
+$ pulsar-admin tenants list
+```
+
+### `get`
+Gets the configuration of a tenant
+
+Usage
+```bash
+$ pulsar-admin tenants get tenant-name
+```
+
+### `create`
+Creates a new tenant
+
+Usage
+```bash
+$ pulsar-admin tenants create tenant-name options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-r`, `--admin-roles`|Comma-separated admin roles||
+|`-c`, `--allowed-clusters`|Comma-separated allowed clusters||
+
+### `update`
+Updates a tenant
+
+Usage
+```bash
+$ pulsar-admin tenants update tenant-name options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-r`, `--admin-roles`|Comma-separated admin roles||
+|`-c`, `--allowed-clusters`|Comma-separated allowed clusters||
+
+
+### `delete`
+Deletes an existing tenant
+
+Usage
+```bash
+$ pulsar-admin tenants delete tenant-name
+```
+
+
+## `resource-quotas`
+Operations for managing resource quotas
+
+Usage
+```bash
+$ pulsar-admin resource-quotas subcommand
+```
+
+Subcommands
+* `get`
+* `set`
+* `reset-namespace-bundle-quota`
+
+
+### `get`
+Get the resource quota for a specified namespace bundle, or default quota if no namespace/bundle is specified.
+
+Usage
+```bash
+$ pulsar-admin resource-quotas get options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-b`, `--bundle`|A bundle of the form {start-boundary}_{end_boundary}. This must be specified together with -n/--namespace.||
+|`-n`, `--namespace`|The namespace||
+
+
+### `set`
+Set the resource quota for the specified namespace bundle, or default quota if no namespace/bundle is specified.
+
+Usage
+```bash
+$ pulsar-admin resource-quotas set options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-bi`, `--bandwidthIn`|The expected inbound bandwidth (in bytes/second)|0|
+|`-bo`, `--bandwidthOut`|Expected outbound bandwidth (in bytes/second)0|
+|`-b`, `--bundle`|A bundle of the form {start-boundary}_{end_boundary}. This must be specified together with -n/--namespace.||
+|`-d`, `--dynamic`|Allow to be dynamically re-calculated (or not)|false|
+|`-mem`, `--memory`|Expectred memory usage (in megabytes)|0|
+|`-mi`, `--msgRateIn`|Expected incoming messages per second|0|
+|`-mo`, `--msgRateOut`|Expected outgoing messages per second|0|
+|`-n`, `--namespace`|The namespace as tenant/namespace, for example my-tenant/my-ns. Must be specified together with -b/--bundle.||
+
+
+### `reset-namespace-bundle-quota`
+Reset the specifed namespace bundle's resource quota to a default value.
+
+Usage
+```bash
+$ pulsar-admin resource-quotas reset-namespace-bundle-quota options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`-b`, `--bundle`|A bundle of the form {start-boundary}_{end_boundary}. This must be specified together with -n/--namespace.||
+|`-n`, `--namespace`|The namespace||
+
+
+
+## `schemas`
+Operations related to Schemas associated with Pulsar topics.
+
+Usage
+```
+$ pulsar-admin schemas subcommand
+```
+
+Subcommands
+* `upload`
+* `delete`
+* `get`
+
+
+### `upload`
+Upload the schema definition for a topic
+
+Usage
+```bash
+$ pulsar-admin schemas upload persistent://tenant/namespace/topic options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--filename`|The path to the schema definition file. An example schema file is available under conf directory.||
+
+
+### `delete`
+Delete the schema definition associated with a topic
+
+Usage
+```bash
+$ pulsar-admin schemas delete persistent://tenant/namespace/topic
+```
+
+
+### `get`
+Retrieve the schema definition assoicated with a topic (at a given version if version is supplied).
+
+Usage
+```bash
+$ pulsar-admin schemas get persistent://tenant/namespace/topic options
+```
+
+Options
+|Flag|Description|Default|
+|----|---|---|
+|`--version`|The version of the schema definition to retrive for a topic.||
+
+
diff --git a/site2/docs/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.1.1-incubating/sql-deployment-configurations.md
similarity index 98%
rename from site2/docs/sql-deployment-configurations.md
rename to site2/website/versioned_docs/version-2.1.1-incubating/sql-deployment-configurations.md
index 2b22cd9..536e613 100644
--- a/site2/docs/sql-deployment-configurations.md
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/sql-deployment-configurations.md
@@ -1,7 +1,8 @@
---
-id: sql-deployment-configurations
+id: version-2.1.1-incubating-sql-deployment-configurations
title: Pulsar SQl Deployment and Configuration
sidebar_label: Deployment and Configuration
+original_id: sql-deployment-configurations
---
## Presto Pulsar Connector Configurations
diff --git a/site2/docs/sql-getting-started.md b/site2/website/versioned_docs/version-2.1.1-incubating/sql-getting-started.md
similarity index 98%
rename from site2/docs/sql-getting-started.md
rename to site2/website/versioned_docs/version-2.1.1-incubating/sql-getting-started.md
index cbadbed..aec0934 100644
--- a/site2/docs/sql-getting-started.md
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/sql-getting-started.md
@@ -1,7 +1,8 @@
---
-id: sql-getting-started
+id: version-2.1.1-incubating-sql-getting-started
title: Pulsar SQL Overview
sidebar_label: Overview
+original_id: sql-getting-started
---
It is super easy to start query data in Pulsar.
diff --git a/site2/docs/sql-overview.md b/site2/website/versioned_docs/version-2.1.1-incubating/sql-overview.md
similarity index 96%
rename from site2/docs/sql-overview.md
rename to site2/website/versioned_docs/version-2.1.1-incubating/sql-overview.md
index 1df9533..494fa9d 100644
--- a/site2/docs/sql-overview.md
+++ b/site2/website/versioned_docs/version-2.1.1-incubating/sql-overview.md
@@ -1,7 +1,8 @@
---
-id: sql-overview
+id: version-2.1.1-incubating-sql-overview
title: Pulsar SQL Overview
sidebar_label: Overview
+original_id: sql-overview
---
One of the common use cases of Pulsar is storing streams of event data. Often the event data is structured which predefined fields. There is tremendous value for users to be able to query the existing data that is already stored in Pulsar topics. With the implementation of the [Schema Registry](concepts-schema-registry.md), structured data can be stored in Pulsar and allows for the potential to query that data via SQL language.
diff --git a/site2/website/versions.json b/site2/website/versions.json
index 627406a..bdeecc7 100644
--- a/site2/website/versions.json
+++ b/site2/website/versions.json
@@ -1,3 +1,4 @@
[
+ "2.1.1-incubating",
"2.1.0-incubating"
]