You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by mm...@apache.org on 2018/08/11 08:37:42 UTC
[incubator-pulsar] branch asf-site updated: Updated site at
revision cb16f55
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 2c374ef Updated site at revision cb16f55
2c374ef is described below
commit 2c374ef0ef11fce607b7ea8748dc7301b5358cae
Author: jenkins <bu...@apache.org>
AuthorDate: Sat Aug 11 08:37:35 2018 +0000
Updated site at revision cb16f55
---
content/docs/en/next/deploy-aws.html | 32 ++++++-
content/docs/en/next/deploy-aws/index.html | 32 ++++++-
content/docs/en/next/deploy-bare-metal.html | 99 ++++++++++++++++++-
content/docs/en/next/deploy-bare-metal/index.html | 99 ++++++++++++++++++-
content/docs/fr/next/deploy-aws.html | 27 +++++-
content/docs/fr/next/deploy-aws/index.html | 27 +++++-
content/docs/fr/next/deploy-bare-metal.html | 106 +++++++++++++++++++--
content/docs/fr/next/deploy-bare-metal/index.html | 106 +++++++++++++++++++--
content/docs/ja/next/deploy-aws.html | 27 +++++-
content/docs/ja/next/deploy-aws/index.html | 27 +++++-
content/docs/ja/next/deploy-bare-metal.html | 106 +++++++++++++++++++--
content/docs/ja/next/deploy-bare-metal/index.html | 106 +++++++++++++++++++--
content/docs/latest/reference/RestApi/index.html | 45 +++++++++
content/docs/zh-CN/next/deploy-aws.html | 27 +++++-
content/docs/zh-CN/next/deploy-aws/index.html | 27 +++++-
content/docs/zh-CN/next/deploy-bare-metal.html | 106 +++++++++++++++++++--
.../docs/zh-CN/next/deploy-bare-metal/index.html | 106 +++++++++++++++++++--
content/ja/reference/RestApi/index.html | 45 +++++++++
content/swagger/swagger.json | 69 ++++++++++----
19 files changed, 1126 insertions(+), 93 deletions(-)
diff --git a/content/docs/en/next/deploy-aws.html b/content/docs/en/next/deploy-aws.html
index cf705f6..b1e2d07 100644
--- a/content/docs/en/next/deploy-aws.html
+++ b/content/docs/en/next/deploy-aws.html
@@ -65,6 +65,22 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting
+in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>,
+you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell">private_key_file=~/.ssh/pulsar_aws
+</code></pre>
+<ol start="2">
+<li>update <code>terraform.tfvars</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell">public_key_path=~/.ssh/pulsar_aws.pub
+</code></pre>
+</blockquote>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +147,19 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes.
+Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform
+config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk.
+It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook,
+it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +171,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +190,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/en/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/en/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/en/next/deploy-aws/index.html b/content/docs/en/next/deploy-aws/index.html
index cf705f6..b1e2d07 100644
--- a/content/docs/en/next/deploy-aws/index.html
+++ b/content/docs/en/next/deploy-aws/index.html
@@ -65,6 +65,22 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting
+in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>,
+you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell">private_key_file=~/.ssh/pulsar_aws
+</code></pre>
+<ol start="2">
+<li>update <code>terraform.tfvars</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell">public_key_path=~/.ssh/pulsar_aws.pub
+</code></pre>
+</blockquote>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +147,19 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes.
+Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform
+config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk.
+It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook,
+it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +171,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +190,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/en/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/en/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/en/next/deploy-bare-metal.html b/content/docs/en/next/deploy-bare-metal.html
index f081bfe..65b0ced 100644
--- a/content/docs/en/next/deploy-bare-metal.html
+++ b/content/docs/en/next/deploy-bare-metal.html
@@ -54,18 +54,23 @@ have run a separate cluster of function workers for <a href="/docs/en/next/funct
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines
+for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -75,7 +80,9 @@ have run a separate cluster of function workers for <a href="/docs/en/next/funct
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -114,7 +121,47 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors.
+If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can
+skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in
+one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code>
+in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/en/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -166,6 +213,13 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/en/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/en/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature,
+you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf">extraServerComponents=org<span class="hljs-selector-class">.apache</span><span class="hljs-selector-class">.bookkeeper</span><span class="hljs-selector-class">.stream</span><span class="hljs-selector-class">.server</span><span class="hljs-selector-class">.StreamStorageLifecycleComponent</span>
+</code></pre>
+</blockquote>
<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
<p>To start the bookie in the background, use the <a href="/docs/en/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
@@ -173,12 +227,18 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/en/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/en/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to
+verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/en/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/en/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, the <c [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -186,6 +246,17 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/en/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/en/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/en/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -210,7 +281,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/en/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cluster" [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/en/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/en/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardware-co [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/en/next/deploy-bare-metal/index.html b/content/docs/en/next/deploy-bare-metal/index.html
index f081bfe..65b0ced 100644
--- a/content/docs/en/next/deploy-bare-metal/index.html
+++ b/content/docs/en/next/deploy-bare-metal/index.html
@@ -54,18 +54,23 @@ have run a separate cluster of function workers for <a href="/docs/en/next/funct
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines
+for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -75,7 +80,9 @@ have run a separate cluster of function workers for <a href="/docs/en/next/funct
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -114,7 +121,47 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors.
+If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can
+skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in
+one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code>
+in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/en/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -166,6 +213,13 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/en/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/en/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature,
+you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf">extraServerComponents=org<span class="hljs-selector-class">.apache</span><span class="hljs-selector-class">.bookkeeper</span><span class="hljs-selector-class">.stream</span><span class="hljs-selector-class">.server</span><span class="hljs-selector-class">.StreamStorageLifecycleComponent</span>
+</code></pre>
+</blockquote>
<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
<p>To start the bookie in the background, use the <a href="/docs/en/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
@@ -173,12 +227,18 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/en/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/en/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to
+verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/en/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/en/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, the <c [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -186,6 +246,17 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/en/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/en/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/en/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -210,7 +281,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/en/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cluster" [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/en/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/en/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/en/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardware-co [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/fr/next/deploy-aws.html b/content/docs/fr/next/deploy-aws.html
index 105df2e..f3a623a 100644
--- a/content/docs/fr/next/deploy-aws.html
+++ b/content/docs/fr/next/deploy-aws.html
@@ -65,6 +65,21 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>, you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell"></code></pre>
+</blockquote>
+<p>private_key_file=~/.ssh/pulsar_aws</p>
+<pre><code class="hljs">
+1. <span class="hljs-keyword">update</span> <span class="hljs-symbol">`terraform.tfvars`</span> <span class="hljs-keyword">with</span> following <span class="hljs-keyword">values</span>:
+
+<span class="hljs-symbol">``</span><span class="hljs-symbol">`shell
+public_key_path=~/.ssh/pulsar_aws.pub
+</span></code></pre>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +146,15 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes. Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk. It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook, it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +166,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +185,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/fr/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/fr/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/fr/next/deploy-aws/index.html b/content/docs/fr/next/deploy-aws/index.html
index 105df2e..f3a623a 100644
--- a/content/docs/fr/next/deploy-aws/index.html
+++ b/content/docs/fr/next/deploy-aws/index.html
@@ -65,6 +65,21 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>, you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell"></code></pre>
+</blockquote>
+<p>private_key_file=~/.ssh/pulsar_aws</p>
+<pre><code class="hljs">
+1. <span class="hljs-keyword">update</span> <span class="hljs-symbol">`terraform.tfvars`</span> <span class="hljs-keyword">with</span> following <span class="hljs-keyword">values</span>:
+
+<span class="hljs-symbol">``</span><span class="hljs-symbol">`shell
+public_key_path=~/.ssh/pulsar_aws.pub
+</span></code></pre>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +146,15 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes. Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk. It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook, it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +166,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +185,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/fr/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/fr/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/fr/next/deploy-bare-metal.html b/content/docs/fr/next/deploy-bare-metal.html
index 915e709..14ac7a2 100644
--- a/content/docs/fr/next/deploy-bare-metal.html
+++ b/content/docs/fr/next/deploy-bare-metal.html
@@ -50,18 +50,22 @@
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -71,7 +75,9 @@
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -110,7 +116,43 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors. If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code> in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/fr/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -162,19 +204,34 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/fr/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
-<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
-<p>To start the bookie in the background, use the <a href="/docs/fr/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
-<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
-</code></pre>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/fr/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature, you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf"></code></pre>
+</blockquote>
+<p>extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent</p>
+<pre><code class="hljs">
+Once you've applied the desired configuration in <span class="hljs-code">`conf/bookkeeper.conf`</span>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using [<span class="hljs-string">nohup</span>](<span class="hljs-link">https://en.wikipedia.org/wiki/Nohup</span>), or in the foreground.
+
+To start the bookie in the background, use the [<span class="hljs-string">`pulsar-daemon`</span>](<span class="hljs-link">/docs/fr/next/reference-cli-tools#pulsar-daemon</span>) CLI tool:
+
+<span class="hljs-code">```bash
+$ bin/pulsar-daemon start bookie
+</span></code></pre>
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/fr/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/fr/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/fr/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/fr/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, the <c [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -182,6 +239,19 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/fr/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+</ol>
+<ol start="2">
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/fr/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/fr/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -206,7 +276,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/fr/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cluster" [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/fr/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/fr/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardware-co [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/fr/next/deploy-bare-metal/index.html b/content/docs/fr/next/deploy-bare-metal/index.html
index 915e709..14ac7a2 100644
--- a/content/docs/fr/next/deploy-bare-metal/index.html
+++ b/content/docs/fr/next/deploy-bare-metal/index.html
@@ -50,18 +50,22 @@
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -71,7 +75,9 @@
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -110,7 +116,43 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors. If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code> in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/fr/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -162,19 +204,34 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/fr/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
-<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
-<p>To start the bookie in the background, use the <a href="/docs/fr/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
-<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
-</code></pre>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/fr/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature, you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf"></code></pre>
+</blockquote>
+<p>extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent</p>
+<pre><code class="hljs">
+Once you've applied the desired configuration in <span class="hljs-code">`conf/bookkeeper.conf`</span>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using [<span class="hljs-string">nohup</span>](<span class="hljs-link">https://en.wikipedia.org/wiki/Nohup</span>), or in the foreground.
+
+To start the bookie in the background, use the [<span class="hljs-string">`pulsar-daemon`</span>](<span class="hljs-link">/docs/fr/next/reference-cli-tools#pulsar-daemon</span>) CLI tool:
+
+<span class="hljs-code">```bash
+$ bin/pulsar-daemon start bookie
+</span></code></pre>
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/fr/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/fr/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/fr/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/fr/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, the <c [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -182,6 +239,19 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/fr/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+</ol>
+<ol start="2">
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/fr/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/fr/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -206,7 +276,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/fr/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cluster" [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/fr/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/fr/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/fr/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardware-co [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/ja/next/deploy-aws.html b/content/docs/ja/next/deploy-aws.html
index 2bdd0f4..37c5ba8 100644
--- a/content/docs/ja/next/deploy-aws.html
+++ b/content/docs/ja/next/deploy-aws.html
@@ -65,6 +65,21 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>, you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell"></code></pre>
+</blockquote>
+<p>private_key_file=~/.ssh/pulsar_aws</p>
+<pre><code class="hljs">
+1. <span class="hljs-keyword">update</span> <span class="hljs-symbol">`terraform.tfvars`</span> <span class="hljs-keyword">with</span> following <span class="hljs-keyword">values</span>:
+
+<span class="hljs-symbol">``</span><span class="hljs-symbol">`shell
+public_key_path=~/.ssh/pulsar_aws.pub
+</span></code></pre>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +146,15 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes. Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk. It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook, it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +166,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +185,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/ja/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/ja/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/ja/next/deploy-aws/index.html b/content/docs/ja/next/deploy-aws/index.html
index 2bdd0f4..37c5ba8 100644
--- a/content/docs/ja/next/deploy-aws/index.html
+++ b/content/docs/ja/next/deploy-aws/index.html
@@ -65,6 +65,21 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>, you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell"></code></pre>
+</blockquote>
+<p>private_key_file=~/.ssh/pulsar_aws</p>
+<pre><code class="hljs">
+1. <span class="hljs-keyword">update</span> <span class="hljs-symbol">`terraform.tfvars`</span> <span class="hljs-keyword">with</span> following <span class="hljs-keyword">values</span>:
+
+<span class="hljs-symbol">``</span><span class="hljs-symbol">`shell
+public_key_path=~/.ssh/pulsar_aws.pub
+</span></code></pre>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +146,15 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes. Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk. It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook, it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +166,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +185,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/ja/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/ja/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#creating [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/ja/next/deploy-bare-metal.html b/content/docs/ja/next/deploy-bare-metal.html
index 5082c22..d6fe5a2 100644
--- a/content/docs/ja/next/deploy-bare-metal.html
+++ b/content/docs/ja/next/deploy-bare-metal.html
@@ -50,18 +50,22 @@
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -71,7 +75,9 @@
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -110,7 +116,43 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors. If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code> in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/ja/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -162,19 +204,34 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/ja/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
-<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
-<p>To start the bookie in the background, use the <a href="/docs/ja/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
-<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
-</code></pre>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/ja/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature, you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf"></code></pre>
+</blockquote>
+<p>extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent</p>
+<pre><code class="hljs">
+Once you've applied the desired configuration in <span class="hljs-code">`conf/bookkeeper.conf`</span>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using [<span class="hljs-string">nohup</span>](<span class="hljs-link">https://en.wikipedia.org/wiki/Nohup</span>), or in the foreground.
+
+To start the bookie in the background, use the [<span class="hljs-string">`pulsar-daemon`</span>](<span class="hljs-link">/docs/ja/next/reference-cli-tools#pulsar-daemon</span>) CLI tool:
+
+<span class="hljs-code">```bash
+$ bin/pulsar-daemon start bookie
+</span></code></pre>
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/ja/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/ja/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/ja/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/ja/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, the <c [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -182,6 +239,19 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/ja/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+</ol>
+<ol start="2">
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/ja/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/ja/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -206,7 +276,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/ja/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cluster" [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/ja/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/ja/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardware-co [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/ja/next/deploy-bare-metal/index.html b/content/docs/ja/next/deploy-bare-metal/index.html
index 5082c22..d6fe5a2 100644
--- a/content/docs/ja/next/deploy-bare-metal/index.html
+++ b/content/docs/ja/next/deploy-bare-metal/index.html
@@ -50,18 +50,22 @@
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -71,7 +75,9 @@
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -110,7 +116,43 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors. If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code> in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/ja/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -162,19 +204,34 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/ja/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
-<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
-<p>To start the bookie in the background, use the <a href="/docs/ja/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
-<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
-</code></pre>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/ja/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature, you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf"></code></pre>
+</blockquote>
+<p>extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent</p>
+<pre><code class="hljs">
+Once you've applied the desired configuration in <span class="hljs-code">`conf/bookkeeper.conf`</span>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using [<span class="hljs-string">nohup</span>](<span class="hljs-link">https://en.wikipedia.org/wiki/Nohup</span>), or in the foreground.
+
+To start the bookie in the background, use the [<span class="hljs-string">`pulsar-daemon`</span>](<span class="hljs-link">/docs/ja/next/reference-cli-tools#pulsar-daemon</span>) CLI tool:
+
+<span class="hljs-code">```bash
+$ bin/pulsar-daemon start bookie
+</span></code></pre>
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/ja/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/ja/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/ja/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/ja/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, the <c [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -182,6 +239,19 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/ja/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+</ol>
+<ol start="2">
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/ja/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/ja/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -206,7 +276,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/ja/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cluster" [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/ja/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/ja/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/ja/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardware-co [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/latest/reference/RestApi/index.html b/content/docs/latest/reference/RestApi/index.html
index 60bac06..8547577 100644
--- a/content/docs/latest/reference/RestApi/index.html
+++ b/content/docs/latest/reference/RestApi/index.html
@@ -8808,6 +8808,11 @@
<td>404</td>
<td>Topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -8962,6 +8967,11 @@
<td>409</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -9119,6 +9129,11 @@
<td>404</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11589,6 +11604,11 @@
<td>404</td>
<td>Topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11675,6 +11695,16 @@
<td>403</td>
<td>Don't have admin permission</td>
</tr>
+
+ <tr>
+ <td>409</td>
+ <td>Partitioned topic does not exist</td>
+ </tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11743,6 +11773,11 @@
<td>409</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11818,6 +11853,11 @@
<td>409</td>
<td>Partitioned topic already exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11900,6 +11940,11 @@
<td>404</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
diff --git a/content/docs/zh-CN/next/deploy-aws.html b/content/docs/zh-CN/next/deploy-aws.html
index 69d791c..e6f8009 100644
--- a/content/docs/zh-CN/next/deploy-aws.html
+++ b/content/docs/zh-CN/next/deploy-aws.html
@@ -65,6 +65,21 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>, you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell"></code></pre>
+</blockquote>
+<p>private_key_file=~/.ssh/pulsar_aws</p>
+<pre><code class="hljs">
+1. <span class="hljs-keyword">update</span> <span class="hljs-symbol">`terraform.tfvars`</span> <span class="hljs-keyword">with</span> following <span class="hljs-keyword">values</span>:
+
+<span class="hljs-symbol">``</span><span class="hljs-symbol">`shell
+public_key_path=~/.ssh/pulsar_aws.pub
+</span></code></pre>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +146,15 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes. Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk. It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook, it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +166,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +185,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#cr [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#cr [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/zh-CN/next/deploy-aws/index.html b/content/docs/zh-CN/next/deploy-aws/index.html
index 69d791c..e6f8009 100644
--- a/content/docs/zh-CN/next/deploy-aws/index.html
+++ b/content/docs/zh-CN/next/deploy-aws/index.html
@@ -65,6 +65,21 @@
$ <span class="hljs-built_in">cd</span> incubator-pulsar/deployment/terraform-ansible/aws
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="ssh-setup"></a><a href="#ssh-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
+<blockquote>
+<p>If you already have an SSH key and would like to use it, you skip generating the SSH keys and update <code>private_key_file</code> setting in <code>ansible.cfg</code> file and <code>public_key_path</code> setting in <code>terraform.tfvars</code> file.</p>
+<p>For example, if you already had a private SSH key in <code>~/.ssh/pulsar_aws</code> and a public key in <code>~/.ssh/pulsar_aws.pub</code>, you can do followings:</p>
+<ol>
+<li>update <code>ansible.cfg</code> with following values:</li>
+</ol>
+<pre><code class="hljs css languages- shell"></code></pre>
+</blockquote>
+<p>private_key_file=~/.ssh/pulsar_aws</p>
+<pre><code class="hljs">
+1. <span class="hljs-keyword">update</span> <span class="hljs-symbol">`terraform.tfvars`</span> <span class="hljs-keyword">with</span> following <span class="hljs-keyword">values</span>:
+
+<span class="hljs-symbol">``</span><span class="hljs-symbol">`shell
+public_key_path=~/.ssh/pulsar_aws.pub
+</span></code></pre>
<p>In order to create the necessary AWS resources using Terraform, you'll need to create an SSH key. To create a private SSH key in <code>~/.ssh/id_rsa</code> and a public key in <code>~/.ssh/id_rsa.pub</code>:</p>
<pre><code class="hljs css languages- bash">$ ssh-keygen -t rsa
</code></pre>
@@ -131,6 +146,15 @@ id_rsa id_rsa.pub
<p>At any point, you can destroy all AWS resources associated with your cluster using Terraform's <code>destroy</code> command:</p>
<pre><code class="hljs css languages- bash">$ terraform destroy
</code></pre>
+<h2><a class="anchor" aria-hidden="true" id="setup-disks"></a><a href="#setup-disks" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
+<p>Before you run the Pulsar playbook, you want to mount the disks to the correct directories on those bookie nodes. Since different type of machines would have different disk layout, if you change the <code>instance_types</code> in your terraform config, you need to update the task defined in <code>setup-disk.yaml</code> file.</p>
+<p>To setup disks on bookie nodes, use this command:</p>
+<pre><code class="hljs css languages- bash">$ ansible-playbook \
+ --user=<span class="hljs-string">'ec2-user'</span> \
+ --inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
+ setup-disk.yaml
+</code></pre>
+<p>After running this command, the disks will be mounted under <code>/mnt/journal</code> as journal disk, and <code>/mnt/storage</code> as ledger disk. It is important to run this command only once! If you attempt to run this command again after you have run Pulsar playbook, it might be potentially erase your disks again and cause the bookies to fail to start up.</p>
<h2><a class="anchor" aria-hidden="true" id="running-the-pulsar-playbook"></a><a href="#running-the-pulsar-playbook" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 1 [...]
<p>Once you've created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. To do so, use this command:</p>
<pre><code class="hljs css languages- bash">$ ansible-playbook \
@@ -142,7 +166,6 @@ id_rsa id_rsa.pub
<pre><code class="hljs css languages- bash">$ ansible-playbook \
--user=<span class="hljs-string">'ec2-user'</span> \
--inventory=`<span class="hljs-built_in">which</span> terraform-inventory` \
- --private-key=<span class="hljs-string">"~/.ssh/some-non-default-key"</span> \
../deploy-pulsar.yaml
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="accessing-the-cluster"></a><a href="#accessing-the-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2 [...]
@@ -162,7 +185,7 @@ id_rsa id_rsa.pub
<span class="hljs-meta">>>> </span>client.close()
</code></pre>
<p>If all of these commands are successful, your cluster can now be used by Pulsar clients!</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#cr [...]
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/io-develop">← Develop Connectors</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-kubernetes">Deploying Pulsar on Kubernetes →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#requirements-and-setup">Requirements and setup</a></li><li><a href="#installation">Installation</a></li><li><a href="#ssh-setup">SSH setup</a></li><li><a href="#cr [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/zh-CN/next/deploy-bare-metal.html b/content/docs/zh-CN/next/deploy-bare-metal.html
index 9731d9f..c3d76ac 100644
--- a/content/docs/zh-CN/next/deploy-bare-metal.html
+++ b/content/docs/zh-CN/next/deploy-bare-metal.html
@@ -50,18 +50,22 @@
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -71,7 +75,9 @@
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -110,7 +116,43 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors. If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code> in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/zh-CN/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -162,19 +204,34 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/zh-CN/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
-<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
-<p>To start the bookie in the background, use the <a href="/docs/zh-CN/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
-<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
-</code></pre>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/zh-CN/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature, you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf"></code></pre>
+</blockquote>
+<p>extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent</p>
+<pre><code class="hljs">
+Once you've applied the desired configuration in <span class="hljs-code">`conf/bookkeeper.conf`</span>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using [<span class="hljs-string">nohup</span>](<span class="hljs-link">https://en.wikipedia.org/wiki/Nohup</span>), or in the foreground.
+
+To start the bookie in the background, use the [<span class="hljs-string">`pulsar-daemon`</span>](<span class="hljs-link">/docs/zh-CN/next/reference-cli-tools#pulsar-daemon</span>) CLI tool:
+
+<span class="hljs-code">```bash
+$ bin/pulsar-daemon start bookie
+</span></code></pre>
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/zh-CN/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/zh-CN/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/zh-CN/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/zh-CN/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -182,6 +239,19 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/zh-CN/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+</ol>
+<ol start="2">
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/zh-CN/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/zh-CN/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -206,7 +276,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cl [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/zh-CN/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardw [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/docs/zh-CN/next/deploy-bare-metal/index.html b/content/docs/zh-CN/next/deploy-bare-metal/index.html
index 9731d9f..c3d76ac 100644
--- a/content/docs/zh-CN/next/deploy-bare-metal/index.html
+++ b/content/docs/zh-CN/next/deploy-bare-metal/index.html
@@ -50,18 +50,22 @@
</blockquote>
<p>Deploying a Pulsar cluster involves doing the following (in order):</p>
<ul>
-<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster</li>
+<li>Deploying a <a href="#deploying-a-zookeeper-cluster">ZooKeeper</a> cluster (optional)</li>
<li>Initializing <a href="#initializing-cluster-metadata">cluster metadata</a></li>
<li>Deploying a <a href="#deploying-a-bookkeeper-cluster">BookKeeper</a> cluster</li>
<li>Deploying one or more Pulsar <a href="#deploying-pulsar-brokers">brokers</a></li>
</ul>
+<h2><a class="anchor" aria-hidden="true" id="preparation"></a><a href="#preparation" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42 [...]
<h3><a class="anchor" aria-hidden="true" id="requirements"></a><a href="#requirements" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83. [...]
+<blockquote>
+<p>If you already have an existing zookeeper cluster and would like to reuse it, you don't need to prepare the machines for running ZooKeeper.</p>
+</blockquote>
<p>To run Pulsar on bare metal, you will need:</p>
<ul>
<li>At least 6 Linux machines or VMs
<ul>
<li>3 running <a href="https://zookeeper.apache.org">ZooKeeper</a></li>
-<li>3 running a Pulsar broker and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
+<li>3 running a Pulsar broker, and a <a href="https://bookkeeper.apache.org">BookKeeper</a> bookie</li>
</ul></li>
<li>A single <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> name covering all of the Pulsar broker hosts</li>
</ul>
@@ -71,7 +75,9 @@
<p>In this diagram, connecting clients need to be able to communicate with the Pulsar cluster using a single URL, in this case <code>pulsar-cluster.acme.com</code>, that abstracts over all of the message-handling brokers. Pulsar message brokers run on machines alongside BookKeeper bookies; brokers and bookies, in turn, rely on ZooKeeper.</p>
<h3><a class="anchor" aria-hidden="true" id="hardware-considerations"></a><a href="#hardware-considerations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 [...]
<p>When deploying a Pulsar cluster, we have some basic recommendations that you should keep in mind when capacity planning.</p>
+<h4><a class="anchor" aria-hidden="true" id="zookeeper"></a><a href="#zookeeper" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.6 [...]
<p>For machines running ZooKeeper, we recommend using lighter-weight machines or VMs. Pulsar uses ZooKeeper only for periodic coordination- and configuration-related tasks, <em>not</em> for basic operations. If you're running Pulsar on <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), for example, a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">t2.small</a> instance would likely suffice.</p>
+<h4><a class="anchor" aria-hidden="true" id="bookies-brokers"></a><a href="#bookies-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 [...]
<p>For machines running a bookie and a Pulsar broker, we recommend using more powerful machines. For an AWS deployment, for example, <a href="https://aws.amazon.com/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/">i3.4xlarge</a> instances may be appropriate. On those machines we also recommend:</p>
<ul>
<li>Fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller">NIC</a> (for Pulsar brokers)</li>
@@ -110,7 +116,43 @@ $ <span class="hljs-built_in">cd</span> apache-pulsar-2.1.0-incubating
<tr><td style="text-align:left"><code>logs</code></td><td style="text-align:left">Logs created by the installation.</td></tr>
</tbody>
</table>
+<h2><a class="anchor" aria-hidden="true" id="installing-builtin-connectors-optional"></a><a href="#installing-builtin-connectors-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1. [...]
+<blockquote>
+<p>Since release <code>2.1.0-incubating</code>, Pulsar releases a separate binary distribution, containing all the <code>builtin</code> connectors. If you would like to enable those <code>builtin</code> connectors, you can follow the instructions as below; otherwise you can skip this section for now.</p>
+</blockquote>
+<p>To get started using builtin connectors, you'll need to download the connectors tarball release on every broker node in one of the following ways:</p>
+<ul>
+<li><p>by clicking the link below and downloading the release from an Apache mirror:</p>
+<ul>
+<li><a href="https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz" download>Pulsar IO Connectors 2.1.0-incubating release</a></li>
+</ul></li>
+<li><p>from the Pulsar <a href="/download">downloads page</a></p></li>
+<li><p>from the Pulsar <a href="https://github.com/apache/incubator-pulsar/releases/latest">releases page</a></p></li>
+<li><p>using <a href="https://www.gnu.org/software/wget">wget</a>:</p>
+<pre><code class="hljs css languages- shell"><span class="hljs-meta">$</span><span class="bash"> wget https://archive.apache.org/dist/incubator/pulsar/pulsar-2.1.0-incubating/apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz</span>
+</code></pre></li>
+</ul>
+<p>Once the tarball is downloaded, in the pulsar directory, untar the io-connectors package and copy the connectors as <code>connectors</code> in the pulsar directory:</p>
+<pre><code class="hljs css languages- bash">$ tar xvfz apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
+
+// you will find a directory named `apache-pulsar-io-connectors-2.1.0-incubating` <span class="hljs-keyword">in</span> the pulsar directory
+// <span class="hljs-keyword">then</span> copy the connectors
+
+$ mv apache-pulsar-io-connectors-2.1.0-incubating/connectors connectors
+
+$ ls connectors
+pulsar-io-aerospike-{{pulsar.version}}.nar
+pulsar-io-cassandra-{{pulsar.version}}.nar
+pulsar-io-kafka-{{pulsar.version}}.nar
+pulsar-io-kinesis-{{pulsar.version}}.nar
+pulsar-io-rabbitmq-{{pulsar.version}}.nar
+pulsar-io-twitter-{{pulsar.version}}.nar
+...
+</code></pre>
<h2><a class="anchor" aria-hidden="true" id="deploying-a-zookeeper-cluster"></a><a href="#deploying-a-zookeeper-cluster" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 [...]
+<blockquote>
+<p>If you already have an exsiting zookeeper cluster and would like to use it, you can skip this section.</p>
+</blockquote>
<p><a href="https://zookeeper.apache.org">ZooKeeper</a> manages a variety of essential coordination- and configuration-related tasks for Pulsar. To deploy a Pulsar cluster you'll need to deploy ZooKeeper first (before all other components). We recommend deploying a 3-node ZooKeeper cluster. Pulsar does not make heavy use of ZooKeeper, so more lightweight machines or VMs should suffice for running ZooKeeper.</p>
<p>To begin, add all ZooKeeper servers to the configuration specified in <a href="/docs/zh-CN/next/reference-configuration#zookeeper"><code>conf/zookeeper.conf</code></a> (in the Pulsar directory you created <a href="#installing-the-pulsar-binary-package">above</a>). Here's an example:</p>
<pre><code class="hljs css languages- properties">server.<span class="hljs-number">1</span>=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2888</span>:<span class="hljs-number">3888</span>
@@ -162,19 +204,34 @@ $ <span class="hljs-built_in">echo</span> 1 > data/zookeeper/myid
<pre><code class="hljs css languages- properties">zkServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span clas [...]
</code></pre>
<p>Once you've appropriately modified the <code>zkServers</code> parameter, you can provide any other configuration modifications you need. You can find a full listing of the available BookKeeper configuration parameters <a href="/docs/zh-CN/next/reference-configuration#bookkeeper">here</a>, although we would recommend consulting the <a href="http://bookkeeper.apache.org/docs/latest/reference/config/">BookKeeper documentation</a> for a more in-depth guide.</p>
-<p>Once you've applied the desired configuration in <code>conf/bookkeeper.conf</code>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using <a href="https://en.wikipedia.org/wiki/Nohup">nohup</a>, or in the foreground.</p>
-<p>To start the bookie in the background, use the <a href="/docs/zh-CN/next/reference-cli-tools#pulsar-daemon"><code>pulsar-daemon</code></a> CLI tool:</p>
-<pre><code class="hljs css languages- bash">$ bin/pulsar-daemon start bookie
-</code></pre>
+<blockquote>
+<h5><a class="anchor" aria-hidden="true" id="notes"></a><a href="#notes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09 [...]
+<p>Since Pulsar 2.1.0 release, Pulsar introduces <a href="/docs/zh-CN/next/functions-state">stateful function</a> for Pulsar Functions. If you would like to enable that feature, you need to enable table service on BookKeeper by setting following setting in <code>conf/bookkeeper.conf</code> file.</p>
+<pre><code class="hljs css languages- conf"></code></pre>
+</blockquote>
+<p>extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent</p>
+<pre><code class="hljs">
+Once you've applied the desired configuration in <span class="hljs-code">`conf/bookkeeper.conf`</span>, you can start up a bookie on each of your BookKeeper hosts. You can start up each bookie either in the background, using [<span class="hljs-string">nohup</span>](<span class="hljs-link">https://en.wikipedia.org/wiki/Nohup</span>), or in the foreground.
+
+To start the bookie in the background, use the [<span class="hljs-string">`pulsar-daemon`</span>](<span class="hljs-link">/docs/zh-CN/next/reference-cli-tools#pulsar-daemon</span>) CLI tool:
+
+<span class="hljs-code">```bash
+$ bin/pulsar-daemon start bookie
+</span></code></pre>
<p>To start the bookie in the foreground:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper bookie
</code></pre>
-<p>You can verify that the bookie is working properly using the <code>bookiesanity</code> command for the <a href="http://localhost:4000/docs/latest/deployment/reference/CliTools#bookkeeper-shell">BookKeeper shell</a>:</p>
+<p>You can verify that a bookie is working properly by running the <code>bookiesanity</code> command for the <a href="/docs/zh-CN/next/reference-cli-tools#shell">BookKeeper shell</a> on it:</p>
<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell bookiesanity
</code></pre>
<p>This will create an ephemeral BookKeeper ledger on the local bookie, write a few entries, read them back, and finally delete the ledger.</p>
+<p>After you have started all the bookies, you can use <code>simpletest</code> command for <a href="/docs/zh-CN/next/reference-cli-tools#shell">BookKeeper shell</a> on any bookie node, to verify all the bookies in the cluster are up running.</p>
+<pre><code class="hljs css languages- bash">$ bin/bookkeeper shell simpletest --ensemble <num-bookies> --writeQuorum <num-bookies> --ackQuorum <num-bookies> --numEntries <num-entries>
+</code></pre>
+<p>This command will create a <code>num-bookies</code> sized ledger on the cluster, write a few entries, and finally delete the ledger.</p>
<h2><a class="anchor" aria-hidden="true" id="deploying-pulsar-brokers"></a><a href="#deploying-pulsar-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-. [...]
<p>Pulsar brokers are the last thing you need to deploy in your Pulsar cluster. Brokers handle Pulsar messages and provide Pulsar's administrative interface. We recommend running <strong>3 brokers</strong>, one for each machine that's already running a BookKeeper bookie.</p>
+<h3><a class="anchor" aria-hidden="true" id="configuring-brokers"></a><a href="#configuring-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.2 [...]
<p>The most important element of broker configuration is ensuring that that each broker is aware of the ZooKeeper cluster that you've deployed. Make sure that the <a href="/docs/zh-CN/next/reference-configuration#broker-zookeeperServers"><code>zookeeperServers</code></a> and <a href="/docs/zh-CN/next/reference-configuration#broker-configurationStoreServers"><code>configurationStoreServers</code></a> parameters. In this case, since we only have 1 cluster and no configuration store setup, [...]
<pre><code class="hljs css languages- properties">zookeeperServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><sp [...]
configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk2<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</span><span class="hljs-selector-class">.com</span>:<span class="hljs-number">2181</span>,zk3<span class="hljs-selector-class">.us-west</span><span class="hljs-selector-class">.example</ [...]
@@ -182,6 +239,19 @@ configurationStoreServers=zk1<span class="hljs-selector-class">.us-west</span><s
<p>You also need to specify the cluster name (matching the name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>:</p>
<pre><code class="hljs css languages- properties"><span class="hljs-attr">clusterName</span>=pulsar-cluster-<span class="hljs-number">1</span>
</code></pre>
+<h3><a class="anchor" aria-hidden="true" id="enabling-pulsar-functions-optional"></a><a href="#enabling-pulsar-functions-optional" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5 [...]
+<p>If you want to enable <a href="/docs/zh-CN/next/functions-overview">Pulsar Functions</a>, you can follow the instructions as below:</p>
+<ol>
+<li><p>Edit <code>conf/broker.conf</code> to enable function worker, by setting <code>functionsWorkerEnabled</code> to <code>true</code>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">functionsWorkerEnabled</span>=<span class="hljs-literal">true</span>
+</code></pre></li>
+</ol>
+<ol start="2">
+<li><p>Edit <code>conf/functions_worker.yml</code> and set <code>pulsarFunctionsCluster</code> to the cluster name that you provided when <a href="#initializing-cluster-metadata">initializing the cluster's metadata</a>.</p>
+<pre><code class="hljs css languages- conf"><span class="hljs-attr">pulsarFunctionsCluster</span>=pulsar-cluster-<span class="hljs-number">1</span>
+</code></pre></li>
+</ol>
+<h3><a class="anchor" aria-hidden="true" id="starting-brokers"></a><a href="#starting-brokers" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2. [...]
<p>You can then provide any other configuration changes that you'd like in the <a href="/docs/zh-CN/next/reference-configuration#broker"><code>conf/broker.conf</code></a> file. Once you've decided on a configuration, you can start up the brokers for your Pulsar cluster. Like ZooKeeper and BookKeeper, brokers can be started either in the foreground or in the background, using nohup.</p>
<p>You can start a broker in the foreground using the <a href="/docs/zh-CN/next/reference-cli-tools#pulsar-broker"><code>pulsar broker</code></a> command:</p>
<pre><code class="hljs css languages- bash">$ bin/pulsar broker
@@ -206,7 +276,27 @@ brokerServiceurl=<span class="hljs-string">pulsar:</span><span class="hljs-comme
<p>You may need to use a different cluster name in the topic if you specified a cluster name different from <code>pulsar-cluster-1</code>.</p>
</blockquote>
<p>This will publish a single message to the Pulsar topic.</p>
-</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#installing-the-pulsar-binary-package">Installing the Pulsar binary package</a></li><li><a href="#deploying-a-zookeeper-cl [...]
+<h2><a class="anchor" aria-hidden="true" id="running-functions"></a><a href="#running-functions" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2- [...]
+<blockquote>
+<p>If you have <a href="#enabling-pulsar-functions-optional">enabled</a> Pulsar Functions, you can also tryout pulsar functions now.</p>
+</blockquote>
+<p>Create a ExclamationFunction <code>exclamation</code>.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> create \
+ --jar examples/api-examples.jar \
+ --className org.apache.pulsar.functions.api.examples.ExclamationFunction \
+ --inputs persistent://public/default/exclamation-input \
+ --output persistent://public/default/exclamation-output \
+ --tenant public \
+ --namespace default \
+ --name exclamation
+</code></pre>
+<p>Check if the function is running as expected by <a href="/docs/zh-CN/next/functions-deploying#triggering-pulsar-functions">triggering</a> the function.</p>
+<pre><code class="hljs css languages- bash">bin/pulsar-admin <span class="hljs-built_in">functions</span> trigger --name exclamation --triggerValue <span class="hljs-string">"hello world"</span>
+</code></pre>
+<p>You will see output as below:</p>
+<pre><code class="hljs css languages- shell">hello world!
+</code></pre>
+</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/zh-CN/next/deploy-kubernetes">← Deploying Pulsar on Kubernetes</a><a class="docs-next button" href="/docs/zh-CN/next/deploy-bare-metal-multi-cluster">Deploying a multi-cluster on bare metal →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#preparation">Preparation</a><ul class="toc-headings"><li><a href="#requirements">Requirements</a></li><li><a href="#hardw [...]
const community = document.querySelector("a[href='#community']").parentNode;
const communityMenu =
'<li>' +
diff --git a/content/ja/reference/RestApi/index.html b/content/ja/reference/RestApi/index.html
index 61d8013..ef81cb5 100644
--- a/content/ja/reference/RestApi/index.html
+++ b/content/ja/reference/RestApi/index.html
@@ -8497,6 +8497,11 @@
<td>404</td>
<td>Topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -8651,6 +8656,11 @@
<td>409</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -8808,6 +8818,11 @@
<td>404</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11278,6 +11293,11 @@
<td>404</td>
<td>Topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11364,6 +11384,16 @@
<td>403</td>
<td>Don't have admin permission</td>
</tr>
+
+ <tr>
+ <td>409</td>
+ <td>Partitioned topic does not exist</td>
+ </tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11432,6 +11462,11 @@
<td>409</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11507,6 +11542,11 @@
<td>409</td>
<td>Partitioned topic already exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
@@ -11589,6 +11629,11 @@
<td>404</td>
<td>Partitioned topic does not exist</td>
</tr>
+
+ <tr>
+ <td>412</td>
+ <td>Partitioned topic name is invalid</td>
+ </tr>
<!-- for param in parameters -->
</tbody>
</table>
diff --git a/content/swagger/swagger.json b/content/swagger/swagger.json
index 3b94d2d..b31d997 100644
--- a/content/swagger/swagger.json
+++ b/content/swagger/swagger.json
@@ -3441,6 +3441,9 @@
},
"404" : {
"description" : "Topic does not exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
}
@@ -3514,6 +3517,9 @@
},
"409" : {
"description" : "Partitioned topic does not exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
},
@@ -3594,6 +3600,9 @@
},
"404" : {
"description" : "Partitioned topic does not exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
}
@@ -4801,6 +4810,9 @@
},
"404" : {
"description" : "Topic does not exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
}
@@ -4843,6 +4855,12 @@
},
"403" : {
"description" : "Don't have admin permission"
+ },
+ "409" : {
+ "description" : "Partitioned topic does not exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
},
@@ -4874,6 +4892,9 @@
},
"409" : {
"description" : "Partitioned topic does not exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
},
@@ -4911,6 +4932,9 @@
},
"409" : {
"description" : "Partitioned topic already exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
},
@@ -4954,6 +4978,9 @@
},
"404" : {
"description" : "Partitioned topic does not exist"
+ },
+ "412" : {
+ "description" : "Partitioned topic name is invalid"
}
}
}
@@ -6660,9 +6687,21 @@
"type" : "number",
"format" : "double"
},
+ "overLoaded" : {
+ "type" : "boolean"
+ },
+ "loadReportType" : {
+ "type" : "string"
+ },
"cpu" : {
"$ref" : "#/definitions/ResourceUsage"
},
+ "bandwidthOut" : {
+ "$ref" : "#/definitions/ResourceUsage"
+ },
+ "memory" : {
+ "$ref" : "#/definitions/ResourceUsage"
+ },
"directMemory" : {
"$ref" : "#/definitions/ResourceUsage"
},
@@ -6670,31 +6709,19 @@
"type" : "integer",
"format" : "int64"
},
- "msgThroughputIn" : {
+ "msgThroughputOut" : {
"type" : "number",
"format" : "double"
},
- "msgThroughputOut" : {
+ "msgThroughputIn" : {
"type" : "number",
"format" : "double"
},
- "bandwidthIn" : {
- "$ref" : "#/definitions/ResourceUsage"
- },
- "bandwidthOut" : {
- "$ref" : "#/definitions/ResourceUsage"
- },
- "memory" : {
- "$ref" : "#/definitions/ResourceUsage"
- },
- "overLoaded" : {
- "type" : "boolean"
- },
- "loadReportType" : {
- "type" : "string"
- },
"underLoaded" : {
"type" : "boolean"
+ },
+ "bandwidthIn" : {
+ "$ref" : "#/definitions/ResourceUsage"
}
}
},
@@ -6854,10 +6881,10 @@
"type" : "number",
"format" : "double"
},
- "connectedSince" : {
+ "producerName" : {
"type" : "string"
},
- "producerName" : {
+ "connectedSince" : {
"type" : "string"
},
"clientVersion" : {
@@ -7558,10 +7585,10 @@
"type" : "string"
}
},
- "connectedSince" : {
+ "producerName" : {
"type" : "string"
},
- "producerName" : {
+ "connectedSince" : {
"type" : "string"
},
"clientVersion" : {