You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/08 04:10:55 UTC
[GitHub] [pulsar] Anonymitaet opened a new pull request #13187: [Doc] add hardware recommendations
Anonymitaet opened a new pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187
Fix https://github.com/apache/pulsar/issues/13078
Preview looks good:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#discussion_r765431053
##########
File path: site2/docs/deploy-bare-metal.md
##########
@@ -63,30 +63,27 @@ For machines running a bookie and a Pulsar broker, more powerful machines are re
* Fast CPUs and 10Gbps [NIC](https://en.wikipedia.org/wiki/Network_interface_controller) (for Pulsar brokers)
* Small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache (for BookKeeper bookies)
+#### Hardware recommendations
+
To start a Pulsar instance, below are the minimum and the recommended hardware settings.
-1. The minimum hardware settings (250 Pulsar topics)
- - Broker
- - CPU: 0.2
- - Memory: 256MB
- - Bookie
- - CPU: 0.2
- - Memory: 256MB
- - Storage:
- - Journal: 8GB, PD-SSD
- - Ledger: 16GB, PD-STANDARD
-
-2. The recommended hardware settings (1000 Pulsar topics)
-
- - Broker
- - CPU: 8
- - Memory: 8GB
- - Bookie
- - CPU: 4
- - Memory: 8GB
- - Storage:
- - Journal: 256GB, PD-SSD
- - Ledger: 2TB, PD-STANDARD
+A cluster consists of 3 broker nodes, 3 bookie nodes, and 3 ZooKeeper nodes. The following recommendation is suitable for one node.
+
+- The minimum hardware settings (**250 Pulsar topics**)
+
+ Component | CPU|Memory|Storage|Throughput |Rate
+ |---|---|---|---|---|---
+ Broker|0.2|256 MB|/|Write throughput: 3 MB/s<br><br />Read throughput: 6 MB/s<br><br />|Write rate: 350 entries/s<br><br />Read rate: 650 entries/s
Review comment:
It should be entries because of the producer side batching, more entries rate consumes more CPU resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#discussion_r764616820
##########
File path: site2/docs/deploy-bare-metal.md
##########
@@ -63,30 +63,27 @@ For machines running a bookie and a Pulsar broker, more powerful machines are re
* Fast CPUs and 10Gbps [NIC](https://en.wikipedia.org/wiki/Network_interface_controller) (for Pulsar brokers)
* Small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache (for BookKeeper bookies)
+#### Hardware recommendations
+
To start a Pulsar instance, below are the minimum and the recommended hardware settings.
-1. The minimum hardware settings (250 Pulsar topics)
- - Broker
- - CPU: 0.2
- - Memory: 256MB
- - Bookie
- - CPU: 0.2
- - Memory: 256MB
- - Storage:
- - Journal: 8GB, PD-SSD
- - Ledger: 16GB, PD-STANDARD
-
-2. The recommended hardware settings (1000 Pulsar topics)
-
- - Broker
- - CPU: 8
- - Memory: 8GB
- - Bookie
- - CPU: 4
- - Memory: 8GB
- - Storage:
- - Journal: 256GB, PD-SSD
- - Ledger: 2TB, PD-STANDARD
+A cluster consists of 3 broker nodes, 3 bookie nodes, and 3 ZooKeeper nodes. The following recommendation is suitable for one node.
Review comment:
@codelipenghui is this accurate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#discussion_r764616820
##########
File path: site2/docs/deploy-bare-metal.md
##########
@@ -63,30 +63,27 @@ For machines running a bookie and a Pulsar broker, more powerful machines are re
* Fast CPUs and 10Gbps [NIC](https://en.wikipedia.org/wiki/Network_interface_controller) (for Pulsar brokers)
* Small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache (for BookKeeper bookies)
+#### Hardware recommendations
+
To start a Pulsar instance, below are the minimum and the recommended hardware settings.
-1. The minimum hardware settings (250 Pulsar topics)
- - Broker
- - CPU: 0.2
- - Memory: 256MB
- - Bookie
- - CPU: 0.2
- - Memory: 256MB
- - Storage:
- - Journal: 8GB, PD-SSD
- - Ledger: 16GB, PD-STANDARD
-
-2. The recommended hardware settings (1000 Pulsar topics)
-
- - Broker
- - CPU: 8
- - Memory: 8GB
- - Bookie
- - CPU: 4
- - Memory: 8GB
- - Storage:
- - Journal: 256GB, PD-SSD
- - Ledger: 2TB, PD-STANDARD
+A cluster consists of 3 broker nodes, 3 bookie nodes, and 3 ZooKeeper nodes. The following recommendation is suitable for one node.
Review comment:
@codelipenghui is the line 70 accurate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#issuecomment-988482119
@codelipenghui @Huanli-Meng can you help review? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a change in pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on a change in pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#discussion_r765864023
##########
File path: site2/docs/deploy-bare-metal.md
##########
@@ -63,30 +63,27 @@ For machines running a bookie and a Pulsar broker, more powerful machines are re
* Fast CPUs and 10Gbps [NIC](https://en.wikipedia.org/wiki/Network_interface_controller) (for Pulsar brokers)
* Small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache (for BookKeeper bookies)
+#### Hardware recommendations
+
To start a Pulsar instance, below are the minimum and the recommended hardware settings.
-1. The minimum hardware settings (250 Pulsar topics)
- - Broker
- - CPU: 0.2
- - Memory: 256MB
- - Bookie
- - CPU: 0.2
- - Memory: 256MB
- - Storage:
- - Journal: 8GB, PD-SSD
- - Ledger: 16GB, PD-STANDARD
-
-2. The recommended hardware settings (1000 Pulsar topics)
-
- - Broker
- - CPU: 8
- - Memory: 8GB
- - Bookie
- - CPU: 4
- - Memory: 8GB
- - Storage:
- - Journal: 256GB, PD-SSD
- - Ledger: 2TB, PD-STANDARD
+A cluster consists of 3 broker nodes, 3 bookie nodes, and 3 ZooKeeper nodes. The following recommendation is suitable for one node.
+
+- The minimum hardware settings (**250 Pulsar topics**)
+
+ Component | CPU|Memory|Storage|Throughput |Rate
+ |---|---|---|---|---|---
+ Broker|0.2|256 MB|/|Write throughput: 3 MB/s<br><br />Read throughput: 6 MB/s<br><br />|Write rate: 350 entries/s<br><br />Read rate: 650 entries/s
Review comment:
Okay, I thought we always used the term messages, even if it meant we were describing batches of messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a change in pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on a change in pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#discussion_r764540817
##########
File path: site2/docs/deploy-bare-metal.md
##########
@@ -63,30 +63,27 @@ For machines running a bookie and a Pulsar broker, more powerful machines are re
* Fast CPUs and 10Gbps [NIC](https://en.wikipedia.org/wiki/Network_interface_controller) (for Pulsar brokers)
* Small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache (for BookKeeper bookies)
+#### Hardware recommendations
+
To start a Pulsar instance, below are the minimum and the recommended hardware settings.
-1. The minimum hardware settings (250 Pulsar topics)
- - Broker
- - CPU: 0.2
- - Memory: 256MB
- - Bookie
- - CPU: 0.2
- - Memory: 256MB
- - Storage:
- - Journal: 8GB, PD-SSD
- - Ledger: 16GB, PD-STANDARD
-
-2. The recommended hardware settings (1000 Pulsar topics)
-
- - Broker
- - CPU: 8
- - Memory: 8GB
- - Bookie
- - CPU: 4
- - Memory: 8GB
- - Storage:
- - Journal: 256GB, PD-SSD
- - Ledger: 2TB, PD-STANDARD
+A cluster consists of 3 broker nodes, 3 bookie nodes, and 3 ZooKeeper nodes. The following recommendation is suitable for one node.
+
+- The minimum hardware settings (**250 Pulsar topics**)
+
+ Component | CPU|Memory|Storage|Throughput |Rate
+ |---|---|---|---|---|---
+ Broker|0.2|256 MB|/|Write throughput: 3 MB/s<br><br />Read throughput: 6 MB/s<br><br />|Write rate: 350 entries/s<br><br />Read rate: 650 entries/s
Review comment:
Nit:
```suggestion
Broker|0.2|256 MB|/|Write throughput: 3 MB/s<br><br />Read throughput: 6 MB/s<br><br />|Write rate: 350 messages/s<br><br />Read rate: 650 messages/s
```
##########
File path: site2/docs/deploy-bare-metal.md
##########
@@ -63,30 +63,27 @@ For machines running a bookie and a Pulsar broker, more powerful machines are re
* Fast CPUs and 10Gbps [NIC](https://en.wikipedia.org/wiki/Network_interface_controller) (for Pulsar brokers)
* Small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache (for BookKeeper bookies)
+#### Hardware recommendations
+
To start a Pulsar instance, below are the minimum and the recommended hardware settings.
-1. The minimum hardware settings (250 Pulsar topics)
- - Broker
- - CPU: 0.2
- - Memory: 256MB
- - Bookie
- - CPU: 0.2
- - Memory: 256MB
- - Storage:
- - Journal: 8GB, PD-SSD
- - Ledger: 16GB, PD-STANDARD
-
-2. The recommended hardware settings (1000 Pulsar topics)
-
- - Broker
- - CPU: 8
- - Memory: 8GB
- - Bookie
- - CPU: 4
- - Memory: 8GB
- - Storage:
- - Journal: 256GB, PD-SSD
- - Ledger: 2TB, PD-STANDARD
+A cluster consists of 3 broker nodes, 3 bookie nodes, and 3 ZooKeeper nodes. The following recommendation is suitable for one node.
+
+- The minimum hardware settings (**250 Pulsar topics**)
+
+ Component | CPU|Memory|Storage|Throughput |Rate
+ |---|---|---|---|---|---
+ Broker|0.2|256 MB|/|Write throughput: 3 MB/s<br><br />Read throughput: 6 MB/s<br><br />|Write rate: 350 entries/s<br><br />Read rate: 650 entries/s
+ Bookie|0.2|256 MB|Journal: 8 GB<br><br />PD-SSDLedger: 16 GB, PD-STANDARD|Write throughput: 2 MB/s<br><br />Read throughput: 2 MB/s<br><br />|Write rate: 200 entries/s<br><br />Read rate: 200 entries/s
+ ZooKeeper|0.05|256 MB|Log: 8 GB, PD-SSD<br><br />Data: 2 GB, PD-STANDARD|/|/
+
+- The recommended hardware settings (**1000 Pulsar topics**)
+
+ Component | CPU|Memory|Storage|Throughput |Rate
+ |---|---|---|---|---|---
+ Broker|8|8 GB|/|Write throughput: 100 MB/s<br><br />Read throughput: 200 MB/s<br><br />|Write rate: 10,000 entries/s<br><br />Read rate: 20,000 entries/s
Review comment:
```suggestion
Broker|8|8 GB|/|Write throughput: 100 MB/s<br><br />Read throughput: 200 MB/s<br><br />|Write rate: 10,000 messages/s<br><br />Read rate: 20,000 messages/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#issuecomment-988575424
> LGTM. @Anonymitaet - out of curiosity, how are you determining the expected throughput for clusters using this hardware? I ask because this kind of throughput may vary from version to version and based on the configurations of the cluster. While it's not necessary to publish this information, it would be interesting to how the throughput varies for each release of Pulsar.
@codelipenghui can you help answer this question? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet merged pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
Anonymitaet merged pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on pull request #13187: [Doc] add hardware recommendations
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on pull request #13187:
URL: https://github.com/apache/pulsar/pull/13187#issuecomment-989925795
> > LGTM. @Anonymitaet - out of curiosity, how are you determining the expected throughput for clusters using this hardware? I ask because this kind of throughput may vary from version to version and based on the configurations of the cluster. While it's not necessary to publish this information, it would be interesting to how the throughput varies for each release of Pulsar.
>
> @codelipenghui can you help answer this question? Thanks!
@codelipenghui - any chance you can answer my additional questsions? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org