You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/05/22 09:11:03 UTC
Slack digest for #general - 2018-05-22
2018-05-21 12:30:37 UTC - Tomer Lev: @Tomer Lev has joined the channel
----
2018-05-21 12:50:18 UTC - Tomer Lev: Hi, I noticed in documentation it recommends to deploy Pulsar brokers on i3.4xlarge instances - consider my throughput is very low (about 50 messages/sec) what is the recommended hardware requirements ? I'm trying to find something cheaper but to keep the latency low as guaranteed (<5 ms)
----
2018-05-21 14:07:29 UTC - Karthik Palanivelu: Team, any inputs on how we can build the zookeeper list and my_id update within say 3 docker containers? We may consider either ECS or K8s for this exercise. Just want to let you know that I am new to K8s. I reviewed the scripts in deployment/docker folder but looking for a simpler view into the dynamic config.
----
2018-05-21 16:22:47 UTC - Sijie Guo: In K8S, it is straightforward using stateful sets. Because statefulset provides incremental instance id, you can use as my id and you can have a service name as the zk connect string, k8s handles DNS resolution stuffs.
----
2018-05-21 16:41:23 UTC - Karthik Palanivelu: Cool, Let me try that option
----
2018-05-21 16:48:33 UTC - Matteo Merli: > consider my throughput is very low (about 50 messages/sec) what is the recommended hardware requirements ? I’m trying to find something cheaper but to keep the latency low as guaranteed (<5 ms)
@Tomer Lev In general to have low latency and durability (data fsynced before ack), one needs to rely either on HDDs with RAID controller durable write cache or SSDs (even then, different models of SSDs have very different performance profiles).
If strong durability is not absolute requirement, you can disable fsync and then any VM with either local disk or EBS volume would be working for you at that rate. Data will be written in page cache and replicated across multiple machines.
To disable fsync, set `journalSyncData=false` in `bookkeeper.conf`.
----
2018-05-22 06:47:02 UTC - Tomer Lev: >@Tomer Lev In general to have low latency and durability (data fsynced before ack), one needs to rely either on HDDs with RAID controller durable >>write cache or SSDs (even then, different models of SSDs have very different performance profiles).
>If strong durability is not absolute requirement, you can disable fsync and then any VM with either local disk or EBS volume would be working for you >at that rate. Data will be written in page cache and replicated across multiple machines.
>To disable fsync, set `journalSyncData=false` in `bookkeeper.conf`.
@Matteo Merli Thanks, anyway there is any minimum requirements for running Pulsar cluster ?
----
2018-05-22 06:55:20 UTC - Ali Ahmed: @Tomer Lev depends on your needs
----
2018-05-22 06:56:12 UTC - Ali Ahmed: A minimal production cluster should have three light weight nodes running zookeeper and 3 nodes for pulsar running brokers and bookies co located.
----
2018-05-22 06:56:34 UTC - Ali Ahmed: you can start with that and just add more nodes to scale
----
2018-05-22 06:56:45 UTC - Ali Ahmed: pulsar can scale with no downtime
----
2018-05-22 06:57:38 UTC - Tomer Lev: @Ali Ahmed OK thanks!
----
2018-05-22 06:58:29 UTC - Ali Ahmed: do you an idea of what your workload looks like ?
----
2018-05-22 07:00:22 UTC - Tomer Lev: Yes in the beginning very low - something like 50 events per second and should be scale to 10,000 in future
----
2018-05-22 07:01:47 UTC - Ali Ahmed: Understood a small cluster should be more then sufficient
----
2018-05-22 07:17:05 UTC - Tomer Lev: :+1:
----