You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by "Pervatoju, Lakshman (CCI-Atlanta)" <la...@cox.com> on 2020/01/17 18:02:44 UTC

Help with architecture

Team,
We have done our POC on DRUID cluster and are pretty much impressed by its performance.
We had deployed it from Horton works stack and so are limited to use 0.10 version, but we want to take full advantage of the latest version 0.16

We are planning to deploy a Apache Druid cluster with the latest available version, but need some guidance.
We have 10 nodes with below configurations:

Cores (CPU): 80
Disk: 3984.7GB
Memory: 503.36GB

We also want to install ZK on 3 servers.
The below is how we are planning to distribute, please check and suggest any changes.

S. No.
Components
1
Co-ord, overlord, zk1, broker1
2
Co-ord, overlord, zk2, broker2
3
zk3, Superset UI, Grafana , router (optional), mySQL server
4
MM, Historical
5
MM, Historical
6
MM, Historical
7
MM, Historical
8
MM, Historical
9
MM, Historical
10
MM, Historical

We want to use Mysql for both druid and superset UI.
Also, as per https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html

We are collocating historical and Middle manager so will split nodes resources by 2
Implies each of it will get 40 cores and 250 GB Memory.

HISTORICAL
Heap : 0.5GB * number of CPU cores = 0.5 * 40 = 20 GB < 24 GB, so we will have heap set to 20 GB
Considering total size of all loaded lookups will make it maximum of 24 GB.

druid.processing.numThreads = 40 -1 = 39
druid.processing.buffer.sizeBytes= 500 MB
druid.processing.numMergeBuffers (a 1:4 ratio of merge buffers to processing threads) implies 9
Direct Memory Sizing
direct memory usage= (druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes
= 39+9+1=49*500 MB ~ 24 GB
These are few memory settings which we are planning, can you please review our plan to distribute druid and supporting services and also the memory settings and suggest any changes.
Please let me know if you need anything else.

Thanks & Regards

Lakshman Pervatoju
Enterprise Data Services
O: 404.269.2224 | C: 205.515.7866
CTECH A05-166D
6305 Peachtree Dunwoody Rd. Atlanta, GA 30328<webextlink://6305%20Peachtree%20Dunwoody%20Rd.%20Atlanta,%20GA%2030328>

[cid:image001.png@01D480D8.ED44B020]

Re: Help with architecture

Posted by "Pervatoju, Lakshman (CCI-Atlanta)" <la...@cox.com>.
Team,
Can you please guide us or route us to someone who can help us with this.

Please let me know if you need anything else.

Thanks & Regards

Lakshman Pervatoju
Enterprise Data Services
O: 404.269.2224 | C: 205.515.7866
CTECH A05-166D
6305 Peachtree Dunwoody Rd. Atlanta, GA 30328<webextlink://6305%20Peachtree%20Dunwoody%20Rd.%20Atlanta,%20GA%2030328>

[cid:image001.png@01D480D8.ED44B020]

From: "Pervatoju, Lakshman (CCI-Atlanta)" <la...@cox.com>
Date: Friday, January 17, 2020 at 1:02 PM
To: "dev@druid.apache.org" <de...@druid.apache.org>
Subject: Help with architecture

Team,
We have done our POC on DRUID cluster and are pretty much impressed by its performance.
We had deployed it from Horton works stack and so are limited to use 0.10 version, but we want to take full advantage of the latest version 0.16

We are planning to deploy a Apache Druid cluster with the latest available version, but need some guidance.
We have 10 nodes with below configurations:

Cores (CPU): 80
Disk: 3984.7GB
Memory: 503.36GB

We also want to install ZK on 3 servers.
The below is how we are planning to distribute, please check and suggest any changes.

S. No.
Components
1
Co-ord, overlord, zk1, broker1
2
Co-ord, overlord, zk2, broker2
3
zk3, Superset UI, Grafana , router (optional), mySQL server
4
MM, Historical
5
MM, Historical
6
MM, Historical
7
MM, Historical
8
MM, Historical
9
MM, Historical
10
MM, Historical

We want to use Mysql for both druid and superset UI.
Also, as per https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html

We are collocating historical and Middle manager so will split nodes resources by 2
Implies each of it will get 40 cores and 250 GB Memory.

HISTORICAL
Heap : 0.5GB * number of CPU cores = 0.5 * 40 = 20 GB < 24 GB, so we will have heap set to 20 GB
Considering total size of all loaded lookups will make it maximum of 24 GB.

druid.processing.numThreads = 40 -1 = 39
druid.processing.buffer.sizeBytes= 500 MB
druid.processing.numMergeBuffers (a 1:4 ratio of merge buffers to processing threads) implies 9
Direct Memory Sizing
direct memory usage= (druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes
= 39+9+1=49*500 MB ~ 24 GB
These are few memory settings which we are planning, can you please review our plan to distribute druid and supporting services and also the memory settings and suggest any changes.
Please let me know if you need anything else.

Thanks & Regards

Lakshman Pervatoju
Enterprise Data Services
O: 404.269.2224 | C: 205.515.7866
CTECH A05-166D
6305 Peachtree Dunwoody Rd. Atlanta, GA 30328<webextlink://6305%20Peachtree%20Dunwoody%20Rd.%20Atlanta,%20GA%2030328>

[cid:image001.png@01D480D8.ED44B020]