You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by ed d <ra...@hotmail.com> on 2017/10/17 19:14:16 UTC

Sizing of components proportional to EPS

Is there a rough guide to match EPS to an architectural sizing guide? I know its very difficult to extrapolate out, but a rough estimate would be nice. This may have already been attempted, and if yes, then please disregard.


Or can anyone share what they have found to work best?


For example,

POC - 1 machine
    1 big machine (16 CPU, 128 RAM, 5 Tb HDD)

100 EPS - 3 machines
    1 Nifi (8 CPU, 64 RAM, 5 Tb HDD)
    1 Hadoop/Metron (8 CPU, 64 RAM, 5 Tb HDD)
    1 Elasticsearch/Kibana (8 CPU, 64 RAM, 5 Tb HDD)

1000 EPS - 8 machines
    2 Nifi cliustered (8 CPU, 64 RAM, 5 Tb HDD)
    2 Hadoop (16 CPU, 128 RAM, 20 Tb HDD)
    1 Metron (16 CPU, 128 RAM, 1 Tb HDD)
    1 Elasticsearch data (8 CPU, 64 RAM, 20 Tb HDD)
    1 Elasticsearch master (8 CPU, 64 RAM, 1 Tb HDD)
    1 Kibana (8 CPU, 64 RAM, 1 Tb HDD)

10000 EPS - 14 machines
    4 Nifi clustered (16 CPU, 64 RAM, 5 Tb HDD)
    2 Hadoop (32 CPU, 128 RAM, 5 Tb HDD)
    2 Hadoop Data Nodes (32 CPU, 128 RAM, 40 Tb HDD)
    1 Metron (16 CPU, 128 RAM, 5 Tb HDD)
    1 Zeppelin (32 CPU, 128 RAM, 5 Tb HDD)
    2 ES data (32 CPU, 64 RAM, 40 Tb HDD)
    1 ES master (32 CPU, 64 RAM, 1 Tb HDD)
    1 Kibana (16 CPU, 64 RAM, 1 Tb HDD)

Re: Sizing of components proportional to EPS

Posted by Simon Elliston Ball <si...@simonellistonball.com>.
To an extent it very much depends on the use case. I have seen over a million EPS on a six node cluster for pcap and basic net flow. If you add a lot of complex enrichment and profiling that will obviously increase the load. Tuning the components for the workload can also make a significant difference. There are some good tips on that in the tuning guide in the source.

It would be great to hear some of the experiences other people on the list have had on eps and infrastructure for deployments. If anyone can post specs of a deployments, that would be fantastic to see.

Simon 


> On 17 Oct 2017, at 20:14, ed d <ra...@hotmail.com> wrote:
> 
> Is there a rough guide to match EPS to an architectural sizing guide? I know its very difficult to extrapolate out, but a rough estimate would be nice. This may have already been attempted, and if yes, then please disregard.
> 
> 
> Or can anyone share what they have found to work best?
> 
> 
> For example, 
> 
> POC - 1 machine
>     1 big machine (16 CPU, 128 RAM, 5 Tb HDD)
>     
> 100 EPS - 3 machines
>     1 Nifi (8 CPU, 64 RAM, 5 Tb HDD)
>     1 Hadoop/Metron (8 CPU, 64 RAM, 5 Tb HDD)
>     1 Elasticsearch/Kibana (8 CPU, 64 RAM, 5 Tb HDD)
> 
> 1000 EPS - 8 machines
>     2 Nifi cliustered (8 CPU, 64 RAM, 5 Tb HDD)
>     2 Hadoop (16 CPU, 128 RAM, 20 Tb HDD)
>     1 Metron (16 CPU, 128 RAM, 1 Tb HDD)
>     1 Elasticsearch data (8 CPU, 64 RAM, 20 Tb HDD)
>     1 Elasticsearch master (8 CPU, 64 RAM, 1 Tb HDD)
>     1 Kibana (8 CPU, 64 RAM, 1 Tb HDD)
> 
> 10000 EPS - 14 machines
>     4 Nifi clustered (16 CPU, 64 RAM, 5 Tb HDD)
>     2 Hadoop (32 CPU, 128 RAM, 5 Tb HDD)
>     2 Hadoop Data Nodes (32 CPU, 128 RAM, 40 Tb HDD)
>     1 Metron (16 CPU, 128 RAM, 5 Tb HDD)
>     1 Zeppelin (32 CPU, 128 RAM, 5 Tb HDD)
>     2 ES data (32 CPU, 64 RAM, 40 Tb HDD)
>     1 ES master (32 CPU, 64 RAM, 1 Tb HDD)
>     1 Kibana (16 CPU, 64 RAM, 1 Tb HDD)
>     
>     
>

Re: Sizing of components proportional to EPS

Posted by Simon Elliston Ball <si...@simonellistonball.com>.
To an extent it very much depends on the use case. I have seen over a million EPS on a six node cluster for pcap and basic net flow. If you add a lot of complex enrichment and profiling that will obviously increase the load. Tuning the components for the workload can also make a significant difference. There are some good tips on that in the tuning guide in the source.

It would be great to hear some of the experiences other people on the list have had on eps and infrastructure for deployments. If anyone can post specs of a deployments, that would be fantastic to see.

Simon 


> On 17 Oct 2017, at 20:14, ed d <ra...@hotmail.com> wrote:
> 
> Is there a rough guide to match EPS to an architectural sizing guide? I know its very difficult to extrapolate out, but a rough estimate would be nice. This may have already been attempted, and if yes, then please disregard.
> 
> 
> Or can anyone share what they have found to work best?
> 
> 
> For example, 
> 
> POC - 1 machine
>     1 big machine (16 CPU, 128 RAM, 5 Tb HDD)
>     
> 100 EPS - 3 machines
>     1 Nifi (8 CPU, 64 RAM, 5 Tb HDD)
>     1 Hadoop/Metron (8 CPU, 64 RAM, 5 Tb HDD)
>     1 Elasticsearch/Kibana (8 CPU, 64 RAM, 5 Tb HDD)
> 
> 1000 EPS - 8 machines
>     2 Nifi cliustered (8 CPU, 64 RAM, 5 Tb HDD)
>     2 Hadoop (16 CPU, 128 RAM, 20 Tb HDD)
>     1 Metron (16 CPU, 128 RAM, 1 Tb HDD)
>     1 Elasticsearch data (8 CPU, 64 RAM, 20 Tb HDD)
>     1 Elasticsearch master (8 CPU, 64 RAM, 1 Tb HDD)
>     1 Kibana (8 CPU, 64 RAM, 1 Tb HDD)
> 
> 10000 EPS - 14 machines
>     4 Nifi clustered (16 CPU, 64 RAM, 5 Tb HDD)
>     2 Hadoop (32 CPU, 128 RAM, 5 Tb HDD)
>     2 Hadoop Data Nodes (32 CPU, 128 RAM, 40 Tb HDD)
>     1 Metron (16 CPU, 128 RAM, 5 Tb HDD)
>     1 Zeppelin (32 CPU, 128 RAM, 5 Tb HDD)
>     2 ES data (32 CPU, 64 RAM, 40 Tb HDD)
>     1 ES master (32 CPU, 64 RAM, 1 Tb HDD)
>     1 Kibana (16 CPU, 64 RAM, 1 Tb HDD)
>     
>     
>