Requesting access to slack

[Tarik Courdy <ta...@gmail.com>]

Hello -

I was wondering if I could receive an invite to the apache metron slack
organization?

I also wanted to ask a question about supported data stores.  One of the
supported data stores is elasticsearch.  Does this mean that all data is
stored directly in elasticsearch or is some other approach taken?  If all
of the data is stored in ES, I imagine that scaling costs could get out of
hand as the size of the data continues to grow.  Is this a valid concern
with metron or no?

Thank you for your time.

-Tarik

Re: Requesting access to slack

[Simon Elliston Ball <si...@simonellistonball.com>]

Hi Tarik,

You’re quite right, ES costs can get very high, which is why most Metron users store a short term amount of data in ES and use the HDFS store for longer term data access.

Most I know of keep 1 to 3 months in elastic, and use HDFS to store data for, in some cases, years. This is usually done by deleting older ES indexes via curator.

Some people also limit the fields stored in ES through templates, which is something we’ve talked about making even more efficient with field level filtering in Metron. That helps keep the hot layer storage costs (ES) down. 

Simon

> On 12 Sep 2018, at 20:08, Tarik Courdy <ta...@gmail.com> wrote:
> 
> Hello - 
> 
> I was wondering if I could receive an invite to the apache metron slack organization?
> 
> I also wanted to ask a question about supported data stores.  One of the supported data stores is elasticsearch.  Does this mean that all data is stored directly in elasticsearch or is some other approach taken?  If all of the data is stored in ES, I imagine that scaling costs could get out of hand as the size of the data continues to grow.  Is this a valid concern with metron or no?
> 
> Thank you for your time.
> 
> -Tarik