Disadvantages of having many cores

[Chellasamy G <ch...@zohocorp.com>]

Hi,



I am working on a log management tool and considering to use solr to index/search the logs.

I have few doubts about how to organize or create the cores.



The tool  should process 200 million events per day with each event containing 40 to 50 fields. Currently I have planned to create a core per day pushing all the data to the day's core. This may lead to the creation of many cores. Is this a good design? If not please suggest a good design.(Also, if multiple cores are used, will it slowdown the solr process' uptime)





Thanks,

Satyan

Re: Disadvantages of having many cores

[David Hastings <ha...@gmail.com>]

You're better off just using one core.  Perhaps think about pre-processing
the logs to "summarize" them into less "documents"
I do this and in my situation i summarize things like, user-hits-item, for
example.  so i find all the times a certain user had hits on a certain item
in one day and put that into one document.  I have about 4/5 years of http
logs and it sits at around 265 million documents and 17gb.  so hardly an
issue for performance

On Fri, Jul 28, 2017 at 10:04 AM, Chellasamy G <ch...@zohocorp.com>
wrote:

> Hi,
>
>
>
> I am working on a log management tool and considering to use solr to
> index/search the logs.
>
> I have few doubts about how to organize or create the cores.
>
>
>
> The tool  should process 200 million events per day with each event
> containing 40 to 50 fields. Currently I have planned to create a core per
> day pushing all the data to the day's core. This may lead to the creation
> of many cores. Is this a good design? If not please suggest a good
> design.(Also, if multiple cores are used, will it slowdown the solr
> process' uptime)
>
>
>
>
>
> Thanks,
>
> Satyan
>
>
>
>

Re: Disadvantages of having many cores

[Otis Gospodnetić <ot...@gmail.com>]

Hi,

Core per day is not too bad.  I assume you'll want to keep 7 days or maybe
30 or 60 or 180 days worth of logs.  That won't result in too many cores,
given adequate hardware.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Fri, Jul 28, 2017 at 9:04 AM, Chellasamy G <ch...@zohocorp.com>
wrote:

> Hi,
>
>
>
> I am working on a log management tool and considering to use solr to
> index/search the logs.
>
> I have few doubts about how to organize or create the cores.
>
>
>
> The tool  should process 200 million events per day with each event
> containing 40 to 50 fields. Currently I have planned to create a core per
> day pushing all the data to the day's core. This may lead to the creation
> of many cores. Is this a good design? If not please suggest a good
> design.(Also, if multiple cores are used, will it slowdown the solr
> process' uptime)
>
>
>
>
>
> Thanks,
>
> Satyan
>
>
>
>