Ignite Performance

[shahidv <sh...@expeedsoftware.com>]

Hi,

We are looking for new database technologies because of some performance
issues in existing.
So we tried ignite with 1 node cluster (28gb ram), loaded 90 million records
to the cache. 
But we faced several issues like,
-when restarted node, It took 6 hrs to activate again, 
-queries are very slower, 
-out of memory issue when loading data. 
-huge size for data.

We are still learning stage to Ignite  and it may be some configuration
issues. 
We have currently large volume of data and processing heavy aggregations. 

Anyone please tell me Ignite will be a solution for performance and what is
the issue with my current setup. Following is my config. file,
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t2499/default-config.xml>  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Performance

[Anton Kurbanov <an...@gmail.com>]

Hi shahidv,

Could you post please the details for configurations of caches, ideas of
queries and activation logs?

It's not easy to blindly give some recommendations to follow as there is
usually no silver bullet for SQL queries execution in a distributed
environment as you have to work on each specific query to mitigate possible
bottlenecks: make them as local as possible using data collocation, use
some precalculated data if possible, look into join order (larger tables
should be joined to smaller ones), probably you might make some small
tables replicated -- there are a lot of things that are might be reasonable
to be considered, but details for SQL tables/queries are necessary for
this.

Here are couple of things I'd like to know besides previously asked ones:

-- What was server node doing while starting up? Was that storage pages
loading, index building, wal files rollover (have you removed by any chance
wal/archive folder)?
-- Do you have any kind of subqueries / group by statements in your queries?
-- What is the conditions for OOM? Are you able to load whole data set?
-- As for the data size, what is the cache entry structure and what is the
median/average size of the record? What is the estimated size of raw data
being inserted into Ignite? Do you have any custom indexes and if so, how
many and what columns are there?

вт, 9 июл. 2019 г. в 16:30, Andrey Dolmatov <it...@gmail.com>:

> Hi,
> *-when restarted node, It took 6 hrs to activate again, *
> If you activated cluster on startup, before data loading, then after
> restart, cluster is active.
>
> *-queries are very slower, *
> it depends. What SQL queries do you have? JOIN, AGG?
>
> *-out of memory issue when loading data. *
> Show your CacheConfiguration
>
> * -huge size for data.*
> If you enable QueryEnity for SQL support, Ignite consumes much more memory.
>
>
> *Best Regards, Andrey Dolmatov*
>
> вт, 9 июл. 2019 г. в 16:10, shahidv <sh...@expeedsoftware.com>:
>
>> Hi,
>>
>> We are looking for new database technologies because of some performance
>> issues in existing.
>> So we tried ignite with 1 node cluster (28gb ram), loaded 90 million
>> records
>> to the cache.
>> But we faced several issues like,
>> -when restarted node, It took 6 hrs to activate again,
>> -queries are very slower,
>> -out of memory issue when loading data.
>> -huge size for data.
>>
>> We are still learning stage to Ignite  and it may be some configuration
>> issues.
>> We have currently large volume of data and processing heavy aggregations.
>>
>> Anyone please tell me Ignite will be a solution for performance and what
>> is
>> the issue with my current setup. Following is my config. file,
>> default-config.xml
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/t2499/default-config.xml>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Ignite Performance

[Andrey Dolmatov <it...@gmail.com>]

Hi,
*-when restarted node, It took 6 hrs to activate again, *
If you activated cluster on startup, before data loading, then after
restart, cluster is active.

*-queries are very slower, *
it depends. What SQL queries do you have? JOIN, AGG?

*-out of memory issue when loading data. *
Show your CacheConfiguration

* -huge size for data.*
If you enable QueryEnity for SQL support, Ignite consumes much more memory.


*Best Regards, Andrey Dolmatov*

вт, 9 июл. 2019 г. в 16:10, shahidv <sh...@expeedsoftware.com>:

> Hi,
>
> We are looking for new database technologies because of some performance
> issues in existing.
> So we tried ignite with 1 node cluster (28gb ram), loaded 90 million
> records
> to the cache.
> But we faced several issues like,
> -when restarted node, It took 6 hrs to activate again,
> -queries are very slower,
> -out of memory issue when loading data.
> -huge size for data.
>
> We are still learning stage to Ignite  and it may be some configuration
> issues.
> We have currently large volume of data and processing heavy aggregations.
>
> Anyone please tell me Ignite will be a solution for performance and what is
> the issue with my current setup. Following is my config. file,
> default-config.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2499/default-config.xml>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>