Ignite performance issues...

[Wolfgang Meyerle <wo...@googlemail.com>]

Hi,

I have a question in regards to Apache Ignite performance tuning.

This is my config so far:

https://pastebin.com/NWDzY3RK

I plan to store 2 billion entries in the database. The machine that I 
have is not great, 32gb ram nothing more.

I wonder how I can tune the performance to speed up things. I know that 
if I leave out indexing that would speed things are but indices are 
necessary. Also an SQL access is necessary though not necessarily during 
inserts. As I'm using C++ I don't know if ignite supports the feature in 
creating an SQL access later on after the data has been inserted in a 
cache. Is this possible?

According to my logging I'm able to insert

42108964 entries in 66 minutes

with 610 Insertions per second:610

and an average of 1057 insertions per second (which is not great in my 
opinion)


According to ignites performance guidelines I segregated already to Wal 
stores from the persistence storage drive.

I'm also getting Apache ignite errors from time to time:

[21:24:15] Possible failure suppressed accordingly to a configured 
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet 
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], 
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class 
o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, 
igniteInstanceName=null, finished=false, heartbeatTs=1605385441327]]]

Tried to find something online and in the Apache Ignite documentation 
but struggles so hopefully any geek out there can drop a comment...

Regards,


Wolfgang

Re: Ignite performance issues...

[Mikael <mi...@gmail.com>]

Hi!

How many nodes will you be running this on ?

What is your use case, what kind of performance is important to you, 
reading, writing ? do you need access to all data or will you just use a 
small subset ?

In terms of writing you can get good performance with a streamer instead 
of using SQL insert if that would be possible.

Mikael


On 2020-11-14 23:35, Wolfgang Meyerle wrote:
>
> Hi,
>
> I have a question in regards to Apache Ignite performance tuning.
>
> This is my config so far:
>
> https://pastebin.com/NWDzY3RK
>
> I plan to store 2 billion entries in the database. The machine that I 
> have is not great, 32gb ram nothing more.
>
> I wonder how I can tune the performance to speed up things. I know 
> that if I leave out indexing that would speed things are but indices 
> are necessary. Also an SQL access is necessary though not necessarily 
> during inserts. As I'm using C++ I don't know if ignite supports the 
> feature in creating an SQL access later on after the data has been 
> inserted in a cache. Is this possible?
>
> According to my logging I'm able to insert
>
> 42108964 entries in 66 minutes
>
> with 610 Insertions per second:610
>
> and an average of 1057 insertions per second (which is not great in my 
> opinion)
>
>
> According to ignites performance guidelines I segregated already to 
> Wal stores from the persistence storage drive.
>
> I'm also getting Apache ignite errors from time to time:
>
> [21:24:15] Possible failure suppressed accordingly to a configured 
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet 
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], 
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class 
> o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, 
> igniteInstanceName=null, finished=false, heartbeatTs=1605385441327]]]
>
> Tried to find something online and in the Apache Ignite documentation 
> but struggles so hopefully any geek out there can drop a comment...
>
> Regards,
>
>
> Wolfgang
>
>
>