You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Shubham Agrawal <ag...@gmail.com> on 2020/04/26 21:31:41 UTC
Insertions slow on high load on Ignite 2.7.5
Hi Team,
I am using ignite 2.7.5.
I am trying to insert 25 million records in the ignite persistent table.
Observations:
I am observing that initial 2.5 million records took 5 mins to insert, next
2.5 million records took 8 mins to insert, following 2.5 records took 12
mins to insert and it kept on increasing . The last 2.5 million records
took over 40 minutes to insert. The CPU of the client and server is around
30%, so nothing on that front. Heap usage looks normal. I have tried
inserting to the table with and without indexes but nothing seems to make a
difference.
Configurations:
I am running a 3 node cluster, 16 Cores 32 GB machines. Ignite Version:
2.7.5
Things I have tried:
JDBI Implementation -
1. @SQLBatch does not work for ignite giving the below error
org.skife.jdbi.v2.exceptions.TransactionException: Failed to start
transaction at
org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.begin(LocalTransactionHandler.java:57)
at org.skife.jdbi.v2.BasicHandle.begin(BasicHandle.java:159)
2. Tried ?streaming=true, but with no effect
3. Increasing threads to insert, but not much effect
JDBC Implementation
1. Tried ?streaming=true, but with no effect
2. Tried batching, but the performance actually degraded
3. Tried ?streaming=true with and without batching but with no effect.
Want to achieve a common pattern in insertions, like 5 mins or 8 mins
constantly for 25 million insertions
Please let me know your thoughts on the same. Your inputs could help me a
lot. Thanks a lot.
Regards,
Shubham Agrawal
Re: Insertions slow on high load on Ignite 2.7.5
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Your understanding is correct.
Yes, if you have more RAM and larger data region, the process should slow
down less rapidly.
Regards,
--
Ilya Kasnacheev
пн, 27 апр. 2020 г. в 17:56, Shubham Agrawal <ag...@gmail.com>:
> Hi Ilya,
>
> This data loading is not a one time activity for our scenario. So, I guess
> turning off the WAL would not be a solution since turning off and on every
> time is not feasible every time the data loading happens. Please let me
> know if my understanding is correct.
>
> I'll try to increase time between checkpoints and also increase checkpoint
> page buffer and let you know.
>
> In the meantime, just wanted to check if the Disk I/O and memory also has
> an effect? Currently, I am using 32 GB machine and increasing the memory
> will make any difference?
>
> Thanks & Regards,
> Shubham Agrawal
>
>
> On Mon, Apr 27, 2020 at 7:31 PM Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> My recommendation is to increase time between checkpoints
>> (checkpointFrequency) and, maybe, also increase checkpoint page buffer.
>>
>> If you do data loading in bursts, it may make sense to turn off WAL why
>> you do data loading.
>>
>> You may still expect some data degradation as time goes on (what's your
>> total allocated after the last 2.5 million?)
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 27 апр. 2020 г. в 00:32, Shubham Agrawal <
>> agrawalshubham1103@gmail.com>:
>>
>>> Hi Team,
>>>
>>> I am using ignite 2.7.5.
>>>
>>> I am trying to insert 25 million records in the ignite persistent table.
>>>
>>> Observations:
>>> I am observing that initial 2.5 million records took 5 mins to insert,
>>> next 2.5 million records took 8 mins to insert, following 2.5 records took
>>> 12 mins to insert and it kept on increasing . The last 2.5 million records
>>> took over 40 minutes to insert. The CPU of the client and server is around
>>> 30%, so nothing on that front. Heap usage looks normal. I have tried
>>> inserting to the table with and without indexes but nothing seems to make a
>>> difference.
>>>
>>> Configurations:
>>> I am running a 3 node cluster, 16 Cores 32 GB machines. Ignite Version:
>>> 2.7.5
>>>
>>> Things I have tried:
>>> JDBI Implementation -
>>> 1. @SQLBatch does not work for ignite giving the below error
>>> org.skife.jdbi.v2.exceptions.TransactionException: Failed to start
>>> transaction at
>>> org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.begin(LocalTransactionHandler.java:57)
>>> at org.skife.jdbi.v2.BasicHandle.begin(BasicHandle.java:159)
>>>
>>> 2. Tried ?streaming=true, but with no effect
>>>
>>> 3. Increasing threads to insert, but not much effect
>>>
>>> JDBC Implementation
>>> 1. Tried ?streaming=true, but with no effect
>>> 2. Tried batching, but the performance actually degraded
>>> 3. Tried ?streaming=true with and without batching but with no effect.
>>>
>>> Want to achieve a common pattern in insertions, like 5 mins or 8 mins
>>> constantly for 25 million insertions
>>>
>>> Please let me know your thoughts on the same. Your inputs could help me
>>> a lot. Thanks a lot.
>>>
>>> Regards,
>>> Shubham Agrawal
>>>
>>
Re: Insertions slow on high load on Ignite 2.7.5
Posted by Shubham Agrawal <ag...@gmail.com>.
Hi Ilya,
This data loading is not a one time activity for our scenario. So, I guess
turning off the WAL would not be a solution since turning off and on every
time is not feasible every time the data loading happens. Please let me
know if my understanding is correct.
I'll try to increase time between checkpoints and also increase checkpoint
page buffer and let you know.
In the meantime, just wanted to check if the Disk I/O and memory also has
an effect? Currently, I am using 32 GB machine and increasing the memory
will make any difference?
Thanks & Regards,
Shubham Agrawal
On Mon, Apr 27, 2020 at 7:31 PM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> My recommendation is to increase time between checkpoints
> (checkpointFrequency) and, maybe, also increase checkpoint page buffer.
>
> If you do data loading in bursts, it may make sense to turn off WAL why
> you do data loading.
>
> You may still expect some data degradation as time goes on (what's your
> total allocated after the last 2.5 million?)
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 27 апр. 2020 г. в 00:32, Shubham Agrawal <agrawalshubham1103@gmail.com
> >:
>
>> Hi Team,
>>
>> I am using ignite 2.7.5.
>>
>> I am trying to insert 25 million records in the ignite persistent table.
>>
>> Observations:
>> I am observing that initial 2.5 million records took 5 mins to insert,
>> next 2.5 million records took 8 mins to insert, following 2.5 records took
>> 12 mins to insert and it kept on increasing . The last 2.5 million records
>> took over 40 minutes to insert. The CPU of the client and server is around
>> 30%, so nothing on that front. Heap usage looks normal. I have tried
>> inserting to the table with and without indexes but nothing seems to make a
>> difference.
>>
>> Configurations:
>> I am running a 3 node cluster, 16 Cores 32 GB machines. Ignite Version:
>> 2.7.5
>>
>> Things I have tried:
>> JDBI Implementation -
>> 1. @SQLBatch does not work for ignite giving the below error
>> org.skife.jdbi.v2.exceptions.TransactionException: Failed to start
>> transaction at
>> org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.begin(LocalTransactionHandler.java:57)
>> at org.skife.jdbi.v2.BasicHandle.begin(BasicHandle.java:159)
>>
>> 2. Tried ?streaming=true, but with no effect
>>
>> 3. Increasing threads to insert, but not much effect
>>
>> JDBC Implementation
>> 1. Tried ?streaming=true, but with no effect
>> 2. Tried batching, but the performance actually degraded
>> 3. Tried ?streaming=true with and without batching but with no effect.
>>
>> Want to achieve a common pattern in insertions, like 5 mins or 8 mins
>> constantly for 25 million insertions
>>
>> Please let me know your thoughts on the same. Your inputs could help me a
>> lot. Thanks a lot.
>>
>> Regards,
>> Shubham Agrawal
>>
>
Re: Insertions slow on high load on Ignite 2.7.5
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
My recommendation is to increase time between checkpoints
(checkpointFrequency) and, maybe, also increase checkpoint page buffer.
If you do data loading in bursts, it may make sense to turn off WAL why you
do data loading.
You may still expect some data degradation as time goes on (what's your
total allocated after the last 2.5 million?)
Regards,
--
Ilya Kasnacheev
пн, 27 апр. 2020 г. в 00:32, Shubham Agrawal <ag...@gmail.com>:
> Hi Team,
>
> I am using ignite 2.7.5.
>
> I am trying to insert 25 million records in the ignite persistent table.
>
> Observations:
> I am observing that initial 2.5 million records took 5 mins to insert,
> next 2.5 million records took 8 mins to insert, following 2.5 records took
> 12 mins to insert and it kept on increasing . The last 2.5 million records
> took over 40 minutes to insert. The CPU of the client and server is around
> 30%, so nothing on that front. Heap usage looks normal. I have tried
> inserting to the table with and without indexes but nothing seems to make a
> difference.
>
> Configurations:
> I am running a 3 node cluster, 16 Cores 32 GB machines. Ignite Version:
> 2.7.5
>
> Things I have tried:
> JDBI Implementation -
> 1. @SQLBatch does not work for ignite giving the below error
> org.skife.jdbi.v2.exceptions.TransactionException: Failed to start
> transaction at
> org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.begin(LocalTransactionHandler.java:57)
> at org.skife.jdbi.v2.BasicHandle.begin(BasicHandle.java:159)
>
> 2. Tried ?streaming=true, but with no effect
>
> 3. Increasing threads to insert, but not much effect
>
> JDBC Implementation
> 1. Tried ?streaming=true, but with no effect
> 2. Tried batching, but the performance actually degraded
> 3. Tried ?streaming=true with and without batching but with no effect.
>
> Want to achieve a common pattern in insertions, like 5 mins or 8 mins
> constantly for 25 million insertions
>
> Please let me know your thoughts on the same. Your inputs could help me a
> lot. Thanks a lot.
>
> Regards,
> Shubham Agrawal
>