You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Simin Eftekhari <kh...@gmail.com> on 2019/10/20 17:02:16 UTC
Performance Question
Hello Everyone,
I am new to apache ignite. I'd appreciate your help with the following
question.
I modified the CacheApiExample, to insert 2,000,000 entries into the cache.
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheApiExample.java
If I run the program as is, the insert all the entries takes less 10
seconds (which I think is still too long). If I start an additional ignite
node (on the same vm), the inserts take over 3 minutes. This is a huge
discrepancy. I'd have thought that an additional instance would actually
speed up the inserts.
Can someone please explain what is going?
thanks
SK
Re: Performance Question
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I think that you should use DataStreamer to populate data. Should remove
most of this discrepancy.
Regards,
--
Ilya Kasnacheev
вс, 20 окт. 2019 г. в 20:02, Simin Eftekhari <kh...@gmail.com>:
> Hello Everyone,
>
> I am new to apache ignite. I'd appreciate your help with the following
> question.
>
> I modified the CacheApiExample, to insert 2,000,000 entries into the
> cache.
>
>
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheApiExample.java
>
> If I run the program as is, the insert all the entries takes less 10
> seconds (which I think is still too long). If I start an additional ignite
> node (on the same vm), the inserts take over 3 minutes. This is a huge
> discrepancy. I'd have thought that an additional instance would actually
> speed up the inserts.
>
> Can someone please explain what is going?
>
> thanks
>
> SK
>
Re: Performance Question
Posted by "kimec.ethome.sk" <ki...@ethome.sk>.
Hi,
I would suggest you to deploy ignite server nodes on separate machines.
If you are using virtualized infrastructure, I would suggest you to
create two separate VMs. Ideally each VM should be running on a
different hypervisor with enough headroom so that both VMs don't compete
for resources (CPUs) at the same time.
In addition to that, you should probably have the client application
running on yet another dedicated machine.
That being said, I believe proper performance testing is possible on
physical hardware only.
Also, are the 2,000,000 entries you use in your test unique? How do you
generate them?
Kind regards,
Kamil Mišúth
On 2019-10-20 20:48, Simin Eftekhari wrote:
> Hello Micheal,
> Thanks for the reply.
>
> I did not make any changes to the configuration (example-ignite.xml),
> in either scenario. There is no backup.
>
> https://github.com/apache/ignite/blob/master/examples/config/example-ignite.xml
>
> Also, in both cases, the ignite nodes are running on the same vm,
> meaning, there should not be any network hops.
> I am executing the same program, in one case as is, in another case,
> as is with an additional ignite node started in the background, using
> this script:
>
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ExampleNodeStartup.java
>
> I am just doing some basic benchmarking using the existing ignite
> examples.
>
> SK
>
> On Sun, Oct 20, 2019 at 1:31 PM Mikael <mi...@telia.com>
> wrote:
>
>> Hi!
>>
>> Well, it depends on what you are doing I guess ;)
>>
>> The example use put(), this is the slowest way of putting anything
>> in
>> the cache compared to streamers for example, anyway, if you run your
>>
>> code on one node in a big loop that does put 2M times it will be
>> slower,
>> with two nodes it will have to send half of the entries to the other
>>
>> node, so 1M entries are being sent over the network to the other
>> node,
>> with one node you don't need to send anything over the network.
>>
>> You say 10 sec is slow, not sure what you compare with, too me that
>> sounds like a good time, Ignite will by default put your objects off
>>
>> heap so the objects have to be serialized and moved off heap and
>> there
>> is a lot of things going on behind the scenes that takes time in
>> case
>> you compare with a HashMap or something.
>>
>> Ignite likes to do things in parallel, having one thread doing all
>> the
>> puts is the worst way to put a lot of entries in a cache, try to
>> create
>> a streamer and run your example with two nodes on that and see what
>> you
>> get, I am not sure what you intend to use it for ? you can feed
>> Ignite a
>> huge amount of entries per second but trying to get all of them
>> through
>> one single thread on one node is not going to do that.
>>
>> How did you configure the cache ? number of backups ? persistence ?
>> partitioned/replicated ?
>>
>> Mikael
>>
>> Den 2019-10-20 kl. 19:02, skrev Simin Eftekhari:
>>> Hello Everyone,
>>>
>>> I am new to apache ignite. I'd appreciate your help with the
>> following
>>> question.
>>>
>>> I modified the CacheApiExample, to insert 2,000,000 entries into
>> the
>>> cache.
>>>
>>>
>>
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheApiExample.java
>>>
>>> If I run the program as is, the insert all the entries takes less
>> 10
>>> seconds (which I think is still too long). If I start an
>> additional
>>> ignite node (on the same vm), the inserts take over 3 minutes.
>> This is
>>> a huge discrepancy. I'd have thought that an additional instance
>> would
>>> actually speed up the inserts.
>>>
>>> Can someone please explain what is going?
>>>
>>> thanks
>>>
>>> SK
Re: Performance Question
Posted by Simin Eftekhari <kh...@gmail.com>.
Hello Micheal,
Thanks for the reply.
I did not make any changes to the configuration (example-ignite.xml), in
either scenario. There is no backup.
https://github.com/apache/ignite/blob/master/examples/config/example-ignite.xml
Also, in both cases, the ignite nodes are running on the same vm, meaning,
there should not be any network hops.
I am executing the same program, in one case as is, in another case, as is
with an additional ignite node started in the background, using this script:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ExampleNodeStartup.java
I am just doing some basic benchmarking using the existing ignite examples.
SK
On Sun, Oct 20, 2019 at 1:31 PM Mikael <mi...@telia.com> wrote:
> Hi!
>
> Well, it depends on what you are doing I guess ;)
>
> The example use put(), this is the slowest way of putting anything in
> the cache compared to streamers for example, anyway, if you run your
> code on one node in a big loop that does put 2M times it will be slower,
> with two nodes it will have to send half of the entries to the other
> node, so 1M entries are being sent over the network to the other node,
> with one node you don't need to send anything over the network.
>
> You say 10 sec is slow, not sure what you compare with, too me that
> sounds like a good time, Ignite will by default put your objects off
> heap so the objects have to be serialized and moved off heap and there
> is a lot of things going on behind the scenes that takes time in case
> you compare with a HashMap or something.
>
> Ignite likes to do things in parallel, having one thread doing all the
> puts is the worst way to put a lot of entries in a cache, try to create
> a streamer and run your example with two nodes on that and see what you
> get, I am not sure what you intend to use it for ? you can feed Ignite a
> huge amount of entries per second but trying to get all of them through
> one single thread on one node is not going to do that.
>
> How did you configure the cache ? number of backups ? persistence ?
> partitioned/replicated ?
>
> Mikael
>
>
> Den 2019-10-20 kl. 19:02, skrev Simin Eftekhari:
> > Hello Everyone,
> >
> > I am new to apache ignite. I'd appreciate your help with the following
> > question.
> >
> > I modified the CacheApiExample, to insert 2,000,000 entries into the
> > cache.
> >
> >
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheApiExample.java
> >
> > If I run the program as is, the insert all the entries takes less 10
> > seconds (which I think is still too long). If I start an additional
> > ignite node (on the same vm), the inserts take over 3 minutes. This is
> > a huge discrepancy. I'd have thought that an additional instance would
> > actually speed up the inserts.
> >
> > Can someone please explain what is going?
> >
> > thanks
> >
> > SK
>
Re: Performance Question
Posted by Mikael <mi...@telia.com>.
Hi!
Well, it depends on what you are doing I guess ;)
The example use put(), this is the slowest way of putting anything in
the cache compared to streamers for example, anyway, if you run your
code on one node in a big loop that does put 2M times it will be slower,
with two nodes it will have to send half of the entries to the other
node, so 1M entries are being sent over the network to the other node,
with one node you don't need to send anything over the network.
You say 10 sec is slow, not sure what you compare with, too me that
sounds like a good time, Ignite will by default put your objects off
heap so the objects have to be serialized and moved off heap and there
is a lot of things going on behind the scenes that takes time in case
you compare with a HashMap or something.
Ignite likes to do things in parallel, having one thread doing all the
puts is the worst way to put a lot of entries in a cache, try to create
a streamer and run your example with two nodes on that and see what you
get, I am not sure what you intend to use it for ? you can feed Ignite a
huge amount of entries per second but trying to get all of them through
one single thread on one node is not going to do that.
How did you configure the cache ? number of backups ? persistence ?
partitioned/replicated ?
Mikael
Den 2019-10-20 kl. 19:02, skrev Simin Eftekhari:
> Hello Everyone,
>
> I am new to apache ignite. I'd appreciate your help with the following
> question.
>
> I modified the CacheApiExample, to insert 2,000,000 entries into the
> cache.
>
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheApiExample.java
>
> If I run the program as is, the insert all the entries takes less 10
> seconds (which I think is still too long). If I start an additional
> ignite node (on the same vm), the inserts take over 3 minutes. This is
> a huge discrepancy. I'd have thought that an additional instance would
> actually speed up the inserts.
>
> Can someone please explain what is going?
>
> thanks
>
> SK