You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Tobias Eriksson <to...@qvantel.com> on 2017/11/12 20:01:49 UTC
Adding more ignite-clients slows down the first client
Hi
I try to achieve very high inserts with Ignite-Cassandra
I have done some extensive testing, measuring all kinds of stuff
So long story short, here is my conclusion, I see the following when measuring INSERT rate per client
As you can see the first ignite-client has ~19,000 INSERTs / second
BUT when I add one more client to try to increased (scale horizontally) I clearly see that the INSERT speed decreases on the first client by a magnitude (dropping about 7,500 inserts/second from it’s 19,000 !!!)
Although the total speed increases to around ~25,000 INSERTs / second, this is still not good, and does not scale very well.
And as you can see adding a 3;rd and a 4;th client has the same affect.
Each client added steels overall performance from the other clients
I believe that Ignite is unable to handle this much inserts, can this be true…
Is this a known behavior ?
Is there a remedy ? (please say it is)
Details
Amazon C4.4XLarge ==== 4 node cluster
Cassandra 3.3 ==== 4 Node cluster
Pojo Class with 50 attributes
Regards
Tobias
[cid:image001.png@01D35BF9.73D48C10]
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="cache_PojoProfile50"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="indexedTypes">
<list>
<value>java.lang.Long</value>
<value>com.qvantel.research.common.PojoProfile50</value>
</list>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="writeBehindEnabled" value="true"/>
<property name="writeBehindFlushFrequency" value="5000" />
<property name="writeBehindBatchSize" value="512" />
<property name="writeBehindFlushThreadCount" value="8" />
<property name="writeBehindFlushSize" value="0" />
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
<property name="dataSourceBean" value="cassandraDataSource"/>
<property name="persistenceSettingsBean" value="persistence_settings_PojoProfile50"/>
</bean>
</property>
</bean>
<persistence keyspace="ignite" table="pojoprofile50" ttl="86400">
<keyspaceOptions>
REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' :2}
AND DURABLE_WRITES = true
</keyspaceOptions>
<tableOption>
comment = 'Cache test'
AND read_repair_chance = 0.2
</tableOption>
<keyPersistence class="java.lang.Long" strategy="PRIMITIVE" column="profile_id"/>
<valuePersistence class="com.qvantel.research.common.PojoProfile50" strategy="POJO"/>
</persistence>
--
Tobias Eriksson
Chief Architect Research – CTO Office
Qvantel Sweden AB
Tel; +46 768 832453
e-mail; Tobias.Eriksson@qvantel.com<ma...@qvantel.com>
Re: Adding more ignite-clients slows down the first client
Posted by Denis Magda <dm...@apache.org>.
Tobias,
I would also recommend to turn off Cassandra for now and confirm that the bottleneck is not in Ignite.
—
Denis
> On Nov 13, 2017, at 7:48 AM, Mikhail <mi...@gmail.com> wrote:
>
> Hi Tobias,
>
> if you need to do an initial loading of a big amount of data, please read
> the following section:
> https://apacheignite.readme.io/docs/streaming--cep
>
>
>> BUT when I add one more client to try to increased (scale horizontally)
>> I clearly see that the INSERT speed decreases on the first client by a
> magnitude
>> (dropping about 7,500 inserts/second from it’s 19,000 !!!)
>
> you didn't scale it horizontally, you just added more load for one ignite
> server, if you need to scale it, you need
> to add more server nodes, you set backup==1, that means that you need to add
> at least 2 more nodes that see that it scales horizontally and linearly.
>
>> Although the total speed increases to around ~25,000 INSERTs / second, this
> is still not good, and does not scale very well.
>
> ignite scales pretty good, looks like *one* ignite server node reaches its
> limit, check cpu loading on the server node, I believe there would be 100%
> CPU utilization. So to scale it you need to add server nodes that will store
> data, because with you set up there's only 1 node that tries to handle
> traffic from 4 client nodes.
> Also, you might be reach network IO limit, anyway in this case you also need
> to add more server nodes to split network a traffic from 4 clients to
> several server nodes.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Adding more ignite-clients slows down the first client
Posted by Mikhail <mi...@gmail.com>.
Hi Tobias,
if you need to do an initial loading of a big amount of data, please read
the following section:
https://apacheignite.readme.io/docs/streaming--cep
>BUT when I add one more client to try to increased (scale horizontally)
>I clearly see that the INSERT speed decreases on the first client by a
magnitude
>(dropping about 7,500 inserts/second from it’s 19,000 !!!)
you didn't scale it horizontally, you just added more load for one ignite
server, if you need to scale it, you need
to add more server nodes, you set backup==1, that means that you need to add
at least 2 more nodes that see that it scales horizontally and linearly.
>Although the total speed increases to around ~25,000 INSERTs / second, this
is still not good, and does not scale very well.
ignite scales pretty good, looks like *one* ignite server node reaches its
limit, check cpu loading on the server node, I believe there would be 100%
CPU utilization. So to scale it you need to add server nodes that will store
data, because with you set up there's only 1 node that tries to handle
traffic from 4 client nodes.
Also, you might be reach network IO limit, anyway in this case you also need
to add more server nodes to split network a traffic from 4 clients to
several server nodes.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/