Ignite performance - sudden drop after upgrade from 2.3.0

["Oleg Yurkivskyy (Luxoft)" <Ol...@tudor.com>]

Hi,

I have an application running on ignite cluster with 4 nodes.

/usr/bin/java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

The application runs on apache-ignite-2.3.0. I need to upgrade it to the latest version of ignite (currently 2.7.5)
I have created a benchmarking project (maven) to check what to expect after upgrade.
It is attached in archived file, called "data-grid-benchmarks".
This is a copy of standard ignite benchmarking project from 2.3.0 release with a custom cache benchmarks CacheBenchmarkBin and CacheBenchmarkExt.
Different classes use different serialization (Binary and Externalizable).
I ran it on 2.3.0 2.4.0 and 2.7.5 versions of apache ignite (3 servers and 1 driver) with the following command:

                ./bin/benchmark-run-all.sh config/benchmark-company.properties

It shows, that starting from 2.4.0 release there is a 2 times drop in performance of partitioned transactional cache.
Replicated cache is not affected in such a way.
Also performance doesn't depend on serialization.
In attached archive you can see results of benchmarks generated by standard yardstick tool for different version of ignite.
There you can see the drop of RELEASE-CacheBenchmarkBinPartitionTransactional-1-backup

Apache Ignite version

2.3.0

2.4.0

2.7.5

Average Test method Throughput (operations/sec)
CacheBenchmarkBinPartitionTransactional-1-backup

8,845.84

4,101.48

3,956.17

CacheBenchmarkBinReplicateTransactional-1-backup

2,204.89

2,197.45

2,124.80


I need to answer the question what can be fixed in the code for test (CacheBenchmarkBin.test or ActionCallable) to avoid the drop of performance?
Does it have something to do with the cache configuration (ignite-cache-partition-tx-config.xml)?
Also, it could be the bug in ignite, that needs to be fixed.

Regards,
Oleg


_________________________________________________________

This communication is intended only for the addressee(s) and may contain confidential information. We do not waive any confidentiality by misdelivery. If you receive this communication in error, any use, dissemination, printing or copying is strictly prohibited; please destroy all electronic and paper copies and notify the sender immediately.

Re: Ignite performance - sudden drop after upgrade from 2.3.0

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

Please collect profiling (e.g. using JFR) from both runs and search for hot
spots in both scenarios. If you find anything suspicious, please file a
ticket.

We can also look at profiling results for you.

Unfortunately I'm not that good with yardstick to run your benchmarks
locally. It would be nice if you could turn it into a reproducer test.

Regards,
-- 
Ilya Kasnacheev


ср, 11 сент. 2019 г. в 12:16, Oleg Yurkivskyy (Luxoft) <
Oleg.Yurkivskyy@tudor.com>:

> Hi,
>
>
>
> I have an application running on ignite cluster with 4 nodes.
>
>
>
> /usr/bin/java -version
>
> openjdk version "1.8.0_191"
>
> OpenJDK Runtime Environment (build 1.8.0_191-b12)
>
> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
>
>
>
> The application runs on apache-ignite-2.3.0. I need to upgrade it to the
> latest version of ignite (currently 2.7.5)
>
> I have created a benchmarking project (maven) to check what to expect
> after upgrade.
>
> It is attached in archived file, called "data-grid-benchmarks".
>
> This is a copy of standard ignite benchmarking project from 2.3.0 release
> with a custom cache benchmarks CacheBenchmarkBin and CacheBenchmarkExt.
>
> Different classes use different serialization (Binary and Externalizable).
>
> I ran it on 2.3.0 2.4.0 and 2.7.5 versions of apache ignite (3 servers and
> 1 driver) with the following command:
>
>
>
>                 ./bin/benchmark-run-all.sh
> config/benchmark-company.properties
>
>
>
> It shows, that starting from 2.4.0 release there is a 2 times drop in
> performance of partitioned transactional cache.
>
> Replicated cache is not affected in such a way.
>
> Also performance doesn't depend on serialization.
>
> In attached archive you can see results of benchmarks generated by
> standard yardstick tool for different version of ignite.
>
> There you can see the drop of
> RELEASE-CacheBenchmarkBinPartitionTransactional-1-backup
>
>
>
> Apache Ignite version
>
> 2.3.0
>
> 2.4.0
>
> 2.7.5
>
> Average Test method Throughput (operations/sec)
>
> CacheBenchmarkBinPartitionTransactional-1-backup
>
> 8,845.84
>
> 4,101.48
>
> 3,956.17
>
> CacheBenchmarkBinReplicateTransactional-1-backup
>
> 2,204.89
>
> 2,197.45
>
> 2,124.80
>
>
>
> I need to answer the question what can be fixed in the code for test
> (CacheBenchmarkBin.test or ActionCallable) to avoid the drop of performance?
>
> Does it have something to do with the cache configuration
> (ignite-cache-partition-tx-config.xml)?
>
> Also, it could be the bug in ignite, that needs to be fixed.
>
>
>
> Regards,
>
> Oleg
>
>
>
> _________________________________________________________
>
> This communication is intended only for the addressee(s) and may contain
> confidential information. We do not waive any confidentiality by
> misdelivery. If you receive this communication in error, any use,
> dissemination, printing or copying is strictly prohibited; please destroy
> all electronic and paper copies and notify the sender immediately.
>