Ignite cache read performance optimization

[JayX <xi...@gmail.com>]

Hello!

I have been working on Ignite and comparing its performance to traditional
databases. However, the performance was not significantly better as
expected. Any advice on how to improve the performance would be greatly
appreciated!

The dataset I have is around 10M records, stored on a cluster with 4
machines. I'm mainly concerned with cache read performance as the service is
expected to handle read requests frequently. 

I have been using Ignite SqlQuery which gives the best performance. The way
I implement it is to generate a sql string from request parameters, create a
SqlQuery instance with that string, use cache.query() to get a QueryCursor
instance, iterate through the QueryCursor and obtain results. Everything is
done following the Ignite API.

The most commonly used sql string here is a simple string of conditions like
"fieldA = a AND fieldB = b ...". I have also added individual indexes for
each of the fields that are commonly queried on. The execution plan I
observed in H2 debug console showed that the indexes were being used.

I have pretty much tried out everything in this link: 
https://apacheignite.readme.io/docs/performance-tips
<https://apacheignite.readme.io/docs/performance-tips>  , but none of them
seem to help significantly. 

I also tried to fire multiple queries on different partitions of the cache
in parallel to increase IO throughput. However, I think this could
potentially make other requests to the same cache starve.

In addition, using ScanQuery instead of SqlQuery made execution several
times slower...

I would really hope to seek help on if I made any mistakes or if there are
any suggestions on what I could try out and see if it results in better
performance? Thank you very much!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite cache read performance optimization

[Denis Magda <dm...@apache.org>]

Hello,

Please share the actual queries and execution plans for each. Use "EXPLAIN
SELECT" in Ignite instead (not H2 version).

--
Denis

On Wed, Jan 2, 2019 at 10:58 AM JayX <xi...@gmail.com> wrote:

> Hello!
>
> I have been working on Ignite and comparing its performance to traditional
> databases. However, the performance was not significantly better as
> expected. Any advice on how to improve the performance would be greatly
> appreciated!
>
> The dataset I have is around 10M records, stored on a cluster with 4
> machines. I'm mainly concerned with cache read performance as the service
> is
> expected to handle read requests frequently.
>
> I have been using Ignite SqlQuery which gives the best performance. The way
> I implement it is to generate a sql string from request parameters, create
> a
> SqlQuery instance with that string, use cache.query() to get a QueryCursor
> instance, iterate through the QueryCursor and obtain results. Everything is
> done following the Ignite API.
>
> The most commonly used sql string here is a simple string of conditions
> like
> "fieldA = a AND fieldB = b ...". I have also added individual indexes for
> each of the fields that are commonly queried on. The execution plan I
> observed in H2 debug console showed that the indexes were being used.
>
> I have pretty much tried out everything in this link:
> https://apacheignite.readme.io/docs/performance-tips
> <https://apacheignite.readme.io/docs/performance-tips>  , but none of them
> seem to help significantly.
>
> I also tried to fire multiple queries on different partitions of the cache
> in parallel to increase IO throughput. However, I think this could
> potentially make other requests to the same cache starve.
>
> In addition, using ScanQuery instead of SqlQuery made execution several
> times slower...
>
> I would really hope to seek help on if I made any mistakes or if there are
> any suggestions on what I could try out and see if it results in better
> performance? Thank you very much!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>