Re: Distributed query is too slow.

[dkarachentsev <dk...@gridgain.com>]

Hi Vladimir,

When you run test on single node it works just locally, without node
communication overhead. Good point will be to test on client node. In that
case, even with one server node, query will be sent over network with all
that communication stuff. Also, if cache is REPLICATED, query always will be
invoked locally, or, if you query from client - sent to only one node.

I tried your example and don't see such big gap in performance with many
nodes, here is my results:
10_000 records (request one-by-one), 
1 node: 445ms
2 nodes: 512ms
3 nodes: 696ms
4 nodes: 839ms

Time to fetch increases, but not so fast, as you mentioned. The root of this
slowdown could be that I run all nodes on single machine (simultaneous
search on all nodes) and it requires a bit more messages.
For replicated cache this value will be constant.

BTW with broadcast compute on 3 nodes I got bigger result: 1489ms.

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-query-is-too-slow-tp10342p10380.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed query is too slow.

[dkarachentsev <dk...@gridgain.com>]

OK, now I see that locally we have equal results. As I understand, remote
means that nodes connected over network and placed on different machines.
Then 5 sec per 10K requests (0.5 ms per request), probably, caused  by
network delays. 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-query-is-too-slow-tp10342p10484.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed query is too slow.

["Vladimir St." <vl...@yandex.ru>]

dkarachentsev wrote
> I've just ran your tests, the project attached.
> Could you please, in turn, send your? This will help understand what goes
> wrong.

My test project is a jumble of many experiments and researches. And it needs
serious cleaning before publishing. I used yours. It works just like mine.
In the zip I attached you can find few shlight modifications but nothing
fundamental. 

The nodes I need are server node. You should run the test on each next node
once the previous one is done without termitating finished nodes.

So, the timings I got by your code are:
   - searching local 1-4 nodes: 184ms, 548ms, 635ms, 722ms
   - searching remote 1-4 nodes: 146ms, 4484ms, 6080ms, 5442ms

It's interesting the the 4th remote node has a bit accelerated the search. 

I case of local search we're charged with nearly x3 price to search over 2
nodes insted of 1. In case of remote search this extra cost grows up to x30.

If we launch from a client node, we might not see so huge difference between
the cluster of single node and of two nodes. But my overal goal is to get a
fast search platform against the DB. In my case, Ignite performs searches
over 2-4 remote nodes approximately two times faster than searching remote
Oracle with similar indexed table for the records. Although, it's a bit
frustrating because any kind of local searches shows 10-15 times higher
performance compared to the DB. But the result-gathering procedures break
all my anticipations. They make cache searching not so perfect as desired. I
still hope some setting are not applied somewhere.

ignite-perf-modified-test.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/n10422/ignite-perf-modified-test.zip>  



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-query-is-too-slow-tp10342p10422.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed query is too slow.

[dkarachentsev <dk...@gridgain.com>]

I've just ran your tests, the project attached.
Could you please, in turn, send your? This will help understand what goes
wrong.

Thanks!

ignite-perf-test.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/n10411/ignite-perf-test.zip>  

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-query-is-too-slow-tp10342p10411.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed query is too slow.

["Vladimir St." <vl...@yandex.ru>]

Hi Dmitry. Thanks for the answer.


dkarachentsev wrote
> When you run test on single node it works just locally, without node
> communication overhead. Good point will be to test on client node. In that
> case, even with one server node, query will be sent over network with all
> that communication stuff. Also, if cache is REPLICATED, query always will
> be invoked locally, or, if you query from client - sent to only one node.

If I'm correct, the search is always local. Any node searches only its local
data. In case of partitioned cache the results are afterwards merged within
shared transaction scope. That's why I've tried splitting and reducing query
results manually.


dkarachentsev wrote
> here is my results:
> 10_000 records (request one-by-one), 
> 1 node: 445ms
> 2 nodes: 512ms
> 3 nodes: 696ms
> 4 nodes: 839ms

Your timings are of some sort I expected but didn't get. Of course, there
will be some communication costs when we run on several nodes, especially in
network. But they souldn't be so huge as in my case.


Here is my results for 10 000 records on same machine:

1 node: 391ms
2 nodes:  1470ms
3 nodes: 1689ms
4 nodes: 2223ms

There is a very large gap between one and two nodes, i.e. when the
partitioning really appears.

Can the problem grow from Ignite settings, not cache settings?

How do you run your test? My scenario:
-	First node starts and acquires the cache.
-	Then it sees that the cache is empty and fills it with n records
-	After, the node prepares a list of known-params (fields) to search for
-	Runs through this list and executes the query
-	When first (previous) node is done, I launch next one
-	Next node does the same except filling the cache.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-query-is-too-slow-tp10342p10387.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.