You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Pim D <pi...@ns.nl> on 2017/10/04 17:03:59 UTC
SqlQuery results missing
Hi,
I have a partitioned cache distributed on 5 nodes without replication nor
persistence.
This cache contains 5000 items (id: [passage-0...4999]) of type:
public class SectiePassage implements Serializable {
@QuerySqlField(index = true)
public String id;
@QuerySqlField
public String sectieId;
@QuerySqlField(index = true)
public String passantId;
/* afgeleidde, hoeft niet te worden bewaard */
private transient AffinityKey<String> key;
public SectiePassage(String id, String sectieId, String passantId) {
this.id = id;
this.sectieId = sectieId;
this.passantId = passantId;
}
public SectiePassage() {
}
public AffinityKey<String> key() {
if (key == null)
key = new AffinityKey<>(id, sectieId);
return key;
}
public String toString() {
return this.getClass().getSimpleName() + "[" + id + "](" + key() +
")";
}
}
This entity is registered with my cache configuration when my client
(mode=true) connects with the cluster using:
setIndexedTypes(String.class,SectiePassage.class);
When the client is connected and executes a SqlQuery:
SqlQuery sql = new SqlQuery(SectiePassage.class,"id=?");
List<String> error = new ArrayList<>();
int found = 0;
for(int i =0 ;i<5000;i++) {
if (passageCache.query(sql.setArgs("passage-" +
i)).getAll().size()==0)
error.add("passage-" + i);
else
found++;
}
System.out.println(error.size() + " not found");
System.out.println(found + " found");
System.out.println(passageCache.size(CachePeekMode.ALL) + " total");
The results would be:
4033 not found
967 found
5000 total
A) I expected 5000 found...
B) Also I expected SqlQuery to perform better as a map-reduce looping
through each and every item in the cache on each node locally (both take
approx. equal amount of time)
It seems SqlQuery only considers the data on the node my client is directly
connected with (thus query is not mapped to the other nodes).
What am I doing wrong here and is it normal the SqlQuery performance is not
faster than a simple map-reduce?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SqlQuery results missing
Posted by Pim D <pi...@ns.nl>.
Ok, I found the problem.
CacheConfiguration.setIndexTypes(a1,a2,...) requires pairs of type key.class
and value.class.
I used String.class for key while I should have been using
AffinityKey.class.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SqlQuery results missing
Posted by Pim D <pi...@ns.nl>.
Hi,
No, not executed locally (would not work either because of client mode).
Here the output generated during the test against 5 nodes in docker
containers having 2 caches where I'm only searching through
'sectie-passage':
[17:40:14] Quiet mode.
[17:40:14] ^-- To see **FULL** console log here add
-DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[17:40:14]
[17:40:14] OS: Windows 10 10.0 amd64
[17:40:14] VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13
[17:40:14] Initial heap size is 128MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[17:40:14] Configured plugins:
[17:40:14] ^-- None
[17:40:14]
[17:40:23] Message queue limit is set to 0 which may lead to potential
OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due
to message queues growth on sender and receiver sides.
[17:40:24] Security status [authentication=off, tls/ssl=off]
[17:40:24] REST protocols do not start on client node. To start the
protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
property.
[17:40:55] Performance suggestions for grid 'ignite-test-client' (fix if
possible)
[17:40:55] To disable, set
-DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[17:40:55] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[17:40:55] ^-- Specify JVM heap max size (add
'-Xmx<size>[g|G|m|M|k|K]' to JVM options)
[17:40:55] ^-- Set max direct memory size if getting 'OOME: Direct
buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM
options)
[17:40:55] ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[17:40:55] ^-- Disable assertions (remove '-ea' from JVM options)
[17:40:55] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[17:40:55]
[17:40:55] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[17:40:55]
[17:40:55] Ignite node started OK (id=18d876dd, instance
name=ignite-test-client)
[17:40:55] Topology snapshot [ver=8, servers=5, clients=1, CPUs=10,
heap=3.8GB]
nl.donna.ignite.workshop.sql.SqlDataGridTest > ordered03_mapDataGridTest
STANDARD_OUT
*** node:sqlNode2 ***
cache:passant-sectie
passant-1[A], items:3
passant-11[A], items:4
passant-2[A], items:3
passant-8[A], items:4
passant-14[A], items:3
passant-12[A], items:3
total:20
cache:sectie-passage
sectie-1[A], items:345
sectie-2[A], items:288
sectie-6[A], items:320
total:953
*** node:sqlNode1 ***
cache:passant-sectie
passant-0[A], items:3
passant-4[A], items:3
passant-13[A], items:3
total:9
cache:sectie-passage
sectie-5[A], items:336
sectie-7[A], items:306
total:642
*** node:sqlNode4 ***
cache:passant-sectie
passant-5[A], items:3
passant-3[A], items:3
passant-7[A], items:3
total:9
cache:sectie-passage
sectie-4[A], items:314
sectie-8[A], items:311
sectie-10[A], items:355
total:980
*** node:sqlNode3 ***
cache:passant-sectie
passant-6[A], items:3
total:3
cache:sectie-passage
sectie-14[A], items:338
sectie-13[A], items:332
sectie-0[A], items:332
sectie-3[A], items:335
sectie-12[A], items:355
total:1692
*** node:sqlNode5 ***
cache:passant-sectie
passant-10[A], items:3
passant-9[A], items:3
total:6
cache:sectie-passage
sectie-11[A], items:363
sectie-9[A], items:370
total:733
total:5047
nl.donna.ignite.workshop.sql.SqlDataGridTest >
ordered04_zoekPassantSectiePassages_AlleNodes_Test STANDARD_OUT
4 items gevonden op node 'sqlNode2'
9 items gevonden op node 'sqlNode4'
0 items gevonden op node 'sqlNode3'
283 items gevonden op node 'sqlNode1'
0 items gevonden op node 'sqlNode5'
*296 *gevonden passages van 'passant-7' met ComputeTask (Map-Reduce)
(zoektijd: ca. 417 msec):
nl.donna.ignite.workshop.sql.SqlDataGridTest >
ordered05_zoekPassantSectiePassages_AffinityNodes_Test STANDARD_OUT
3 secties hebben affiniteit met 'passant-7'
sqlNode2: [sectie-6]
sqlNode4: [sectie-8]
sqlNode1: [sectie-7]
*296* gevonden passages van 'passant-7' met Callables en Affinity Keys
(zoektijd: ca. 377 msec):
nl.donna.ignite.workshop.sql.SqlDataGridTest >
ordered06_zoekPassantSectiePassages_SqlQuery_Test STANDARD_OUT
*53 *gevonden passages van 'passant-7' met SQL
(zoektijd: ca. 357 msec)
As you can see from this output the test searches the sectie-passage cache
for all SectiePassage where passantId equals passant-7. Each test verifies
that the actual results match the expectation, thus a SqlQuery (see below)
should find 296 results.
I get 53 results which doesn't even match with a amount off matching items
on a single node.
The query:
SqlQuery sql = new SqlQuery(SectiePassage.class,"passantId=?");
sql.setLocal(false); // default
List<Cache.Entry<String, SectiePassage>> result =
passageCache.query(sql.setArgs(passant)).getAll();
The client configuration:
setClientMode(true);
setPeerClassLoadingEnabled(false); // aanbevolen voor productie
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setLocalPort(47100);
spi.setLocalPortRange(10);
TcpDiscoveryMulticastIpFinder mc = new
TcpDiscoveryMulticastIpFinder();
mc.setAddresses(servers);
mc.setResponseWaitTime(1000); // speedup discovery
spi.setIpFinder(mc);
The logging on the nodes (and stdout) does not show any issues, could this
be a timeout related issue?
Something else?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SqlQuery results missing
Posted by Alexey Kukushkin <ku...@gmail.com>.
Hi Pim,
Can you make sure:
- Your SqlQuery is NOT local (sql.isLocal() must be false)
- You did not restart any of your 5 server nodes during your exercises.
Remember, if you restarted a node after loading data to the cluster, the
data on that node is gone and you no longer have 5000 items in the cache.
Re: SqlQuery results missing
Posted by Pim D <pi...@ns.nl>.
The data is generated during start-up of the test.
Data has a non expiration policy
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SqlQuery results missing
Posted by Pim D <pi...@ns.nl>.
Hi,
Yes, in fact I also have a map task to search each cache on each node.
That task does find all items.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SqlQuery results missing
Posted by Alexey Kukushkin <ku...@gmail.com>.
Pim,
Can you confirm you really have 5000 items loaded to the cache (using visor
or Web Console). Can you also share how you load the data.
Normally your queries must find all 5000 items as you expected.