You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Dominik Przybysz <al...@gmail.com> on 2020/03/04 07:40:57 UTC
Cache metrics on server nodes does not update correctly
Hi,
I am trying to use partitioned cache on server nodes to which I connect
with client node. Statistics of cache in the cluster are updated, but only
for hits metric - misses metric is always 0.
To reproduce this problem I created cluster of two nodes:
Server node 1 adds 100 random test cases and prints cache statistics
continuously:
public class IgniteClusterNode1 {
public static void main(String[] args) throws InterruptedException {
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
CacheConfiguration cacheConfiguration = new CacheConfiguration();
cacheConfiguration.setName("test");
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfiguration.setStatisticsEnabled(true);
igniteConfiguration.setCacheConfiguration(cacheConfiguration);
TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
communicationSpi.setLocalPort(47500);
igniteConfiguration.setCommunicationSpi(communicationSpi);
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
discoverySpi.setLocalPort(47100);
discoverySpi.setLocalPortRange(100);
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
"127.0.0.1:48100..48200"));
igniteConfiguration.setDiscoverySpi(discoverySpi);
try (Ignite ignite = Ignition.start(igniteConfiguration)) {
try (IgniteCache<String, String> cache =
ignite.getOrCreateCache("test")) {
new Random().ints(1000).map(i -> Math.abs(i %
1000)).distinct().limit(100).forEach(i -> {
String key = "data_" + i;
String value = UUID.randomUUID().toString();
cache.put(key, value);
}
);
}
while (true) {
System.out.println(ignite.cache("test").metrics());
Thread.sleep(5000);
}
}
}
}
Server node 2 only prints cache statistics continuously:
public class IgniteClusterNode2 {
public static void main(String[] args) throws InterruptedException {
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
CacheConfiguration cacheConfiguration = new CacheConfiguration();
cacheConfiguration.setName("test");
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setStatisticsEnabled(true);
igniteConfiguration.setCacheConfiguration(cacheConfiguration);
TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
communicationSpi.setLocalPort(48500);
igniteConfiguration.setCommunicationSpi(communicationSpi);
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
discoverySpi.setLocalPort(48100);
discoverySpi.setLocalPortRange(100);
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
"127.0.0.1:48100..48200"));
igniteConfiguration.setDiscoverySpi(discoverySpi);
try (Ignite ignite = Ignition.start(igniteConfiguration)) {
while (true) {
System.out.println(ignite.cache("test").metrics());
Thread.sleep(5000);
}
}
}
}
Next I start a client node which continuously read data from the cluster:
public class CacheClusterReader {
public static void main(String[] args) throws InterruptedException {
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setClientMode(true);
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
"127.0.0.1:48100..48200"));
spi.setIpFinder(tcMp);
cfg.setDiscoverySpi(spi);
CacheConfiguration<String, String> cacheConfig = new
CacheConfiguration<>("test");
cacheConfig.setStatisticsEnabled(true);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cfg.setCacheConfiguration(cacheConfig);
try (Ignite ignite = Ignition.start(cfg)) {
System.out.println(ignite.cacheNames());
while (true) {
try (IgniteCache<String, String> cache =
ignite.getOrCreateCache(cacheConfig)) {
cache.enableStatistics(true);
new Random().ints(1000).map(i -> Math.abs(i %
2000)).distinct().limit(100).forEach(i -> {
String key = "data_" + i;
String value = cache.get(key);
System.out.println("Found " + key + " with
value " + value);
}
);
System.out.println(cache.metrics());
System.out.println(cache.localMetrics());
}
Thread.sleep(3000);
}
}
}
}
On server and client nodes cluster statistics are synchronized, but misses
counter is always equal to 0, e. g.
CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
entryProcessorReadOnlyInvocations=0,
entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
entryProcessorMaxInvocationTime=0.0,
entryProcessorMinInvocationTime=0.0, *hits=93,
misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
isWriteBehindEnabled=false, writeBehindFlushSize=-1,
writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
In comparision, I also have also a thin client which connects to server and
read some random data once:
public class CacheClusterThinClientReader {
public static void main(String[] args) throws Exception {
ClientConfiguration clientConfiguration = new ClientConfiguration()
.setAddresses("127.0.0.1:10800");
try (IgniteClient ignite =
Ignition.startClient(clientConfiguration)) {
ClientCache<String, String> cache =
ignite.getOrCreateCache("test");
Set<String> misses = new HashSet<>();
new Random().ints(1000).map(i -> Math.abs(i %
2000)).distinct().limit(100).forEach(i -> {
String key = "data_" + i;
String value = cache.get(key);
System.out.println("Found " + key + " with value "
+ value);
if (value == null) {
misses.add(key);
}
}
);
System.out.println("Misses count: " + misses.size());
}
}
}
and finally prints (in my try):
Misses count: 97
On the server nodes misses were counted, but only for one server node:
CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
entryProcessorReadOnlyInvocations=0,
entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
entryProcessorMaxInvocationTime=0.0,
entryProcessorMinInvocationTime=0.0, *hits=3,
misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
isWriteBehindEnabled=false, writeBehindFlushSize=-1,
writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
Visor confirms my observation:
Nodes for: test(@c0)
+==================================================================================================================+
| Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time |
Size (Primary / Backup) | Hi/Mi/Rd/Wr |
+==================================================================================================================+
| D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291 |
Total: 47 (47 / 0) | Hi: 0 |
| | | | | |
Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | | |
Off-Heap: 47 (47 / 0) | Rd: 0 |
| | | | | |
Off-Heap Memory: <n/a> | Wr: 0 |
+----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
| B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122 |
Total: 53 (53 / 0) | Hi: 3 |
| | | | | |
Heap: 0 (0 / <n/a>) | Mi: 48 |
| | | | | |
Off-Heap: 53 (53 / 0) | Rd: 51 |
| | | | | |
Off-Heap Memory: <n/a> | Wr: 100 |
+------------------------------------------------------------------------------------------------------------------+
I have seen the same behaviour with rest api as with the thin client, but I
don't have prepared test case for this.
Is there a good way to correctly configure collecting statistics on server
nodes?
Should I create an issue for this problem?
--
Pozdrawiam / Regards,
Dominik Przybysz
Re: Cache metrics on server nodes does not update correctly
Posted by Andrey Gura <ag...@apache.org>.
Hi,
AI 2.7.6 doesn't contain a bug with aggregation of cache hits/misses.
I don't sure that described problem is related with IGNITE-3495 [1].
So it makes sense to file an issue.
[1] https://issues.apache.org/jira/browse/IGNITE-3495
On Thu, Mar 12, 2020 at 8:21 PM Dominik Przybysz <al...@gmail.com> wrote:
>
> Hi,
> I used ignite in version 2.7.6 (but I have also seen this behaviour on other 2.7.x versions) and there aren't any near or local cache.
> I expect that if I ask distributed cache about key which does not exist then the miss metric will be incremented.
>
>
> śr., 11 mar 2020 o 11:35 Andrey Gura <ag...@apache.org> napisał(a):
>>
>> Denis,
>>
>> I don't sure that I understand what is expected behavior should be.
>> There are local and aggregated cluster wide metrics. I don't know
>> which one used by Visor because I never used it :)
>>
>> Also it would be great to know what version of Apache Ignite used in
>> described case. I remember some bug with metrics aggregation during
>> discovery metrics message round trip.
>>
>> On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dm...@apache.org> wrote:
>> >
>> > @Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
>> > could you folks check out this thread?
>> >
>> > I have a feeling that what Dominik is describing was talked out before and
>> > rather some sort of a limitation than an issue with the current
>> > implementation.
>> >
>> > -
>> > Denis
>> >
>> >
>> > On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > > I am trying to use partitioned cache on server nodes to which I connect
>> > > with client node. Statistics of cache in the cluster are updated, but only
>> > > for hits metric - misses metric is always 0.
>> > >
>> > > To reproduce this problem I created cluster of two nodes:
>> > >
>> > > Server node 1 adds 100 random test cases and prints cache statistics
>> > > continuously:
>> > >
>> > > public class IgniteClusterNode1 {
>> > > public static void main(String[] args) throws InterruptedException {
>> > > IgniteConfiguration igniteConfiguration = new
>> > > IgniteConfiguration();
>> > >
>> > > CacheConfiguration cacheConfiguration = new CacheConfiguration();
>> > > cacheConfiguration.setName("test");
>> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
>> > > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>> > > cacheConfiguration.setStatisticsEnabled(true);
>> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>> > >
>> > > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
>> > > communicationSpi.setLocalPort(47500);
>> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
>> > >
>> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
>> > > discoverySpi.setLocalPort(47100);
>> > > discoverySpi.setLocalPortRange(100);
>> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
>> > > "127.0.0.1:48100..48200"));
>> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
>> > >
>> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
>> > > try (IgniteCache<String, String> cache =
>> > > ignite.getOrCreateCache("test")) {
>> > > new Random().ints(1000).map(i -> Math.abs(i %
>> > > 1000)).distinct().limit(100).forEach(i -> {
>> > > String key = "data_" + i;
>> > > String value = UUID.randomUUID().toString();
>> > > cache.put(key, value);
>> > > }
>> > > );
>> > > }
>> > > while (true) {
>> > > System.out.println(ignite.cache("test").metrics());
>> > > Thread.sleep(5000);
>> > > }
>> > > }
>> > > }
>> > > }
>> > >
>> > > Server node 2 only prints cache statistics continuously:
>> > >
>> > > public class IgniteClusterNode2 {
>> > > public static void main(String[] args) throws InterruptedException {
>> > > IgniteConfiguration igniteConfiguration = new
>> > > IgniteConfiguration();
>> > >
>> > > CacheConfiguration cacheConfiguration = new CacheConfiguration();
>> > > cacheConfiguration.setName("test");
>> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
>> > > cacheConfiguration.setStatisticsEnabled(true);
>> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>> > >
>> > > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
>> > > communicationSpi.setLocalPort(48500);
>> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
>> > >
>> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
>> > > discoverySpi.setLocalPort(48100);
>> > > discoverySpi.setLocalPortRange(100);
>> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
>> > > "127.0.0.1:48100..48200"));
>> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
>> > >
>> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
>> > > while (true) {
>> > > System.out.println(ignite.cache("test").metrics());
>> > > Thread.sleep(5000);
>> > > }
>> > > }
>> > > }
>> > > }
>> > >
>> > > Next I start a client node which continuously read data from the cluster:
>> > >
>> > > public class CacheClusterReader {
>> > > public static void main(String[] args) throws InterruptedException {
>> > > IgniteConfiguration cfg = new IgniteConfiguration();
>> > > cfg.setClientMode(true);
>> > >
>> > > TcpDiscoverySpi spi = new TcpDiscoverySpi();
>> > > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
>> > > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
>> > > "127.0.0.1:48100..48200"));
>> > > spi.setIpFinder(tcMp);
>> > > cfg.setDiscoverySpi(spi);
>> > >
>> > > CacheConfiguration<String, String> cacheConfig = new
>> > > CacheConfiguration<>("test");
>> > > cacheConfig.setStatisticsEnabled(true);
>> > > cacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> > > cfg.setCacheConfiguration(cacheConfig);
>> > >
>> > > try (Ignite ignite = Ignition.start(cfg)) {
>> > > System.out.println(ignite.cacheNames());
>> > >
>> > > while (true) {
>> > > try (IgniteCache<String, String> cache =
>> > > ignite.getOrCreateCache(cacheConfig)) {
>> > > cache.enableStatistics(true);
>> > >
>> > > new Random().ints(1000).map(i -> Math.abs(i %
>> > > 2000)).distinct().limit(100).forEach(i -> {
>> > > String key = "data_" + i;
>> > > String value = cache.get(key);
>> > > System.out.println("Found " + key + " with
>> > > value " + value);
>> > > }
>> > > );
>> > >
>> > > System.out.println(cache.metrics());
>> > > System.out.println(cache.localMetrics());
>> > > }
>> > > Thread.sleep(3000);
>> > > }
>> > > }
>> > > }
>> > > }
>> > >
>> > > On server and client nodes cluster statistics are synchronized, but misses
>> > > counter is always equal to 0, e. g.
>> > >
>> > > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
>> > > entryProcessorReadOnlyInvocations=0,
>> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
>> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
>> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
>> > > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=93,
>> > > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
>> > > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
>> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
>> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
>> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
>> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
>> > > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
>> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
>> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
>> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
>> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
>> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
>> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
>> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
>> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
>> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
>> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
>> > > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
>> > > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
>> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
>> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
>> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
>> > > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
>> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>> > >
>> > > In comparision, I also have also a thin client which connects to server
>> > > and read some random data once:
>> > >
>> > > public class CacheClusterThinClientReader {
>> > > public static void main(String[] args) throws Exception {
>> > > ClientConfiguration clientConfiguration = new ClientConfiguration()
>> > > .setAddresses("127.0.0.1:10800");
>> > >
>> > > try (IgniteClient ignite =
>> > > Ignition.startClient(clientConfiguration)) {
>> > >
>> > > ClientCache<String, String> cache =
>> > > ignite.getOrCreateCache("test");
>> > >
>> > > Set<String> misses = new HashSet<>();
>> > > new Random().ints(1000).map(i -> Math.abs(i %
>> > > 2000)).distinct().limit(100).forEach(i -> {
>> > > String key = "data_" + i;
>> > > String value = cache.get(key);
>> > > System.out.println("Found " + key + " with value "
>> > > + value);
>> > > if (value == null) {
>> > > misses.add(key);
>> > > }
>> > > }
>> > > );
>> > > System.out.println("Misses count: " + misses.size());
>> > > }
>> > > }
>> > > }
>> > >
>> > > and finally prints (in my try):
>> > >
>> > > Misses count: 97
>> > >
>> > > On the server nodes misses were counted, but only for one server node:
>> > >
>> > > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
>> > > entryProcessorReadOnlyInvocations=0,
>> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
>> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
>> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
>> > > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=3,
>> > > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
>> > > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
>> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
>> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
>> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
>> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
>> > > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
>> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
>> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
>> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
>> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
>> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
>> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
>> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
>> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
>> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
>> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
>> > > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
>> > > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
>> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
>> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
>> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
>> > > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
>> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>> > >
>> > > Visor confirms my observation:
>> > >
>> > > Nodes for: test(@c0)
>> > >
>> > > +==================================================================================================================+
>> > > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
>> > > | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
>> > >
>> > > +==================================================================================================================+
>> > > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291
>> > > | Total: 47 (47 / 0) | Hi: 0 |
>> > > | | | | |
>> > > | Heap: 0 (0 / <n/a>) | Mi: 0 |
>> > > | | | | |
>> > > | Off-Heap: 47 (47 / 0) | Rd: 0 |
>> > > | | | | |
>> > > | Off-Heap Memory: <n/a> | Wr: 0 |
>> > >
>> > > +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
>> > > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122
>> > > | Total: 53 (53 / 0) | Hi: 3 |
>> > > | | | | |
>> > > | Heap: 0 (0 / <n/a>) | Mi: 48 |
>> > > | | | | |
>> > > | Off-Heap: 53 (53 / 0) | Rd: 51 |
>> > > | | | | |
>> > > | Off-Heap Memory: <n/a> | Wr: 100 |
>> > >
>> > > +------------------------------------------------------------------------------------------------------------------+
>> > >
>> > > I have seen the same behaviour with rest api as with the thin client, but
>> > > I don't have prepared test case for this.
>> > >
>> > > Is there a good way to correctly configure collecting statistics on server
>> > > nodes?
>> > > Should I create an issue for this problem?
>> > >
>> > > --
>> > > Pozdrawiam / Regards,
>> > > Dominik Przybysz
>> > >
>
>
>
> --
> Pozdrawiam / Regards,
> Dominik Przybysz
Re: Cache metrics on server nodes does not update correctly
Posted by Andrey Gura <ag...@apache.org>.
Hi,
AI 2.7.6 doesn't contain a bug with aggregation of cache hits/misses.
I don't sure that described problem is related with IGNITE-3495 [1].
So it makes sense to file an issue.
[1] https://issues.apache.org/jira/browse/IGNITE-3495
On Thu, Mar 12, 2020 at 8:21 PM Dominik Przybysz <al...@gmail.com> wrote:
>
> Hi,
> I used ignite in version 2.7.6 (but I have also seen this behaviour on other 2.7.x versions) and there aren't any near or local cache.
> I expect that if I ask distributed cache about key which does not exist then the miss metric will be incremented.
>
>
> śr., 11 mar 2020 o 11:35 Andrey Gura <ag...@apache.org> napisał(a):
>>
>> Denis,
>>
>> I don't sure that I understand what is expected behavior should be.
>> There are local and aggregated cluster wide metrics. I don't know
>> which one used by Visor because I never used it :)
>>
>> Also it would be great to know what version of Apache Ignite used in
>> described case. I remember some bug with metrics aggregation during
>> discovery metrics message round trip.
>>
>> On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dm...@apache.org> wrote:
>> >
>> > @Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
>> > could you folks check out this thread?
>> >
>> > I have a feeling that what Dominik is describing was talked out before and
>> > rather some sort of a limitation than an issue with the current
>> > implementation.
>> >
>> > -
>> > Denis
>> >
>> >
>> > On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > > I am trying to use partitioned cache on server nodes to which I connect
>> > > with client node. Statistics of cache in the cluster are updated, but only
>> > > for hits metric - misses metric is always 0.
>> > >
>> > > To reproduce this problem I created cluster of two nodes:
>> > >
>> > > Server node 1 adds 100 random test cases and prints cache statistics
>> > > continuously:
>> > >
>> > > public class IgniteClusterNode1 {
>> > > public static void main(String[] args) throws InterruptedException {
>> > > IgniteConfiguration igniteConfiguration = new
>> > > IgniteConfiguration();
>> > >
>> > > CacheConfiguration cacheConfiguration = new CacheConfiguration();
>> > > cacheConfiguration.setName("test");
>> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
>> > > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>> > > cacheConfiguration.setStatisticsEnabled(true);
>> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>> > >
>> > > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
>> > > communicationSpi.setLocalPort(47500);
>> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
>> > >
>> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
>> > > discoverySpi.setLocalPort(47100);
>> > > discoverySpi.setLocalPortRange(100);
>> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
>> > > "127.0.0.1:48100..48200"));
>> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
>> > >
>> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
>> > > try (IgniteCache<String, String> cache =
>> > > ignite.getOrCreateCache("test")) {
>> > > new Random().ints(1000).map(i -> Math.abs(i %
>> > > 1000)).distinct().limit(100).forEach(i -> {
>> > > String key = "data_" + i;
>> > > String value = UUID.randomUUID().toString();
>> > > cache.put(key, value);
>> > > }
>> > > );
>> > > }
>> > > while (true) {
>> > > System.out.println(ignite.cache("test").metrics());
>> > > Thread.sleep(5000);
>> > > }
>> > > }
>> > > }
>> > > }
>> > >
>> > > Server node 2 only prints cache statistics continuously:
>> > >
>> > > public class IgniteClusterNode2 {
>> > > public static void main(String[] args) throws InterruptedException {
>> > > IgniteConfiguration igniteConfiguration = new
>> > > IgniteConfiguration();
>> > >
>> > > CacheConfiguration cacheConfiguration = new CacheConfiguration();
>> > > cacheConfiguration.setName("test");
>> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
>> > > cacheConfiguration.setStatisticsEnabled(true);
>> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>> > >
>> > > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
>> > > communicationSpi.setLocalPort(48500);
>> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
>> > >
>> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
>> > > discoverySpi.setLocalPort(48100);
>> > > discoverySpi.setLocalPortRange(100);
>> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
>> > > "127.0.0.1:48100..48200"));
>> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
>> > >
>> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
>> > > while (true) {
>> > > System.out.println(ignite.cache("test").metrics());
>> > > Thread.sleep(5000);
>> > > }
>> > > }
>> > > }
>> > > }
>> > >
>> > > Next I start a client node which continuously read data from the cluster:
>> > >
>> > > public class CacheClusterReader {
>> > > public static void main(String[] args) throws InterruptedException {
>> > > IgniteConfiguration cfg = new IgniteConfiguration();
>> > > cfg.setClientMode(true);
>> > >
>> > > TcpDiscoverySpi spi = new TcpDiscoverySpi();
>> > > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
>> > > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
>> > > "127.0.0.1:48100..48200"));
>> > > spi.setIpFinder(tcMp);
>> > > cfg.setDiscoverySpi(spi);
>> > >
>> > > CacheConfiguration<String, String> cacheConfig = new
>> > > CacheConfiguration<>("test");
>> > > cacheConfig.setStatisticsEnabled(true);
>> > > cacheConfig.setCacheMode(CacheMode.PARTITIONED);
>> > > cfg.setCacheConfiguration(cacheConfig);
>> > >
>> > > try (Ignite ignite = Ignition.start(cfg)) {
>> > > System.out.println(ignite.cacheNames());
>> > >
>> > > while (true) {
>> > > try (IgniteCache<String, String> cache =
>> > > ignite.getOrCreateCache(cacheConfig)) {
>> > > cache.enableStatistics(true);
>> > >
>> > > new Random().ints(1000).map(i -> Math.abs(i %
>> > > 2000)).distinct().limit(100).forEach(i -> {
>> > > String key = "data_" + i;
>> > > String value = cache.get(key);
>> > > System.out.println("Found " + key + " with
>> > > value " + value);
>> > > }
>> > > );
>> > >
>> > > System.out.println(cache.metrics());
>> > > System.out.println(cache.localMetrics());
>> > > }
>> > > Thread.sleep(3000);
>> > > }
>> > > }
>> > > }
>> > > }
>> > >
>> > > On server and client nodes cluster statistics are synchronized, but misses
>> > > counter is always equal to 0, e. g.
>> > >
>> > > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
>> > > entryProcessorReadOnlyInvocations=0,
>> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
>> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
>> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
>> > > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=93,
>> > > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
>> > > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
>> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
>> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
>> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
>> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
>> > > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
>> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
>> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
>> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
>> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
>> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
>> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
>> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
>> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
>> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
>> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
>> > > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
>> > > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
>> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
>> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
>> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
>> > > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
>> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>> > >
>> > > In comparision, I also have also a thin client which connects to server
>> > > and read some random data once:
>> > >
>> > > public class CacheClusterThinClientReader {
>> > > public static void main(String[] args) throws Exception {
>> > > ClientConfiguration clientConfiguration = new ClientConfiguration()
>> > > .setAddresses("127.0.0.1:10800");
>> > >
>> > > try (IgniteClient ignite =
>> > > Ignition.startClient(clientConfiguration)) {
>> > >
>> > > ClientCache<String, String> cache =
>> > > ignite.getOrCreateCache("test");
>> > >
>> > > Set<String> misses = new HashSet<>();
>> > > new Random().ints(1000).map(i -> Math.abs(i %
>> > > 2000)).distinct().limit(100).forEach(i -> {
>> > > String key = "data_" + i;
>> > > String value = cache.get(key);
>> > > System.out.println("Found " + key + " with value "
>> > > + value);
>> > > if (value == null) {
>> > > misses.add(key);
>> > > }
>> > > }
>> > > );
>> > > System.out.println("Misses count: " + misses.size());
>> > > }
>> > > }
>> > > }
>> > >
>> > > and finally prints (in my try):
>> > >
>> > > Misses count: 97
>> > >
>> > > On the server nodes misses were counted, but only for one server node:
>> > >
>> > > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
>> > > entryProcessorReadOnlyInvocations=0,
>> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
>> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
>> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
>> > > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=3,
>> > > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
>> > > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
>> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
>> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
>> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
>> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
>> > > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
>> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
>> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
>> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
>> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
>> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
>> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
>> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
>> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
>> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
>> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
>> > > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
>> > > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
>> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
>> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
>> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
>> > > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
>> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>> > >
>> > > Visor confirms my observation:
>> > >
>> > > Nodes for: test(@c0)
>> > >
>> > > +==================================================================================================================+
>> > > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
>> > > | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
>> > >
>> > > +==================================================================================================================+
>> > > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291
>> > > | Total: 47 (47 / 0) | Hi: 0 |
>> > > | | | | |
>> > > | Heap: 0 (0 / <n/a>) | Mi: 0 |
>> > > | | | | |
>> > > | Off-Heap: 47 (47 / 0) | Rd: 0 |
>> > > | | | | |
>> > > | Off-Heap Memory: <n/a> | Wr: 0 |
>> > >
>> > > +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
>> > > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122
>> > > | Total: 53 (53 / 0) | Hi: 3 |
>> > > | | | | |
>> > > | Heap: 0 (0 / <n/a>) | Mi: 48 |
>> > > | | | | |
>> > > | Off-Heap: 53 (53 / 0) | Rd: 51 |
>> > > | | | | |
>> > > | Off-Heap Memory: <n/a> | Wr: 100 |
>> > >
>> > > +------------------------------------------------------------------------------------------------------------------+
>> > >
>> > > I have seen the same behaviour with rest api as with the thin client, but
>> > > I don't have prepared test case for this.
>> > >
>> > > Is there a good way to correctly configure collecting statistics on server
>> > > nodes?
>> > > Should I create an issue for this problem?
>> > >
>> > > --
>> > > Pozdrawiam / Regards,
>> > > Dominik Przybysz
>> > >
>
>
>
> --
> Pozdrawiam / Regards,
> Dominik Przybysz
Re: Cache metrics on server nodes does not update correctly
Posted by Dominik Przybysz <al...@gmail.com>.
Hi,
I used ignite in version 2.7.6 (but I have also seen this behaviour on
other 2.7.x versions) and there aren't any near or local cache.
I expect that if I ask distributed cache about key which does not exist
then the miss metric will be incremented.
śr., 11 mar 2020 o 11:35 Andrey Gura <ag...@apache.org> napisał(a):
> Denis,
>
> I don't sure that I understand what is expected behavior should be.
> There are local and aggregated cluster wide metrics. I don't know
> which one used by Visor because I never used it :)
>
> Also it would be great to know what version of Apache Ignite used in
> described case. I remember some bug with metrics aggregation during
> discovery metrics message round trip.
>
> On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dm...@apache.org> wrote:
> >
> > @Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
> > could you folks check out this thread?
> >
> > I have a feeling that what Dominik is describing was talked out before
> and
> > rather some sort of a limitation than an issue with the current
> > implementation.
> >
> > -
> > Denis
> >
> >
> > On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I am trying to use partitioned cache on server nodes to which I connect
> > > with client node. Statistics of cache in the cluster are updated, but
> only
> > > for hits metric - misses metric is always 0.
> > >
> > > To reproduce this problem I created cluster of two nodes:
> > >
> > > Server node 1 adds 100 random test cases and prints cache statistics
> > > continuously:
> > >
> > > public class IgniteClusterNode1 {
> > > public static void main(String[] args) throws InterruptedException
> {
> > > IgniteConfiguration igniteConfiguration = new
> > > IgniteConfiguration();
> > >
> > > CacheConfiguration cacheConfiguration = new
> CacheConfiguration();
> > > cacheConfiguration.setName("test");
> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> > > cacheConfiguration.setStatisticsEnabled(true);
> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> > >
> > > TcpCommunicationSpi communicationSpi = new
> TcpCommunicationSpi();
> > > communicationSpi.setLocalPort(47500);
> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
> > >
> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > > discoverySpi.setLocalPort(47100);
> > > discoverySpi.setLocalPortRange(100);
> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > > "127.0.0.1:48100..48200"));
> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
> > >
> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > > try (IgniteCache<String, String> cache =
> > > ignite.getOrCreateCache("test")) {
> > > new Random().ints(1000).map(i -> Math.abs(i %
> > > 1000)).distinct().limit(100).forEach(i -> {
> > > String key = "data_" + i;
> > > String value =
> UUID.randomUUID().toString();
> > > cache.put(key, value);
> > > }
> > > );
> > > }
> > > while (true) {
> > > System.out.println(ignite.cache("test").metrics());
> > > Thread.sleep(5000);
> > > }
> > > }
> > > }
> > > }
> > >
> > > Server node 2 only prints cache statistics continuously:
> > >
> > > public class IgniteClusterNode2 {
> > > public static void main(String[] args) throws InterruptedException
> {
> > > IgniteConfiguration igniteConfiguration = new
> > > IgniteConfiguration();
> > >
> > > CacheConfiguration cacheConfiguration = new
> CacheConfiguration();
> > > cacheConfiguration.setName("test");
> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > > cacheConfiguration.setStatisticsEnabled(true);
> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> > >
> > > TcpCommunicationSpi communicationSpi = new
> TcpCommunicationSpi();
> > > communicationSpi.setLocalPort(48500);
> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
> > >
> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > > discoverySpi.setLocalPort(48100);
> > > discoverySpi.setLocalPortRange(100);
> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > > "127.0.0.1:48100..48200"));
> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
> > >
> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > > while (true) {
> > > System.out.println(ignite.cache("test").metrics());
> > > Thread.sleep(5000);
> > > }
> > > }
> > > }
> > > }
> > >
> > > Next I start a client node which continuously read data from the
> cluster:
> > >
> > > public class CacheClusterReader {
> > > public static void main(String[] args) throws InterruptedException
> {
> > > IgniteConfiguration cfg = new IgniteConfiguration();
> > > cfg.setClientMode(true);
> > >
> > > TcpDiscoverySpi spi = new TcpDiscoverySpi();
> > > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> > > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > > "127.0.0.1:48100..48200"));
> > > spi.setIpFinder(tcMp);
> > > cfg.setDiscoverySpi(spi);
> > >
> > > CacheConfiguration<String, String> cacheConfig = new
> > > CacheConfiguration<>("test");
> > > cacheConfig.setStatisticsEnabled(true);
> > > cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > > cfg.setCacheConfiguration(cacheConfig);
> > >
> > > try (Ignite ignite = Ignition.start(cfg)) {
> > > System.out.println(ignite.cacheNames());
> > >
> > > while (true) {
> > > try (IgniteCache<String, String> cache =
> > > ignite.getOrCreateCache(cacheConfig)) {
> > > cache.enableStatistics(true);
> > >
> > > new Random().ints(1000).map(i -> Math.abs(i %
> > > 2000)).distinct().limit(100).forEach(i -> {
> > > String key = "data_" + i;
> > > String value = cache.get(key);
> > > System.out.println("Found " + key + "
> with
> > > value " + value);
> > > }
> > > );
> > >
> > > System.out.println(cache.metrics());
> > > System.out.println(cache.localMetrics());
> > > }
> > > Thread.sleep(3000);
> > > }
> > > }
> > > }
> > > }
> > >
> > > On server and client nodes cluster statistics are synchronized, but
> misses
> > > counter is always equal to 0, e. g.
> > >
> > > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
> > > entryProcessorReadOnlyInvocations=0,
> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > > entryProcessorMaxInvocationTime=0.0,
> entryProcessorMinInvocationTime=0.0, *hits=93,
> > > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100,
> heapEntriesCnt=0,
> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56,
> isEmpty=false,
> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > > writeBehindBufSize=-1, totalPartitionsCnt=1024,
> rebalancingPartitionsCnt=0,
> > > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > > isStatisticsEnabled=true, isManagementEnabled=false,
> isReadThrough=false,
> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> > >
> > > In comparision, I also have also a thin client which connects to server
> > > and read some random data once:
> > >
> > > public class CacheClusterThinClientReader {
> > > public static void main(String[] args) throws Exception {
> > > ClientConfiguration clientConfiguration = new
> ClientConfiguration()
> > > .setAddresses("127.0.0.1:10800");
> > >
> > > try (IgniteClient ignite =
> > > Ignition.startClient(clientConfiguration)) {
> > >
> > > ClientCache<String, String> cache =
> > > ignite.getOrCreateCache("test");
> > >
> > > Set<String> misses = new HashSet<>();
> > > new Random().ints(1000).map(i -> Math.abs(i %
> > > 2000)).distinct().limit(100).forEach(i -> {
> > > String key = "data_" + i;
> > > String value = cache.get(key);
> > > System.out.println("Found " + key + " with
> value "
> > > + value);
> > > if (value == null) {
> > > misses.add(key);
> > > }
> > > }
> > > );
> > > System.out.println("Misses count: " + misses.size());
> > > }
> > > }
> > > }
> > >
> > > and finally prints (in my try):
> > >
> > > Misses count: 97
> > >
> > > On the server nodes misses were counted, but only for one server node:
> > >
> > > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
> > > entryProcessorReadOnlyInvocations=0,
> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > > entryProcessorMaxInvocationTime=0.0,
> entryProcessorMinInvocationTime=0.0, *hits=3,
> > > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735,
> rmvAvgTimeNanos=0.0,
> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100,
> heapEntriesCnt=0,
> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53,
> isEmpty=false,
> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > > writeBehindBufSize=-1, totalPartitionsCnt=1024,
> rebalancingPartitionsCnt=0,
> > > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > > isStatisticsEnabled=true, isManagementEnabled=false,
> isReadThrough=false,
> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> > >
> > > Visor confirms my observation:
> > >
> > > Nodes for: test(@c0)
> > >
> > >
> +==================================================================================================================+
> > > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
> > > | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
> > >
> > >
> +==================================================================================================================+
> > > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % |
> 00:15:00.291
> > > | Total: 47 (47 / 0) | Hi: 0 |
> > > | | | | |
> > > | Heap: 0 (0 / <n/a>) | Mi: 0 |
> > > | | | | |
> > > | Off-Heap: 47 (47 / 0) | Rd: 0 |
> > > | | | | |
> > > | Off-Heap Memory: <n/a> | Wr: 0 |
> > >
> > >
> +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
> > > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % |
> 00:15:04.122
> > > | Total: 53 (53 / 0) | Hi: 3 |
> > > | | | | |
> > > | Heap: 0 (0 / <n/a>) | Mi: 48 |
> > > | | | | |
> > > | Off-Heap: 53 (53 / 0) | Rd: 51 |
> > > | | | | |
> > > | Off-Heap Memory: <n/a> | Wr: 100 |
> > >
> > >
> +------------------------------------------------------------------------------------------------------------------+
> > >
> > > I have seen the same behaviour with rest api as with the thin client,
> but
> > > I don't have prepared test case for this.
> > >
> > > Is there a good way to correctly configure collecting statistics on
> server
> > > nodes?
> > > Should I create an issue for this problem?
> > >
> > > --
> > > Pozdrawiam / Regards,
> > > Dominik Przybysz
> > >
>
--
Pozdrawiam / Regards,
Dominik Przybysz
Re: Cache metrics on server nodes does not update correctly
Posted by Dominik Przybysz <al...@gmail.com>.
Hi,
I used ignite in version 2.7.6 (but I have also seen this behaviour on
other 2.7.x versions) and there aren't any near or local cache.
I expect that if I ask distributed cache about key which does not exist
then the miss metric will be incremented.
śr., 11 mar 2020 o 11:35 Andrey Gura <ag...@apache.org> napisał(a):
> Denis,
>
> I don't sure that I understand what is expected behavior should be.
> There are local and aggregated cluster wide metrics. I don't know
> which one used by Visor because I never used it :)
>
> Also it would be great to know what version of Apache Ignite used in
> described case. I remember some bug with metrics aggregation during
> discovery metrics message round trip.
>
> On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dm...@apache.org> wrote:
> >
> > @Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
> > could you folks check out this thread?
> >
> > I have a feeling that what Dominik is describing was talked out before
> and
> > rather some sort of a limitation than an issue with the current
> > implementation.
> >
> > -
> > Denis
> >
> >
> > On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I am trying to use partitioned cache on server nodes to which I connect
> > > with client node. Statistics of cache in the cluster are updated, but
> only
> > > for hits metric - misses metric is always 0.
> > >
> > > To reproduce this problem I created cluster of two nodes:
> > >
> > > Server node 1 adds 100 random test cases and prints cache statistics
> > > continuously:
> > >
> > > public class IgniteClusterNode1 {
> > > public static void main(String[] args) throws InterruptedException
> {
> > > IgniteConfiguration igniteConfiguration = new
> > > IgniteConfiguration();
> > >
> > > CacheConfiguration cacheConfiguration = new
> CacheConfiguration();
> > > cacheConfiguration.setName("test");
> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> > > cacheConfiguration.setStatisticsEnabled(true);
> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> > >
> > > TcpCommunicationSpi communicationSpi = new
> TcpCommunicationSpi();
> > > communicationSpi.setLocalPort(47500);
> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
> > >
> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > > discoverySpi.setLocalPort(47100);
> > > discoverySpi.setLocalPortRange(100);
> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > > "127.0.0.1:48100..48200"));
> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
> > >
> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > > try (IgniteCache<String, String> cache =
> > > ignite.getOrCreateCache("test")) {
> > > new Random().ints(1000).map(i -> Math.abs(i %
> > > 1000)).distinct().limit(100).forEach(i -> {
> > > String key = "data_" + i;
> > > String value =
> UUID.randomUUID().toString();
> > > cache.put(key, value);
> > > }
> > > );
> > > }
> > > while (true) {
> > > System.out.println(ignite.cache("test").metrics());
> > > Thread.sleep(5000);
> > > }
> > > }
> > > }
> > > }
> > >
> > > Server node 2 only prints cache statistics continuously:
> > >
> > > public class IgniteClusterNode2 {
> > > public static void main(String[] args) throws InterruptedException
> {
> > > IgniteConfiguration igniteConfiguration = new
> > > IgniteConfiguration();
> > >
> > > CacheConfiguration cacheConfiguration = new
> CacheConfiguration();
> > > cacheConfiguration.setName("test");
> > > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > > cacheConfiguration.setStatisticsEnabled(true);
> > > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> > >
> > > TcpCommunicationSpi communicationSpi = new
> TcpCommunicationSpi();
> > > communicationSpi.setLocalPort(48500);
> > > igniteConfiguration.setCommunicationSpi(communicationSpi);
> > >
> > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > > discoverySpi.setLocalPort(48100);
> > > discoverySpi.setLocalPortRange(100);
> > > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > > "127.0.0.1:48100..48200"));
> > > igniteConfiguration.setDiscoverySpi(discoverySpi);
> > >
> > > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > > while (true) {
> > > System.out.println(ignite.cache("test").metrics());
> > > Thread.sleep(5000);
> > > }
> > > }
> > > }
> > > }
> > >
> > > Next I start a client node which continuously read data from the
> cluster:
> > >
> > > public class CacheClusterReader {
> > > public static void main(String[] args) throws InterruptedException
> {
> > > IgniteConfiguration cfg = new IgniteConfiguration();
> > > cfg.setClientMode(true);
> > >
> > > TcpDiscoverySpi spi = new TcpDiscoverySpi();
> > > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> > > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > > "127.0.0.1:48100..48200"));
> > > spi.setIpFinder(tcMp);
> > > cfg.setDiscoverySpi(spi);
> > >
> > > CacheConfiguration<String, String> cacheConfig = new
> > > CacheConfiguration<>("test");
> > > cacheConfig.setStatisticsEnabled(true);
> > > cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > > cfg.setCacheConfiguration(cacheConfig);
> > >
> > > try (Ignite ignite = Ignition.start(cfg)) {
> > > System.out.println(ignite.cacheNames());
> > >
> > > while (true) {
> > > try (IgniteCache<String, String> cache =
> > > ignite.getOrCreateCache(cacheConfig)) {
> > > cache.enableStatistics(true);
> > >
> > > new Random().ints(1000).map(i -> Math.abs(i %
> > > 2000)).distinct().limit(100).forEach(i -> {
> > > String key = "data_" + i;
> > > String value = cache.get(key);
> > > System.out.println("Found " + key + "
> with
> > > value " + value);
> > > }
> > > );
> > >
> > > System.out.println(cache.metrics());
> > > System.out.println(cache.localMetrics());
> > > }
> > > Thread.sleep(3000);
> > > }
> > > }
> > > }
> > > }
> > >
> > > On server and client nodes cluster statistics are synchronized, but
> misses
> > > counter is always equal to 0, e. g.
> > >
> > > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
> > > entryProcessorReadOnlyInvocations=0,
> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > > entryProcessorMaxInvocationTime=0.0,
> entryProcessorMinInvocationTime=0.0, *hits=93,
> > > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100,
> heapEntriesCnt=0,
> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56,
> isEmpty=false,
> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > > writeBehindBufSize=-1, totalPartitionsCnt=1024,
> rebalancingPartitionsCnt=0,
> > > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > > isStatisticsEnabled=true, isManagementEnabled=false,
> isReadThrough=false,
> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> > >
> > > In comparision, I also have also a thin client which connects to server
> > > and read some random data once:
> > >
> > > public class CacheClusterThinClientReader {
> > > public static void main(String[] args) throws Exception {
> > > ClientConfiguration clientConfiguration = new
> ClientConfiguration()
> > > .setAddresses("127.0.0.1:10800");
> > >
> > > try (IgniteClient ignite =
> > > Ignition.startClient(clientConfiguration)) {
> > >
> > > ClientCache<String, String> cache =
> > > ignite.getOrCreateCache("test");
> > >
> > > Set<String> misses = new HashSet<>();
> > > new Random().ints(1000).map(i -> Math.abs(i %
> > > 2000)).distinct().limit(100).forEach(i -> {
> > > String key = "data_" + i;
> > > String value = cache.get(key);
> > > System.out.println("Found " + key + " with
> value "
> > > + value);
> > > if (value == null) {
> > > misses.add(key);
> > > }
> > > }
> > > );
> > > System.out.println("Misses count: " + misses.size());
> > > }
> > > }
> > > }
> > >
> > > and finally prints (in my try):
> > >
> > > Misses count: 97
> > >
> > > On the server nodes misses were counted, but only for one server node:
> > >
> > > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
> > > entryProcessorReadOnlyInvocations=0,
> > > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > > entryProcessorMaxInvocationTime=0.0,
> entryProcessorMinInvocationTime=0.0, *hits=3,
> > > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735,
> rmvAvgTimeNanos=0.0,
> > > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100,
> heapEntriesCnt=0,
> > > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53,
> isEmpty=false,
> > > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > > writeBehindBufSize=-1, totalPartitionsCnt=1024,
> rebalancingPartitionsCnt=0,
> > > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
> > > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > > isStatisticsEnabled=true, isManagementEnabled=false,
> isReadThrough=false,
> > > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> > >
> > > Visor confirms my observation:
> > >
> > > Nodes for: test(@c0)
> > >
> > >
> +==================================================================================================================+
> > > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
> > > | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
> > >
> > >
> +==================================================================================================================+
> > > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % |
> 00:15:00.291
> > > | Total: 47 (47 / 0) | Hi: 0 |
> > > | | | | |
> > > | Heap: 0 (0 / <n/a>) | Mi: 0 |
> > > | | | | |
> > > | Off-Heap: 47 (47 / 0) | Rd: 0 |
> > > | | | | |
> > > | Off-Heap Memory: <n/a> | Wr: 0 |
> > >
> > >
> +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
> > > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % |
> 00:15:04.122
> > > | Total: 53 (53 / 0) | Hi: 3 |
> > > | | | | |
> > > | Heap: 0 (0 / <n/a>) | Mi: 48 |
> > > | | | | |
> > > | Off-Heap: 53 (53 / 0) | Rd: 51 |
> > > | | | | |
> > > | Off-Heap Memory: <n/a> | Wr: 100 |
> > >
> > >
> +------------------------------------------------------------------------------------------------------------------+
> > >
> > > I have seen the same behaviour with rest api as with the thin client,
> but
> > > I don't have prepared test case for this.
> > >
> > > Is there a good way to correctly configure collecting statistics on
> server
> > > nodes?
> > > Should I create an issue for this problem?
> > >
> > > --
> > > Pozdrawiam / Regards,
> > > Dominik Przybysz
> > >
>
--
Pozdrawiam / Regards,
Dominik Przybysz
Re: Cache metrics on server nodes does not update correctly
Posted by Andrey Gura <ag...@apache.org>.
Denis,
I don't sure that I understand what is expected behavior should be.
There are local and aggregated cluster wide metrics. I don't know
which one used by Visor because I never used it :)
Also it would be great to know what version of Apache Ignite used in
described case. I remember some bug with metrics aggregation during
discovery metrics message round trip.
On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dm...@apache.org> wrote:
>
> @Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
> could you folks check out this thread?
>
> I have a feeling that what Dominik is describing was talked out before and
> rather some sort of a limitation than an issue with the current
> implementation.
>
> -
> Denis
>
>
> On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
> wrote:
>
> > Hi,
> > I am trying to use partitioned cache on server nodes to which I connect
> > with client node. Statistics of cache in the cluster are updated, but only
> > for hits metric - misses metric is always 0.
> >
> > To reproduce this problem I created cluster of two nodes:
> >
> > Server node 1 adds 100 random test cases and prints cache statistics
> > continuously:
> >
> > public class IgniteClusterNode1 {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration igniteConfiguration = new
> > IgniteConfiguration();
> >
> > CacheConfiguration cacheConfiguration = new CacheConfiguration();
> > cacheConfiguration.setName("test");
> > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> > cacheConfiguration.setStatisticsEnabled(true);
> > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> >
> > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> > communicationSpi.setLocalPort(47500);
> > igniteConfiguration.setCommunicationSpi(communicationSpi);
> >
> > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > discoverySpi.setLocalPort(47100);
> > discoverySpi.setLocalPortRange(100);
> > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > igniteConfiguration.setDiscoverySpi(discoverySpi);
> >
> > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > try (IgniteCache<String, String> cache =
> > ignite.getOrCreateCache("test")) {
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 1000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = UUID.randomUUID().toString();
> > cache.put(key, value);
> > }
> > );
> > }
> > while (true) {
> > System.out.println(ignite.cache("test").metrics());
> > Thread.sleep(5000);
> > }
> > }
> > }
> > }
> >
> > Server node 2 only prints cache statistics continuously:
> >
> > public class IgniteClusterNode2 {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration igniteConfiguration = new
> > IgniteConfiguration();
> >
> > CacheConfiguration cacheConfiguration = new CacheConfiguration();
> > cacheConfiguration.setName("test");
> > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > cacheConfiguration.setStatisticsEnabled(true);
> > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> >
> > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> > communicationSpi.setLocalPort(48500);
> > igniteConfiguration.setCommunicationSpi(communicationSpi);
> >
> > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > discoverySpi.setLocalPort(48100);
> > discoverySpi.setLocalPortRange(100);
> > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > igniteConfiguration.setDiscoverySpi(discoverySpi);
> >
> > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > while (true) {
> > System.out.println(ignite.cache("test").metrics());
> > Thread.sleep(5000);
> > }
> > }
> > }
> > }
> >
> > Next I start a client node which continuously read data from the cluster:
> >
> > public class CacheClusterReader {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setClientMode(true);
> >
> > TcpDiscoverySpi spi = new TcpDiscoverySpi();
> > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > spi.setIpFinder(tcMp);
> > cfg.setDiscoverySpi(spi);
> >
> > CacheConfiguration<String, String> cacheConfig = new
> > CacheConfiguration<>("test");
> > cacheConfig.setStatisticsEnabled(true);
> > cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > cfg.setCacheConfiguration(cacheConfig);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > System.out.println(ignite.cacheNames());
> >
> > while (true) {
> > try (IgniteCache<String, String> cache =
> > ignite.getOrCreateCache(cacheConfig)) {
> > cache.enableStatistics(true);
> >
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 2000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = cache.get(key);
> > System.out.println("Found " + key + " with
> > value " + value);
> > }
> > );
> >
> > System.out.println(cache.metrics());
> > System.out.println(cache.localMetrics());
> > }
> > Thread.sleep(3000);
> > }
> > }
> > }
> > }
> >
> > On server and client nodes cluster statistics are synchronized, but misses
> > counter is always equal to 0, e. g.
> >
> > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
> > entryProcessorReadOnlyInvocations=0,
> > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=93,
> > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
> > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
> > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
> > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> >
> > In comparision, I also have also a thin client which connects to server
> > and read some random data once:
> >
> > public class CacheClusterThinClientReader {
> > public static void main(String[] args) throws Exception {
> > ClientConfiguration clientConfiguration = new ClientConfiguration()
> > .setAddresses("127.0.0.1:10800");
> >
> > try (IgniteClient ignite =
> > Ignition.startClient(clientConfiguration)) {
> >
> > ClientCache<String, String> cache =
> > ignite.getOrCreateCache("test");
> >
> > Set<String> misses = new HashSet<>();
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 2000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = cache.get(key);
> > System.out.println("Found " + key + " with value "
> > + value);
> > if (value == null) {
> > misses.add(key);
> > }
> > }
> > );
> > System.out.println("Misses count: " + misses.size());
> > }
> > }
> > }
> >
> > and finally prints (in my try):
> >
> > Misses count: 97
> >
> > On the server nodes misses were counted, but only for one server node:
> >
> > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
> > entryProcessorReadOnlyInvocations=0,
> > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=3,
> > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
> > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
> > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
> > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> >
> > Visor confirms my observation:
> >
> > Nodes for: test(@c0)
> >
> > +==================================================================================================================+
> > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
> > | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
> >
> > +==================================================================================================================+
> > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291
> > | Total: 47 (47 / 0) | Hi: 0 |
> > | | | | |
> > | Heap: 0 (0 / <n/a>) | Mi: 0 |
> > | | | | |
> > | Off-Heap: 47 (47 / 0) | Rd: 0 |
> > | | | | |
> > | Off-Heap Memory: <n/a> | Wr: 0 |
> >
> > +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
> > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122
> > | Total: 53 (53 / 0) | Hi: 3 |
> > | | | | |
> > | Heap: 0 (0 / <n/a>) | Mi: 48 |
> > | | | | |
> > | Off-Heap: 53 (53 / 0) | Rd: 51 |
> > | | | | |
> > | Off-Heap Memory: <n/a> | Wr: 100 |
> >
> > +------------------------------------------------------------------------------------------------------------------+
> >
> > I have seen the same behaviour with rest api as with the thin client, but
> > I don't have prepared test case for this.
> >
> > Is there a good way to correctly configure collecting statistics on server
> > nodes?
> > Should I create an issue for this problem?
> >
> > --
> > Pozdrawiam / Regards,
> > Dominik Przybysz
> >
Re: Cache metrics on server nodes does not update correctly
Posted by Andrey Gura <ag...@apache.org>.
Denis,
I don't sure that I understand what is expected behavior should be.
There are local and aggregated cluster wide metrics. I don't know
which one used by Visor because I never used it :)
Also it would be great to know what version of Apache Ignite used in
described case. I remember some bug with metrics aggregation during
discovery metrics message round trip.
On Wed, Mar 11, 2020 at 12:05 AM Denis Magda <dm...@apache.org> wrote:
>
> @Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
> could you folks check out this thread?
>
> I have a feeling that what Dominik is describing was talked out before and
> rather some sort of a limitation than an issue with the current
> implementation.
>
> -
> Denis
>
>
> On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
> wrote:
>
> > Hi,
> > I am trying to use partitioned cache on server nodes to which I connect
> > with client node. Statistics of cache in the cluster are updated, but only
> > for hits metric - misses metric is always 0.
> >
> > To reproduce this problem I created cluster of two nodes:
> >
> > Server node 1 adds 100 random test cases and prints cache statistics
> > continuously:
> >
> > public class IgniteClusterNode1 {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration igniteConfiguration = new
> > IgniteConfiguration();
> >
> > CacheConfiguration cacheConfiguration = new CacheConfiguration();
> > cacheConfiguration.setName("test");
> > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> > cacheConfiguration.setStatisticsEnabled(true);
> > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> >
> > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> > communicationSpi.setLocalPort(47500);
> > igniteConfiguration.setCommunicationSpi(communicationSpi);
> >
> > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > discoverySpi.setLocalPort(47100);
> > discoverySpi.setLocalPortRange(100);
> > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > igniteConfiguration.setDiscoverySpi(discoverySpi);
> >
> > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > try (IgniteCache<String, String> cache =
> > ignite.getOrCreateCache("test")) {
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 1000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = UUID.randomUUID().toString();
> > cache.put(key, value);
> > }
> > );
> > }
> > while (true) {
> > System.out.println(ignite.cache("test").metrics());
> > Thread.sleep(5000);
> > }
> > }
> > }
> > }
> >
> > Server node 2 only prints cache statistics continuously:
> >
> > public class IgniteClusterNode2 {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration igniteConfiguration = new
> > IgniteConfiguration();
> >
> > CacheConfiguration cacheConfiguration = new CacheConfiguration();
> > cacheConfiguration.setName("test");
> > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> > cacheConfiguration.setStatisticsEnabled(true);
> > igniteConfiguration.setCacheConfiguration(cacheConfiguration);
> >
> > TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> > communicationSpi.setLocalPort(48500);
> > igniteConfiguration.setCommunicationSpi(communicationSpi);
> >
> > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> > discoverySpi.setLocalPort(48100);
> > discoverySpi.setLocalPortRange(100);
> > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > igniteConfiguration.setDiscoverySpi(discoverySpi);
> >
> > try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> > while (true) {
> > System.out.println(ignite.cache("test").metrics());
> > Thread.sleep(5000);
> > }
> > }
> > }
> > }
> >
> > Next I start a client node which continuously read data from the cluster:
> >
> > public class CacheClusterReader {
> > public static void main(String[] args) throws InterruptedException {
> > IgniteConfiguration cfg = new IgniteConfiguration();
> > cfg.setClientMode(true);
> >
> > TcpDiscoverySpi spi = new TcpDiscoverySpi();
> > TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> > tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> > "127.0.0.1:48100..48200"));
> > spi.setIpFinder(tcMp);
> > cfg.setDiscoverySpi(spi);
> >
> > CacheConfiguration<String, String> cacheConfig = new
> > CacheConfiguration<>("test");
> > cacheConfig.setStatisticsEnabled(true);
> > cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> > cfg.setCacheConfiguration(cacheConfig);
> >
> > try (Ignite ignite = Ignition.start(cfg)) {
> > System.out.println(ignite.cacheNames());
> >
> > while (true) {
> > try (IgniteCache<String, String> cache =
> > ignite.getOrCreateCache(cacheConfig)) {
> > cache.enableStatistics(true);
> >
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 2000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = cache.get(key);
> > System.out.println("Found " + key + " with
> > value " + value);
> > }
> > );
> >
> > System.out.println(cache.metrics());
> > System.out.println(cache.localMetrics());
> > }
> > Thread.sleep(3000);
> > }
> > }
> > }
> > }
> >
> > On server and client nodes cluster statistics are synchronized, but misses
> > counter is always equal to 0, e. g.
> >
> > CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
> > entryProcessorReadOnlyInvocations=0,
> > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=93,
> > misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
> > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
> > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> > rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
> > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> >
> > In comparision, I also have also a thin client which connects to server
> > and read some random data once:
> >
> > public class CacheClusterThinClientReader {
> > public static void main(String[] args) throws Exception {
> > ClientConfiguration clientConfiguration = new ClientConfiguration()
> > .setAddresses("127.0.0.1:10800");
> >
> > try (IgniteClient ignite =
> > Ignition.startClient(clientConfiguration)) {
> >
> > ClientCache<String, String> cache =
> > ignite.getOrCreateCache("test");
> >
> > Set<String> misses = new HashSet<>();
> > new Random().ints(1000).map(i -> Math.abs(i %
> > 2000)).distinct().limit(100).forEach(i -> {
> > String key = "data_" + i;
> > String value = cache.get(key);
> > System.out.println("Found " + key + " with value "
> > + value);
> > if (value == null) {
> > misses.add(key);
> > }
> > }
> > );
> > System.out.println("Misses count: " + misses.size());
> > }
> > }
> > }
> >
> > and finally prints (in my try):
> >
> > Misses count: 97
> >
> > On the server nodes misses were counted, but only for one server node:
> >
> > CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
> > entryProcessorReadOnlyInvocations=0,
> > entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> > entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> > entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> > entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=3,
> > misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> > putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
> > commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> > offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> > offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> > offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> > offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
> > dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> > txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> > txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> > txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> > txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> > txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> > isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> > writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> > writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> > writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> > writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> > rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
> > rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> > rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> > keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> > isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> > isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
> >
> > Visor confirms my observation:
> >
> > Nodes for: test(@c0)
> >
> > +==================================================================================================================+
> > | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
> > | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
> >
> > +==================================================================================================================+
> > | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291
> > | Total: 47 (47 / 0) | Hi: 0 |
> > | | | | |
> > | Heap: 0 (0 / <n/a>) | Mi: 0 |
> > | | | | |
> > | Off-Heap: 47 (47 / 0) | Rd: 0 |
> > | | | | |
> > | Off-Heap Memory: <n/a> | Wr: 0 |
> >
> > +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
> > | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122
> > | Total: 53 (53 / 0) | Hi: 3 |
> > | | | | |
> > | Heap: 0 (0 / <n/a>) | Mi: 48 |
> > | | | | |
> > | Off-Heap: 53 (53 / 0) | Rd: 51 |
> > | | | | |
> > | Off-Heap Memory: <n/a> | Wr: 100 |
> >
> > +------------------------------------------------------------------------------------------------------------------+
> >
> > I have seen the same behaviour with rest api as with the thin client, but
> > I don't have prepared test case for this.
> >
> > Is there a good way to correctly configure collecting statistics on server
> > nodes?
> > Should I create an issue for this problem?
> >
> > --
> > Pozdrawiam / Regards,
> > Dominik Przybysz
> >
Re: Cache metrics on server nodes does not update correctly
Posted by Denis Magda <dm...@apache.org>.
@Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
could you folks check out this thread?
I have a feeling that what Dominik is describing was talked out before and
rather some sort of a limitation than an issue with the current
implementation.
-
Denis
On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
wrote:
> Hi,
> I am trying to use partitioned cache on server nodes to which I connect
> with client node. Statistics of cache in the cluster are updated, but only
> for hits metric - misses metric is always 0.
>
> To reproduce this problem I created cluster of two nodes:
>
> Server node 1 adds 100 random test cases and prints cache statistics
> continuously:
>
> public class IgniteClusterNode1 {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
> CacheConfiguration cacheConfiguration = new CacheConfiguration();
> cacheConfiguration.setName("test");
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheConfiguration.setStatisticsEnabled(true);
> igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>
> TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> communicationSpi.setLocalPort(47500);
> igniteConfiguration.setCommunicationSpi(communicationSpi);
>
> TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> discoverySpi.setLocalPort(47100);
> discoverySpi.setLocalPortRange(100);
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> igniteConfiguration.setDiscoverySpi(discoverySpi);
>
> try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> try (IgniteCache<String, String> cache =
> ignite.getOrCreateCache("test")) {
> new Random().ints(1000).map(i -> Math.abs(i %
> 1000)).distinct().limit(100).forEach(i -> {
> String key = "data_" + i;
> String value = UUID.randomUUID().toString();
> cache.put(key, value);
> }
> );
> }
> while (true) {
> System.out.println(ignite.cache("test").metrics());
> Thread.sleep(5000);
> }
> }
> }
> }
>
> Server node 2 only prints cache statistics continuously:
>
> public class IgniteClusterNode2 {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
> CacheConfiguration cacheConfiguration = new CacheConfiguration();
> cacheConfiguration.setName("test");
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setStatisticsEnabled(true);
> igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>
> TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> communicationSpi.setLocalPort(48500);
> igniteConfiguration.setCommunicationSpi(communicationSpi);
>
> TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> discoverySpi.setLocalPort(48100);
> discoverySpi.setLocalPortRange(100);
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> igniteConfiguration.setDiscoverySpi(discoverySpi);
>
> try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> while (true) {
> System.out.println(ignite.cache("test").metrics());
> Thread.sleep(5000);
> }
> }
> }
> }
>
> Next I start a client node which continuously read data from the cluster:
>
> public class CacheClusterReader {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration cfg = new IgniteConfiguration();
> cfg.setClientMode(true);
>
> TcpDiscoverySpi spi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> spi.setIpFinder(tcMp);
> cfg.setDiscoverySpi(spi);
>
> CacheConfiguration<String, String> cacheConfig = new
> CacheConfiguration<>("test");
> cacheConfig.setStatisticsEnabled(true);
> cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> cfg.setCacheConfiguration(cacheConfig);
>
> try (Ignite ignite = Ignition.start(cfg)) {
> System.out.println(ignite.cacheNames());
>
> while (true) {
> try (IgniteCache<String, String> cache =
> ignite.getOrCreateCache(cacheConfig)) {
> cache.enableStatistics(true);
>
> new Random().ints(1000).map(i -> Math.abs(i %
> 2000)).distinct().limit(100).forEach(i -> {
> String key = "data_" + i;
> String value = cache.get(key);
> System.out.println("Found " + key + " with
> value " + value);
> }
> );
>
> System.out.println(cache.metrics());
> System.out.println(cache.localMetrics());
> }
> Thread.sleep(3000);
> }
> }
> }
> }
>
> On server and client nodes cluster statistics are synchronized, but misses
> counter is always equal to 0, e. g.
>
> CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
> entryProcessorReadOnlyInvocations=0,
> entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=93,
> misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
> commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
> dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
> rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>
> In comparision, I also have also a thin client which connects to server
> and read some random data once:
>
> public class CacheClusterThinClientReader {
> public static void main(String[] args) throws Exception {
> ClientConfiguration clientConfiguration = new ClientConfiguration()
> .setAddresses("127.0.0.1:10800");
>
> try (IgniteClient ignite =
> Ignition.startClient(clientConfiguration)) {
>
> ClientCache<String, String> cache =
> ignite.getOrCreateCache("test");
>
> Set<String> misses = new HashSet<>();
> new Random().ints(1000).map(i -> Math.abs(i %
> 2000)).distinct().limit(100).forEach(i -> {
> String key = "data_" + i;
> String value = cache.get(key);
> System.out.println("Found " + key + " with value "
> + value);
> if (value == null) {
> misses.add(key);
> }
> }
> );
> System.out.println("Misses count: " + misses.size());
> }
> }
> }
>
> and finally prints (in my try):
>
> Misses count: 97
>
> On the server nodes misses were counted, but only for one server node:
>
> CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
> entryProcessorReadOnlyInvocations=0,
> entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=3,
> misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
> commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
> dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
> rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>
> Visor confirms my observation:
>
> Nodes for: test(@c0)
>
> +==================================================================================================================+
> | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
> | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
>
> +==================================================================================================================+
> | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291
> | Total: 47 (47 / 0) | Hi: 0 |
> | | | | |
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
> | Off-Heap: 47 (47 / 0) | Rd: 0 |
> | | | | |
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
> | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122
> | Total: 53 (53 / 0) | Hi: 3 |
> | | | | |
> | Heap: 0 (0 / <n/a>) | Mi: 48 |
> | | | | |
> | Off-Heap: 53 (53 / 0) | Rd: 51 |
> | | | | |
> | Off-Heap Memory: <n/a> | Wr: 100 |
>
> +------------------------------------------------------------------------------------------------------------------+
>
> I have seen the same behaviour with rest api as with the thin client, but
> I don't have prepared test case for this.
>
> Is there a good way to correctly configure collecting statistics on server
> nodes?
> Should I create an issue for this problem?
>
> --
> Pozdrawiam / Regards,
> Dominik Przybysz
>
Re: Cache metrics on server nodes does not update correctly
Posted by Denis Magda <dm...@apache.org>.
@Nikolay Izhikov <ni...@apache.org>, @Andrey Gura <ag...@apache.org>,
could you folks check out this thread?
I have a feeling that what Dominik is describing was talked out before and
rather some sort of a limitation than an issue with the current
implementation.
-
Denis
On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz <al...@gmail.com>
wrote:
> Hi,
> I am trying to use partitioned cache on server nodes to which I connect
> with client node. Statistics of cache in the cluster are updated, but only
> for hits metric - misses metric is always 0.
>
> To reproduce this problem I created cluster of two nodes:
>
> Server node 1 adds 100 random test cases and prints cache statistics
> continuously:
>
> public class IgniteClusterNode1 {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
> CacheConfiguration cacheConfiguration = new CacheConfiguration();
> cacheConfiguration.setName("test");
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheConfiguration.setStatisticsEnabled(true);
> igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>
> TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> communicationSpi.setLocalPort(47500);
> igniteConfiguration.setCommunicationSpi(communicationSpi);
>
> TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> discoverySpi.setLocalPort(47100);
> discoverySpi.setLocalPortRange(100);
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> igniteConfiguration.setDiscoverySpi(discoverySpi);
>
> try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> try (IgniteCache<String, String> cache =
> ignite.getOrCreateCache("test")) {
> new Random().ints(1000).map(i -> Math.abs(i %
> 1000)).distinct().limit(100).forEach(i -> {
> String key = "data_" + i;
> String value = UUID.randomUUID().toString();
> cache.put(key, value);
> }
> );
> }
> while (true) {
> System.out.println(ignite.cache("test").metrics());
> Thread.sleep(5000);
> }
> }
> }
> }
>
> Server node 2 only prints cache statistics continuously:
>
> public class IgniteClusterNode2 {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
> CacheConfiguration cacheConfiguration = new CacheConfiguration();
> cacheConfiguration.setName("test");
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setStatisticsEnabled(true);
> igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>
> TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> communicationSpi.setLocalPort(48500);
> igniteConfiguration.setCommunicationSpi(communicationSpi);
>
> TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> discoverySpi.setLocalPort(48100);
> discoverySpi.setLocalPortRange(100);
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> igniteConfiguration.setDiscoverySpi(discoverySpi);
>
> try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> while (true) {
> System.out.println(ignite.cache("test").metrics());
> Thread.sleep(5000);
> }
> }
> }
> }
>
> Next I start a client node which continuously read data from the cluster:
>
> public class CacheClusterReader {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration cfg = new IgniteConfiguration();
> cfg.setClientMode(true);
>
> TcpDiscoverySpi spi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> spi.setIpFinder(tcMp);
> cfg.setDiscoverySpi(spi);
>
> CacheConfiguration<String, String> cacheConfig = new
> CacheConfiguration<>("test");
> cacheConfig.setStatisticsEnabled(true);
> cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> cfg.setCacheConfiguration(cacheConfig);
>
> try (Ignite ignite = Ignition.start(cfg)) {
> System.out.println(ignite.cacheNames());
>
> while (true) {
> try (IgniteCache<String, String> cache =
> ignite.getOrCreateCache(cacheConfig)) {
> cache.enableStatistics(true);
>
> new Random().ints(1000).map(i -> Math.abs(i %
> 2000)).distinct().limit(100).forEach(i -> {
> String key = "data_" + i;
> String value = cache.get(key);
> System.out.println("Found " + key + " with
> value " + value);
> }
> );
>
> System.out.println(cache.metrics());
> System.out.println(cache.localMetrics());
> }
> Thread.sleep(3000);
> }
> }
> }
> }
>
> On server and client nodes cluster statistics are synchronized, but misses
> counter is always equal to 0, e. g.
>
> CacheMetricsSnapshot [reads=93, puts=100, entryProcessorPuts=0,
> entryProcessorReadOnlyInvocations=0,
> entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=93,
> misses=0*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> putAvgTimeNanos=259.4794, getAvgTimeNanos=0.0, rmvAvgTimeNanos=0.0,
> commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> offHeapAllocatedSize=0, size=56, cacheSize=56, keySize=56, isEmpty=false,
> dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> rebalancedKeys=44, estimatedRebalancingKeys=44, keysToRebalanceLeft=0,
> rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>
> In comparision, I also have also a thin client which connects to server
> and read some random data once:
>
> public class CacheClusterThinClientReader {
> public static void main(String[] args) throws Exception {
> ClientConfiguration clientConfiguration = new ClientConfiguration()
> .setAddresses("127.0.0.1:10800");
>
> try (IgniteClient ignite =
> Ignition.startClient(clientConfiguration)) {
>
> ClientCache<String, String> cache =
> ignite.getOrCreateCache("test");
>
> Set<String> misses = new HashSet<>();
> new Random().ints(1000).map(i -> Math.abs(i %
> 2000)).distinct().limit(100).forEach(i -> {
> String key = "data_" + i;
> String value = cache.get(key);
> System.out.println("Found " + key + " with value "
> + value);
> if (value == null) {
> misses.add(key);
> }
> }
> );
> System.out.println("Misses count: " + misses.size());
> }
> }
> }
>
> and finally prints (in my try):
>
> Misses count: 97
>
> On the server nodes misses were counted, but only for one server node:
>
> CacheMetricsSnapshot [reads=51, puts=100, entryProcessorPuts=0,
> entryProcessorReadOnlyInvocations=0,
> entryProcessorAverageInvocationTime=0.0, entryProcessorInvocations=0,
> entryProcessorRemovals=0, entryProcessorMisses=0, entryProcessorHits=0,
> entryProcessorMissPercentage=0.0, entryProcessorHitPercentage=0.0,
> entryProcessorMaxInvocationTime=0.0, entryProcessorMinInvocationTime=0.0, *hits=3,
> misses=48*, txCommits=0, txRollbacks=0, evicts=0, removes=0,
> putAvgTimeNanos=276.13132, getAvgTimeNanos=850.90735, rmvAvgTimeNanos=0.0,
> commitAvgTimeNanos=0.0, rollbackAvgTimeNanos=0.0, cacheName=test,
> offHeapGets=0, offHeapPuts=0, offHeapRemoves=0, offHeapEvicts=0,
> offHeapHits=0, offHeapMisses=0, offHeapEntriesCnt=100, heapEntriesCnt=0,
> offHeapPrimaryEntriesCnt=100, offHeapBackupEntriesCnt=0,
> offHeapAllocatedSize=0, size=53, cacheSize=53, keySize=53, isEmpty=false,
> dhtEvictQueueCurrSize=-1, txThreadMapSize=0, txXidMapSize=0,
> txCommitQueueSize=0, txPrepareQueueSize=0, txStartVerCountsSize=0,
> txCommittedVersionsSize=0, txRolledbackVersionsSize=0,
> txDhtThreadMapSize=0, txDhtXidMapSize=-1, txDhtCommitQueueSize=0,
> txDhtPrepareQueueSize=0, txDhtStartVerCountsSize=0,
> txDhtCommittedVersionsSize=-1, txDhtRolledbackVersionsSize=-1,
> isWriteBehindEnabled=false, writeBehindFlushSize=-1,
> writeBehindFlushThreadCnt=-1, writeBehindFlushFreq=-1,
> writeBehindStoreBatchSize=-1, writeBehindTotalCriticalOverflowCnt=-1,
> writeBehindCriticalOverflowCnt=-1, writeBehindErrorRetryCnt=-1,
> writeBehindBufSize=-1, totalPartitionsCnt=1024, rebalancingPartitionsCnt=0,
> rebalancedKeys=47, estimatedRebalancingKeys=47, keysToRebalanceLeft=0,
> rebalancingKeysRate=0, rebalancingBytesRate=0, rebalanceStartTime=0,
> rebalanceFinishTime=0, rebalanceClearingPartitionsLeft=0,
> keyType=java.lang.Object, valType=java.lang.Object, isStoreByVal=true,
> isStatisticsEnabled=true, isManagementEnabled=false, isReadThrough=false,
> isWriteThrough=false, isValidForReading=true, isValidForWriting=true]
>
> Visor confirms my observation:
>
> Nodes for: test(@c0)
>
> +==================================================================================================================+
> | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
> | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
>
> +==================================================================================================================+
> | D09713CD(@n1), 10.10.10.14 | 8 | 2.81 % | 0.53 % | 00:15:00.291
> | Total: 47 (47 / 0) | Hi: 0 |
> | | | | |
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
> | Off-Heap: 47 (47 / 0) | Rd: 0 |
> | | | | |
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +----------------------------+------+-----------+----------+--------------+--------------------------+-------------+
> | B8CCBCAB(@n0), 10.10.10.14 | 8 | 1.15 % | 0.50 % | 00:15:04.122
> | Total: 53 (53 / 0) | Hi: 3 |
> | | | | |
> | Heap: 0 (0 / <n/a>) | Mi: 48 |
> | | | | |
> | Off-Heap: 53 (53 / 0) | Rd: 51 |
> | | | | |
> | Off-Heap Memory: <n/a> | Wr: 100 |
>
> +------------------------------------------------------------------------------------------------------------------+
>
> I have seen the same behaviour with rest api as with the thin client, but
> I don't have prepared test case for this.
>
> Is there a good way to correctly configure collecting statistics on server
> nodes?
> Should I create an issue for this problem?
>
> --
> Pozdrawiam / Regards,
> Dominik Przybysz
>