You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2022/05/25 13:09:48 UTC
[GitHub] [ignite] NSAmelchev opened a new pull request, #10036: IGNITE-17005 Implement metrics for a snapshot create operation
NSAmelchev opened a new pull request, #10036:
URL: https://github.com/apache/ignite/pull/10036
The following metrics implemented:
**CurrentSnapshotTotalSize** - Estimated size of current cluster snapshot in bytes on this node. The value may grow during snapshotting.
**CurrentSnapshotProcessedSize** - Processed size of current cluster snapshot in bytes on this node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] NSAmelchev commented on a diff in pull request #10036: IGNITE-17005 Implement metrics for a snapshot create operation
Posted by GitBox <gi...@apache.org>.
NSAmelchev commented on code in PR #10036:
URL: https://github.com/apache/ignite/pull/10036#discussion_r882823001
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteClusterSnapshotMetricsTest.java:
##########
@@ -212,6 +226,87 @@ public void testRestoreSnapshotError() throws Exception {
}
}
+ /** @throws Exception If fails. */
+ @Test
+ public void testCreateSnapshotProgress() throws Exception {
+ CacheConfiguration<Integer, Object> ccfg1 = cacheConfig("cache1");
+
+ IgniteEx ignite = startGridsWithCache(DEDICATED_CNT, CACHE_KEYS_RANGE, key -> new Account(key, key), ccfg1);
+
+ MetricRegistry mreg = ignite.context().metric().registry(SNAPSHOT_METRICS);
+
+ LongMetric totalSize = mreg.findMetric("CurrentSnapshotTotalSize");
+ LongMetric processedSize = mreg.findMetric("CurrentSnapshotProcessedSize");
+
+ assertEquals(-1, totalSize.value());
+ assertEquals(-1, processedSize.value());
+
+ // Calculate transfer rate limit.
+ PdsFolderSettings<?> folderSettings = ignite.context().pdsFolderResolver().resolveFolders();
+ File storeWorkDir = new File(folderSettings.persistentStoreRootPath(), folderSettings.folderName());
+
+ long rate = FileUtils.sizeOfDirectory(storeWorkDir) / 5;
+
+ // Limit snapshot transfer rate.
+ DistributedChangeableProperty<Serializable> rateProp =
+ ignite.context().distributedConfiguration().property(SNAPSHOT_TRANSFER_RATE_DMS_KEY);
+
+ rateProp.propagate(rate);
+
+ // Start cluster snapshot.
+ IgniteFuture<Void> fut = ignite.snapshot().createSnapshot(SNAPSHOT_NAME);
+
+ // Run load.
+ IgniteInternalFuture<?> loadFut = GridTestUtils.runAsync(() -> {
+ IgniteCache<Integer, Object> cache = ignite.getOrCreateCache(ccfg1);
+
+ while (!fut.isDone()) {
Review Comment:
This is unnecessary. The snapshot future will be done on node stopping:
`clusterSnpFut.onDone(new NodeStoppingException(SNP_NODE_STOPPING_ERR_MSG));`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] xtern commented on a diff in pull request #10036: IGNITE-17005 Implement metrics for a snapshot create operation
Posted by GitBox <gi...@apache.org>.
xtern commented on code in PR #10036:
URL: https://github.com/apache/ignite/pull/10036#discussion_r882728714
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotFutureTask.java:
##########
@@ -512,9 +523,13 @@ else if (affNode && missed.isEmpty() && cctx.kernalContext().query().moduleEnabl
snpSndr.sendDelta(delta, cacheDirName, pair);
+ long deltaLen = delta.length();
Review Comment:
processedSize.addAndGet(delta.length()) ?
I think assertion can only be thrown during development?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] xtern commented on a diff in pull request #10036: IGNITE-17005 Implement metrics for a snapshot create operation
Posted by GitBox <gi...@apache.org>.
xtern commented on code in PR #10036:
URL: https://github.com/apache/ignite/pull/10036#discussion_r882721198
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotManager.java:
##########
@@ -509,6 +509,24 @@ public static String partDeltaFileName(int partId) {
"The list of names of all snapshots currently saved on the local node with respect to " +
"the configured via IgniteConfiguration snapshot working path.");
+ mreg.register("CurrentSnapshotTotalSize", () -> {
+ SnapshotFutureTask task = currentSnapshotTask();
+
+ if (task == null)
+ return -1;
+
+ return task.totalSize();
Review Comment:
```suggestion
return task == null ? -1 : task.totalSize();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] xtern commented on a diff in pull request #10036: IGNITE-17005 Implement metrics for a snapshot create operation
Posted by GitBox <gi...@apache.org>.
xtern commented on code in PR #10036:
URL: https://github.com/apache/ignite/pull/10036#discussion_r882810124
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteClusterSnapshotMetricsTest.java:
##########
@@ -212,6 +226,87 @@ public void testRestoreSnapshotError() throws Exception {
}
}
+ /** @throws Exception If fails. */
+ @Test
+ public void testCreateSnapshotProgress() throws Exception {
+ CacheConfiguration<Integer, Object> ccfg1 = cacheConfig("cache1");
+
+ IgniteEx ignite = startGridsWithCache(DEDICATED_CNT, CACHE_KEYS_RANGE, key -> new Account(key, key), ccfg1);
+
+ MetricRegistry mreg = ignite.context().metric().registry(SNAPSHOT_METRICS);
+
+ LongMetric totalSize = mreg.findMetric("CurrentSnapshotTotalSize");
+ LongMetric processedSize = mreg.findMetric("CurrentSnapshotProcessedSize");
+
+ assertEquals(-1, totalSize.value());
+ assertEquals(-1, processedSize.value());
+
+ // Calculate transfer rate limit.
+ PdsFolderSettings<?> folderSettings = ignite.context().pdsFolderResolver().resolveFolders();
+ File storeWorkDir = new File(folderSettings.persistentStoreRootPath(), folderSettings.folderName());
+
+ long rate = FileUtils.sizeOfDirectory(storeWorkDir) / 5;
+
+ // Limit snapshot transfer rate.
+ DistributedChangeableProperty<Serializable> rateProp =
+ ignite.context().distributedConfiguration().property(SNAPSHOT_TRANSFER_RATE_DMS_KEY);
+
+ rateProp.propagate(rate);
+
+ // Start cluster snapshot.
+ IgniteFuture<Void> fut = ignite.snapshot().createSnapshot(SNAPSHOT_NAME);
+
+ // Run load.
+ IgniteInternalFuture<?> loadFut = GridTestUtils.runAsync(() -> {
+ IgniteCache<Integer, Object> cache = ignite.getOrCreateCache(ccfg1);
+
+ while (!fut.isDone()) {
Review Comment:
|| Thread.currentThread().isInterrupted() ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] xtern commented on a diff in pull request #10036: IGNITE-17005 Implement metrics for a snapshot create operation
Posted by GitBox <gi...@apache.org>.
xtern commented on code in PR #10036:
URL: https://github.com/apache/ignite/pull/10036#discussion_r882723383
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotFutureTask.java:
##########
@@ -141,6 +142,12 @@ class SnapshotFutureTask extends AbstractSnapshotFutureTask<Set<GroupPartitionId
/** Flag indicates that task already scheduled on checkpoint. */
private final AtomicBoolean started = new AtomicBoolean();
+ /** Estimated snapshot size in bytes. The value may grow during snapshotting. */
Review Comment:
```suggestion
/** Estimated snapshot size in bytes. The value may grow during snapshot creation. */
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] NSAmelchev merged pull request #10036: IGNITE-17005 Implement metrics for a snapshot create operation
Posted by GitBox <gi...@apache.org>.
NSAmelchev merged PR #10036:
URL: https://github.com/apache/ignite/pull/10036
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org