You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@ignite.apache.org by "nizhikov (via GitHub)" <gi...@apache.org> on 2023/09/22 07:59:18 UTC

[GitHub] [ignite] nizhikov opened a new pull request, #10950: IGNITE-19950 Cache dumps implemented (#10936)

nizhikov opened a new pull request, #10950:
URL: https://github.com/apache/ignite/pull/10950

   Thank you for submitting the pull request to the Apache Ignite.
   
   In order to streamline the review of the contribution 
   we ask you to ensure the following steps have been taken:
   
   ### The Contribution Checklist
   - [ ] There is a single JIRA ticket related to the pull request. 
   - [ ] The web-link to the pull request is attached to the JIRA ticket.
   - [ ] The JIRA ticket has the _Patch Available_ state.
   - [ ] The pull request body describes changes that have been made. 
   The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
   - [ ] The pull request title is treated as the final commit message. 
   The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
   - [ ] A reviewer has been mentioned through the JIRA comments 
   (see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers)) 
   - [ ] The pull request has been checked by the Teamcity Bot and 
   the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
   
   ### Notes
   - [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
   - [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
   - [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
   - [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
   
   If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349865376


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();

Review Comment:
   For what reason? Can you, please, explain, what is wrong with current approach and how memory allocator will make things better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1750227559

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [71 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349862953


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /**
+     * Kernal context for each node in dump.

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "petrov-mg (via GitHub)" <gi...@apache.org>.

petrov-mg commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349665333


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param keepBinary If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}.

Review Comment:
   Parameter javadocs are missing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [ignite] sonarcloud[bot] commented on pull request #10950: [WIP] IGNITE-19950 Cache dumps implemented (#10936)

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1730992850

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [48 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [ignite] sonarcloud[bot] commented on pull request #10950: [WIP] IGNITE-19950 Cache dumps implemented (#10936)

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1735202646

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [59 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [ignite] sonarcloud[bot] commented on pull request #10950: [WIP] IGNITE-19950 Cache dumps implemented (#10936)

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1740649473

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [59 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349865121


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected synchronized CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */
+        final Map<Integer, Set<KeyCacheObject>> changed;
+
+        /** Count of entries changed during dump creation. */
+        LongAdder changedCnt = new LongAdder();
+
+        /** Partition dump file. Lazily initialized to prevent creation files for empty partitions. */
+        final FileIO file;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        @Nullable final GridCacheVersion startVer;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        final GridCacheVersion isolatedStreamerVer;
+
+        /** Topology Version. */
+        private final AffinityTopologyVersion topVer;
+
+        /** Partition serializer. */
+        private final DumpEntrySerializer serdes;
+
+        /** If {@code true} context is closed. */
+        volatile boolean closed;
+
+        /** Count of writers. When count becomes {@code 0} context must be closed. */
+        private final AtomicInteger writers = new AtomicInteger(1); // Iterator writing entries to this context, by default.

Review Comment:
   Yes. 
   Javadoc rewritten to be clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349865448


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();

Review Comment:
   For what reason? Can you, please, explain, what is wrong with current approach and how memory allocator will make things better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1750167360

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [71 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [ignite] sonarcloud[bot] commented on pull request #10950: [WIP] IGNITE-19950 Cache dumps implemented (#10936)

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1735156101

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [59 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov merged PR #10950:
URL: https://github.com/apache/ignite/pull/10950


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [ignite] sonarcloud[bot] commented on pull request #10950: [WIP] IGNITE-19950 Cache dumps implemented (#10936)

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1733470414

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [48 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349919500


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();

Review Comment:
   https://issues.apache.org/jira/browse/IGNITE-20594?jql=labels%20%3D%20IEP-109 created after private discussion.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();

Review Comment:
   https://issues.apache.org/jira/browse/IGNITE-20594?jql=labels%20%3D%20IEP-109 created after private discussion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1752470456

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [74 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [WIP] IGNITE-19950 Cache dumps implemented (#10936) [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1743104949

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [59 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1752364880

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [74 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349861579


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */
+        final Map<Integer, Set<KeyCacheObject>> changed;
+
+        /** Count of entries changed during dump creation. */
+        LongAdder changedCnt = new LongAdder();
+
+        /** Partition dump file. Lazily initialized to prevent creation files for empty partitions. */
+        final FileIO file;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        @Nullable final GridCacheVersion startVer;
+
+        /** Last version on time of dump start. Can be used only for primary. */

Review Comment:
   Good catch, thanks. Javadoc extended.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1752379559

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [74 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349862744


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */
+        final Map<Integer, Set<KeyCacheObject>> changed;
+
+        /** Count of entries changed during dump creation. */
+        LongAdder changedCnt = new LongAdder();
+
+        /** Partition dump file. Lazily initialized to prevent creation files for empty partitions. */
+        final FileIO file;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        @Nullable final GridCacheVersion startVer;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        final GridCacheVersion isolatedStreamerVer;
+
+        /** Topology Version. */
+        private final AffinityTopologyVersion topVer;
+
+        /** Partition serializer. */
+        private final DumpEntrySerializer serdes;

Review Comment:
   renamed to `serializer`.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349863307


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java:
##########
@@ -694,4 +694,22 @@ private void setField(IgniteEx kernal, String name, Object val) throws NoSuchFie
     @Override public Executor getAsyncContinuationExecutor() {
         return null;
     }
+
+    /**
+     * @param kctx Kernal context.
+     * @throws IgniteCheckedException In case of any error.
+     */
+    public static void startAll(GridKernalContext kctx) throws IgniteCheckedException {
+        for (GridComponent comp : kctx)
+            comp.start();
+    }
+
+    /**
+     * @param kctx Kernal context.
+     * @throws IgniteCheckedException In case of any error.
+     */
+    public static void closeAll(GridKernalContext kctx) throws IgniteCheckedException {

Review Comment:
   renamed



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java:
##########
@@ -694,4 +694,22 @@ private void setField(IgniteEx kernal, String name, Object val) throws NoSuchFie
     @Override public Executor getAsyncContinuationExecutor() {
         return null;
     }
+
+    /**
+     * @param kctx Kernal context.
+     * @throws IgniteCheckedException In case of any error.
+     */
+    public static void startAll(GridKernalContext kctx) throws IgniteCheckedException {

Review Comment:
   renamed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1752408479

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [74 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1751994864

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [74 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [ignite] sonarcloud[bot] commented on pull request #10950: [WIP] IGNITE-19950 Cache dumps implemented (#10936)

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1737120947

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [59 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [WIP] IGNITE-19950 Cache dumps implemented (#10936) [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1746426712

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [59 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "petrov-mg (via GitHub)" <gi...@apache.org>.

petrov-mg commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349665333


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param keepBinary If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}.

Review Comment:
   Parameter javadocs are missing.



##########
modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java:
##########
@@ -867,4 +850,26 @@ private void ackAsciiLogo() {
     public static String cdcInstanceName(String igniteInstanceName) {
         return "cdc-" + igniteInstanceName;
     }
+
+    /**
+     * @param files Mapping files.
+     * @return Type mapping iterator.
+     */
+    public static Iterator<TypeMapping> typeMappingIterator(File[] files, Predicate<TypeMapping> filter) {

Review Comment:
   Javadoc for the filter parameter is missing.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected synchronized CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */
+        final Map<Integer, Set<KeyCacheObject>> changed;
+
+        /** Count of entries changed during dump creation. */
+        LongAdder changedCnt = new LongAdder();
+
+        /** Partition dump file. Lazily initialized to prevent creation files for empty partitions. */
+        final FileIO file;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        @Nullable final GridCacheVersion startVer;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        final GridCacheVersion isolatedStreamerVer;
+
+        /** Topology Version. */
+        private final AffinityTopologyVersion topVer;
+
+        /** Partition serializer. */
+        private final DumpEntrySerializer serdes;
+
+        /** If {@code true} context is closed. */
+        volatile boolean closed;
+
+        /** Count of writers. When count becomes {@code 0} context must be closed. */
+        private final AtomicInteger writers = new AtomicInteger(1); // Iterator writing entries to this context, by default.
+
+        /**
+         * @param gctx Group context.
+         * @param part Partition id.
+         * @param thLocBufs Thread local buffers.
+         */
+        public PartitionDumpContext(CacheGroupContext gctx, int part, ConcurrentMap<Long, ByteBuffer> thLocBufs) {
+            assert gctx != null;
+
+            try {
+                this.part = part;
+                grp = gctx.groupId();
+                topVer = gctx.topology().lastTopologyChangeVersion();
+
+                startVer = grpPrimaries.get(gctx.groupId()).contains(part) ? gctx.shared().versions().last() : null;
+                isolatedStreamerVer = cctx.versions().isolatedStreamerVersion();
+
+                serdes = new DumpEntrySerializer(thLocBufs);
+                changed = new HashMap<>();
+
+                for (int cache : gctx.cacheIds())
+                    changed.put(cache, new GridConcurrentHashSet<>());
+
+                File dumpFile = new File(groupDirectory(gctx), PART_FILE_PREFIX + part + DUMP_FILE_EXT);
+
+                if (!dumpFile.createNewFile())
+                    throw new IgniteException("Dump file can't be created: " + dumpFile);
+
+                file = ioFactory.create(dumpFile);
+            }
+            catch (IOException | IgniteCheckedException e) {
+                throw new IgniteException(e);
+            }
+        }
+
+        /**
+         * Writes entry that changed while dump creation in progress.
+         * @param cache Cache id.
+         * @param expireTime Expire time.
+         * @param key Key.
+         * @param val Value before change.
+         * @param ver Version before change.
+         */
+        public void writeChanged(int cache, long expireTime, KeyCacheObject key, CacheObject val, GridCacheVersion ver) {
+            String reasonToSkip = null;
+
+            if (closed) // Quick exit. Partition already saved in dump.
+                reasonToSkip = "partition already saved";
+            else {
+                writers.getAndIncrement();
+
+                try {
+                    if (closed) // Partition already saved in dump.
+                        reasonToSkip = "partition already saved";
+                    else if (isAfterStart(ver))
+                        reasonToSkip = "greater version";
+                    else if (!changed.get(cache).add(key)) // Entry changed several time during dump.
+                        reasonToSkip = "changed several times";
+                    else if (val == null)
+                        reasonToSkip = "newly created or already removed"; // Previous value is null. Entry created after dump start, skip.
+                    else {
+                        write(cache, expireTime, key, val);
+
+                        changedCnt.increment();
+                    }
+                }
+                finally {
+                    writers.decrementAndGet();
+                }
+            }
+
+            if (log.isTraceEnabled()) {
+                log.trace("Listener [grp=" + grp +
+                    ", cache=" + cache +
+                    ", part=" + part +
+                    ", key=" + key +
+                    ", written=" + (reasonToSkip == null ? "true" : reasonToSkip) + ']');
+            }
+        }
+
+        /**
+         * Writes entry fetched from partition iterator.
+         *
+         * @param cache Cache id.
+         * @param expireTime Expire time.
+         * @param key Key.
+         * @param val Value.
+         * @param ver Version.
+         * @return {@code True} if entry was written in dump,

Review Comment:
   `,` -> `.`



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();

Review Comment:
   I still believe that it's better to use some kind of pooled memory allocator here. 



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /** Specific consistent id. */
+    private final @Nullable String consistentId;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param keepBinary If {@code true} then keep read entries in binary form.
+     * @param raw If {@code true} then keep read entries in form of {@link KeyCacheObject} and {@link CacheObject}.
+     * @param log Logger.
+     */
+    public Dump(File dumpDir, boolean keepBinary, boolean raw, IgniteLogger log) {
+        this(dumpDir, null, keepBinary, raw, log);
+    }
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param consistentId If specified, read dump data only for specific node.
+     * @param keepBinary If {@code true} then keep read entries in binary form.
+     * @param raw If {@code true} then keep read entries in form of {@link KeyCacheObject} and {@link CacheObject}.
+     * @param log Logger.
+     */
+    public Dump(File dumpDir, @Nullable String consistentId, boolean keepBinary, boolean raw, IgniteLogger log) {

Review Comment:
   The `Dump` class API doesn't seem consistent to me. `#partitions()`, `#metadata()`, `#iterator()` etc methods require a `node` argument, although we have a dedicated constructor to create a node-specific dump.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */
+        final Map<Integer, Set<KeyCacheObject>> changed;
+
+        /** Count of entries changed during dump creation. */
+        LongAdder changedCnt = new LongAdder();
+
+        /** Partition dump file. Lazily initialized to prevent creation files for empty partitions. */
+        final FileIO file;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        @Nullable final GridCacheVersion startVer;
+
+        /** Last version on time of dump start. Can be used only for primary. */

Review Comment:
   It's better to describe what the difference is with "startVer".



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /**
+     * Kernal context for each node in dump.

Review Comment:
   Lets's collapse it into one line.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractCreateSnapshotFutureTask.java:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.internal.IgniteFutureCancelledCheckedException;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition;
+import org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionState;
+import org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointListener;
+import org.apache.ignite.internal.processors.marshaller.MappedName;
+import org.apache.ignite.internal.util.lang.IgniteThrowableRunner;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+
+/**
+ *

Review Comment:
   Let's collapse it into one line or provide a meaningful Javadoc.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();

Review Comment:
   I still believe that it's better to use some kind of pooled memory allocator here. 



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java:
##########
@@ -694,4 +694,22 @@ private void setField(IgniteEx kernal, String name, Object val) throws NoSuchFie
     @Override public Executor getAsyncContinuationExecutor() {
         return null;
     }
+
+    /**
+     * @param kctx Kernal context.
+     * @throws IgniteCheckedException In case of any error.
+     */
+    public static void startAll(GridKernalContext kctx) throws IgniteCheckedException {
+        for (GridComponent comp : kctx)
+            comp.start();
+    }
+
+    /**
+     * @param kctx Kernal context.
+     * @throws IgniteCheckedException In case of any error.
+     */
+    public static void closeAll(GridKernalContext kctx) throws IgniteCheckedException {

Review Comment:
   `closeAll` -> `closeAllComponents`?



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */

Review Comment:
   There are no “Hashes of cache keys” here.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */
+        final Map<Integer, Set<KeyCacheObject>> changed;
+
+        /** Count of entries changed during dump creation. */
+        LongAdder changedCnt = new LongAdder();
+
+        /** Partition dump file. Lazily initialized to prevent creation files for empty partitions. */
+        final FileIO file;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        @Nullable final GridCacheVersion startVer;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        final GridCacheVersion isolatedStreamerVer;
+
+        /** Topology Version. */
+        private final AffinityTopologyVersion topVer;
+
+        /** Partition serializer. */
+        private final DumpEntrySerializer serdes;

Review Comment:
   IMHO, `des` part is confusing when you read this name for the first time.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java:
##########
@@ -694,4 +694,22 @@ private void setField(IgniteEx kernal, String name, Object val) throws NoSuchFie
     @Override public Executor getAsyncContinuationExecutor() {
         return null;
     }
+
+    /**
+     * @param kctx Kernal context.
+     * @throws IgniteCheckedException In case of any error.
+     */
+    public static void startAll(GridKernalContext kctx) throws IgniteCheckedException {

Review Comment:
   `startAll` -> `startAllComponents`?



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected synchronized CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */
+        final Map<Integer, Set<KeyCacheObject>> changed;
+
+        /** Count of entries changed during dump creation. */
+        LongAdder changedCnt = new LongAdder();
+
+        /** Partition dump file. Lazily initialized to prevent creation files for empty partitions. */
+        final FileIO file;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        @Nullable final GridCacheVersion startVer;
+
+        /** Last version on time of dump start. Can be used only for primary. */
+        final GridCacheVersion isolatedStreamerVer;
+
+        /** Topology Version. */
+        private final AffinityTopologyVersion topVer;
+
+        /** Partition serializer. */
+        private final DumpEntrySerializer serdes;
+
+        /** If {@code true} context is closed. */
+        volatile boolean closed;
+
+        /** Count of writers. When count becomes {@code 0} context must be closed. */
+        private final AtomicInteger writers = new AtomicInteger(1); // Iterator writing entries to this context, by default.

Review Comment:
   `// Iterator writing entries to this context, by default.` is this comment relevant?



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.LockSupport;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor;
+import org.apache.ignite.internal.processors.cache.GridCacheContext;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.persistence.CacheDataRow;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.partstate.GroupPartitionId;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractCreateSnapshotFutureTask;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotSender;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.GridConcurrentHashSet;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.lang.GridCloseableIterator;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.cacheDataFilename;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
+
+/**
+ * Task creates cache group dump.
+ * Dump is a consistent snapshot of cache entries.
+ * Directories structure is same as a full snapshot but each partitions saved in "part-0.dump" file.
+ * Files structure is a set of {@link DumpEntry} written one by one.
+ *
+ * @see Dump
+ * @see DumpEntry
+ */
+public class CreateDumpFutureTask extends AbstractCreateSnapshotFutureTask implements DumpEntryChangeListener {
+    /** Dump files name. */
+    public static final String DUMP_FILE_EXT = ".dump";
+
+    /** Root dump directory. */
+    private final File dumpDir;
+
+    /** */
+    private final FileIOFactory ioFactory;
+
+    /**
+     * Dump contextes.
+     * Key is [group_id, partition_id] combined in single long value.
+     *
+     * @see #toLong(int, int)
+     */
+    private final Map<Long, PartitionDumpContext> dumpCtxs = new ConcurrentHashMap<>();
+
+    /** Local node is primary for set of group partitions. */
+    private final Map<Integer, Set<Integer>> grpPrimaries = new ConcurrentHashMap<>();
+
+    /**
+     * Map shared across all instances of {@link PartitionDumpContext} and {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param cctx Cache context.
+     * @param srcNodeId Node id which cause snapshot task creation.
+     * @param reqId Snapshot operation request ID.
+     * @param dumpName Dump name.
+     * @param ioFactory IO factory.
+     * @param snpSndr Snapshot sender.
+     * @param parts Parts to dump.
+     */
+    public CreateDumpFutureTask(
+        GridCacheSharedContext<?, ?> cctx,
+        UUID srcNodeId,
+        UUID reqId,
+        String dumpName,
+        File dumpDir,
+        FileIOFactory ioFactory,
+        SnapshotSender snpSndr,
+        Map<Integer, Set<Integer>> parts
+    ) {
+        super(
+            cctx,
+            srcNodeId,
+            reqId,
+            dumpName,
+            snpSndr,
+            parts
+        );
+
+        this.dumpDir = dumpDir;
+        this.ioFactory = ioFactory;
+    }
+
+    /** {@inheritDoc} */
+    @Override public boolean start() {
+        try {
+            if (log.isInfoEnabled())
+                log.info("Start cache dump [name=" + snpName + ", grps=" + parts.keySet() + ']');
+
+            createDumpLock();
+
+            processPartitions();
+
+            prepare();
+
+            saveSnapshotData();
+        }
+        catch (IgniteCheckedException | IOException e) {
+            acceptException(e);
+        }
+
+        return false; // Don't wait for checkpoint.
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void processPartitions() throws IgniteCheckedException {
+        super.processPartitions();
+
+        processed.values().forEach(parts -> parts.remove(INDEX_PARTITION));
+    }
+
+    /** Prepares all data structures to dump entries. */
+    private void prepare() throws IOException, IgniteCheckedException {
+        for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+            int grp = e.getKey();
+
+            File grpDumpDir = groupDirectory(cctx.cache().cacheGroup(grp));
+
+            if (!grpDumpDir.mkdirs())
+                throw new IgniteCheckedException("Dump directory can't be created: " + grpDumpDir);
+
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            for (GridCacheContext<?, ?> cctx : gctx.caches())
+                cctx.dumpListener(this);
+
+            grpPrimaries.put(
+                grp,
+                gctx.affinity().primaryPartitions(gctx.shared().kernalContext().localNodeId(), gctx.affinity().lastVersion())
+            );
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveCacheConfigs() {
+        return processed.keySet().stream().map(grp -> runAsync(() -> {
+            CacheGroupContext gctx = cctx.cache().cacheGroup(grp);
+
+            File grpDir = groupDirectory(gctx);
+
+            IgniteUtils.ensureDirectory(grpDir, "dump group directory", null);
+
+            for (GridCacheContext<?, ?> cacheCtx : gctx.caches()) {
+                DynamicCacheDescriptor desc = cctx.kernalContext().cache().cacheDescriptor(cacheCtx.cacheId());
+
+                StoredCacheData cacheData = new StoredCacheData(new CacheConfiguration(desc.cacheConfiguration()));
+
+                cacheData.queryEntities(desc.schema().entities());
+                cacheData.sql(desc.sql());
+
+                cctx.cache().configManager().writeCacheData(cacheData, new File(grpDir, cacheDataFilename(cacheData.config())));
+            }
+        })).collect(Collectors.toList());
+    }
+
+    /** {@inheritDoc} */
+    @Override protected List<CompletableFuture<Void>> saveGroup(int grp, Set<Integer> grpParts) {
+        long start = System.currentTimeMillis();
+
+        AtomicLong entriesCnt = new AtomicLong();
+        AtomicLong writtenEntriesCnt = new AtomicLong();
+        AtomicLong changedEntriesCnt = new AtomicLong();
+
+        String name = cctx.cache().cacheGroup(grp).cacheOrGroupName();
+
+        CacheGroupContext gctx = cctx.kernalContext().cache().cacheGroup(grp);
+
+        if (log.isInfoEnabled())
+            log.info("Start group dump [name=" + name + ", id=" + grp + ']');
+
+        List<CompletableFuture<Void>> futs = grpParts.stream().map(part -> runAsync(() -> {
+            long entriesCnt0 = 0;
+            long writtenEntriesCnt0 = 0;
+
+            try (PartitionDumpContext dumpCtx = dumpContext(grp, part)) {
+                try (GridCloseableIterator<CacheDataRow> rows = gctx.offheap().reservedIterator(part, dumpCtx.topVer)) {
+                    if (rows == null)
+                        throw new IgniteCheckedException("Partition missing [part=" + part + ']');
+
+                    while (rows.hasNext()) {
+                        CacheDataRow row = rows.next();
+
+                        assert row.partition() == part;
+
+                        int cache = row.cacheId() == 0 ? grp : row.cacheId();
+
+                        if (dumpCtx.writeForIterator(cache, row.expireTime(), row.key(), row.value(), row.version()))
+                            writtenEntriesCnt0++;
+
+                        entriesCnt0++;
+                    }
+                }
+
+                entriesCnt.addAndGet(entriesCnt0);
+                writtenEntriesCnt.addAndGet(writtenEntriesCnt0);
+                changedEntriesCnt.addAndGet(dumpCtx.changedCnt.intValue());
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Finish group partition dump [name=" + name +
+                        ", id=" + grp +
+                        ", part=" + part +
+                        ", time=" + (System.currentTimeMillis() - start) +
+                        ", iterEntriesCnt=" + entriesCnt +
+                        ", writtenIterEntriesCnt=" + entriesCnt +
+                        ", changedEntriesCnt=" + changedEntriesCnt + ']');
+
+                }
+            }
+        })).collect(Collectors.toList());
+
+        int futsSize = futs.size();
+
+        CompletableFuture.allOf(futs.toArray(new CompletableFuture[futsSize])).whenComplete((res, t) -> {
+            clearDumpListener(gctx);
+
+            if (log.isInfoEnabled()) {
+                log.info("Finish group dump [name=" + name +
+                    ", id=" + grp +
+                    ", time=" + (System.currentTimeMillis() - start) +
+                    ", iterEntriesCnt=" + entriesCnt.get() +
+                    ", writtenIterEntriesCnt=" + writtenEntriesCnt.get() +
+                    ", changedEntriesCnt=" + changedEntriesCnt.get() + ']');
+            }
+        });
+
+        return futs;
+    }
+
+    /** {@inheritDoc} */
+    @Override public void beforeChange(GridCacheContext cctx, KeyCacheObject key, CacheObject val, long expireTime, GridCacheVersion ver) {
+        try {
+            int part = key.partition();
+            int grp = cctx.groupId();
+
+            assert part != -1;
+
+            if (!processed.get(grp).contains(part))
+                return;
+
+            dumpContext(grp, part).writeChanged(cctx.cacheId(), expireTime, key, val, ver);
+        }
+        catch (IgniteException e) {
+            acceptException(e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override protected CompletableFuture<Void> closeAsync() {
+        if (closeFut == null) {
+            dumpCtxs.values().forEach(PartitionDumpContext::close);
+
+            Throwable err0 = err.get();
+
+            Set<GroupPartitionId> taken = new HashSet<>();
+
+            for (Map.Entry<Integer, Set<Integer>> e : processed.entrySet()) {
+                int grp = e.getKey();
+
+                clearDumpListener(cctx.cache().cacheGroup(grp));
+
+                for (Integer part : e.getValue())
+                    taken.add(new GroupPartitionId(grp, part));
+            }
+
+            closeFut = CompletableFuture.runAsync(
+                () -> {
+                    thLocBufs.clear();
+                    onDone(new SnapshotFutureTaskResult(taken, null), err0);
+                },
+                cctx.kernalContext().pools().getSystemExecutorService()
+            );
+        }
+
+        return closeFut;
+    }
+
+    /** */
+    private void clearDumpListener(CacheGroupContext gctx) {
+        for (GridCacheContext<?, ?> cctx : gctx.caches())
+            cctx.dumpListener(null);
+    }
+
+    /** */
+    private void createDumpLock() throws IgniteCheckedException, IOException {
+        File nodeDumpDir = IgniteSnapshotManager.nodeDumpDirectory(dumpDir, cctx);
+
+        if (!nodeDumpDir.mkdirs())
+            throw new IgniteCheckedException("Can't create node dump directory: " + nodeDumpDir.getAbsolutePath());
+
+        File lock = new File(nodeDumpDir, DUMP_LOCK);
+
+        if (!lock.createNewFile())
+            throw new IgniteCheckedException("Lock file can't be created or already exists: " + lock.getAbsolutePath());
+    }
+
+    /** */
+    private PartitionDumpContext dumpContext(int grp, int part) {
+        return dumpCtxs.computeIfAbsent(
+            toLong(grp, part),
+            key -> new PartitionDumpContext(cctx.kernalContext().cache().cacheGroup(grp), part, thLocBufs)
+        );
+    }
+
+    /**
+     * Context of dump single partition.
+     */
+    private class PartitionDumpContext implements Closeable {
+        /** Group id. */
+        final int grp;
+
+        /** Partition id. */
+        final int part;
+
+        /** Hashes of cache keys of entries changed by the user during partition dump. */

Review Comment:
   There are no “Hashes of cache keys” here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.

nizhikov commented on code in PR #10950:
URL: https://github.com/apache/ignite/pull/10950#discussion_r1349871367


##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /** Specific consistent id. */
+    private final @Nullable String consistentId;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param keepBinary If {@code true} then keep read entries in binary form.
+     * @param raw If {@code true} then keep read entries in form of {@link KeyCacheObject} and {@link CacheObject}.
+     * @param log Logger.
+     */
+    public Dump(File dumpDir, boolean keepBinary, boolean raw, IgniteLogger log) {
+        this(dumpDir, null, keepBinary, raw, log);
+    }
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param consistentId If specified, read dump data only for specific node.
+     * @param keepBinary If {@code true} then keep read entries in binary form.
+     * @param raw If {@code true} then keep read entries in form of {@link KeyCacheObject} and {@link CacheObject}.
+     * @param log Logger.
+     */
+    public Dump(File dumpDir, @Nullable String consistentId, boolean keepBinary, boolean raw, IgniteLogger log) {

Review Comment:
   There are two scenario when we analyzing dump content:
   
   1. User exploring dump content with `DumpReader`. So `Dump` provide a way to consume all data saved in dump.
   
   2. Ignite checks dump consistency during creation. In this case we don't want to know or even read data belong to other nodes, because, other node can be in the middle of dump creation. That's why we want to have the way to filter out all node except one in constructor.
   
   Node, that `Dump` is internal class, so it's not an `API`.



##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.binary.BinaryObject;
+import org.apache.ignite.binary.BinaryType;
+import org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.dump.DumpEntry;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.StoredCacheData;
+import org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.internal.A;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.marshaller.MarshallerUtils;
+import org.apache.ignite.marshaller.jdk.JdkMarshaller;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
+import static org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNAPSHOT_METAFILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.DUMP_FILE_EXT;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.closeAll;
+import static org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext.startAll;
+
+/**
+ * This class provides ability to work with saved cache dump.
+ */
+public class Dump implements AutoCloseable {
+    /** Snapshot meta. */
+    private final List<SnapshotMetadata> metadata;
+
+    /** Dump directory. */
+    private final File dumpDir;
+
+    /** Specific consistent id. */
+    private final @Nullable String consistentId;
+
+    /**
+     * Kernal context for each node in dump.
+     */
+    private final GridKernalContext cctx;
+
+    /** If {@code true} then return data in form of {@link BinaryObject}. */
+    private final boolean keepBinary;
+
+    /** If {@code true} then don't deserialize {@link KeyCacheObject} and {@link CacheObject}. */
+    private final boolean raw;
+
+    /**
+     * Map shared across all instances of {@link DumpEntrySerializer}.
+     * We use per thread buffer because number of threads is fewer then number of partitions.
+     * Regular count of partitions is {@link RendezvousAffinityFunction#DFLT_PARTITION_COUNT}
+     * and thread is {@link IgniteConfiguration#DFLT_PUBLIC_THREAD_CNT} whic is significantly less.
+     */
+    private final ConcurrentMap<Long, ByteBuffer> thLocBufs = new ConcurrentHashMap<>();
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param keepBinary If {@code true} then keep read entries in binary form.
+     * @param raw If {@code true} then keep read entries in form of {@link KeyCacheObject} and {@link CacheObject}.
+     * @param log Logger.
+     */
+    public Dump(File dumpDir, boolean keepBinary, boolean raw, IgniteLogger log) {
+        this(dumpDir, null, keepBinary, raw, log);
+    }
+
+    /**
+     * @param dumpDir Dump directory.
+     * @param consistentId If specified, read dump data only for specific node.
+     * @param keepBinary If {@code true} then keep read entries in binary form.
+     * @param raw If {@code true} then keep read entries in form of {@link KeyCacheObject} and {@link CacheObject}.
+     * @param log Logger.
+     */
+    public Dump(File dumpDir, @Nullable String consistentId, boolean keepBinary, boolean raw, IgniteLogger log) {

Review Comment:
   There are two scenario when we analyzing dump content:
   
   1. User exploring dump content with `DumpReader`. So `Dump` provide a way to consume all data saved in dump.
   
   2. Ignite checks dump consistency during creation. In this case we don't want to know or even read data belong to other nodes, because, other node can be in the middle of dump creation. That's why we want to have the way to filter out all node except one in constructor.
   
   Note, that `Dump` is internal class, so it's not an `API`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] IGNITE-19950 Cache dumps implemented [ignite]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10950:
URL: https://github.com/apache/ignite/pull/10950#issuecomment-1752737343

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=10950)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=10950&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL) [74 Code Smells](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=10950&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=10950&metric=new_duplicated_lines_density&view=list)
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org