You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/10/29 14:22:13 UTC
[jira] [Commented] (IGNITE-10048) Bounded iteration in standalone
WAL iterator with compaction enabled may skip records
[ https://issues.apache.org/jira/browse/IGNITE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667277#comment-16667277 ]
ASF GitHub Bot commented on IGNITE-10048:
-----------------------------------------
GitHub user glukos opened a pull request:
https://github.com/apache/ignite/pull/5197
IGNITE-10048 Bounded iteration in standalone WAL iterator with compac…
…tion enabled may skip records
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gridgain/apache-ignite ignite-10048
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/ignite/pull/5197.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5197
----
commit 8185c620a57bb6869b2068eee2ba1a7fec0d7170
Author: Ivan Rakov <ir...@...>
Date: 2018-10-29T14:17:29Z
IGNITE-10048 Bounded iteration in standalone WAL iterator with compaction enabled may skip records
----
> Bounded iteration in standalone WAL iterator with compaction enabled may skip records
> -------------------------------------------------------------------------------------
>
> Key: IGNITE-10048
> URL: https://issues.apache.org/jira/browse/IGNITE-10048
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ivan Rakov
> Priority: Major
> Fix For: 2.8
>
>
> Bounded iteration with non-zero start/end offsets may skip some records in intermediate segments. Reproducer (wal compaction should be enabled):
> {noformat}
> /**
> *
> */
> public void testBoundedIterationOverSeveralSegments() throws Exception {
> walCompactionEnabled = true;
> IgniteEx ig = (IgniteEx)startGrid();
> String archiveWalDir = getArchiveWalDirPath(ig);
> ig.cluster().active(true);
> IgniteCache<Object, Object> cache = ig.getOrCreateCache(
> new CacheConfiguration<>().setName("c-n").setAffinity(new RendezvousAffinityFunction(false, 32)));
> IgniteCacheDatabaseSharedManager sharedMgr = ig.context().cache().context().database();
> IgniteWriteAheadLogManager walMgr = ig.context().cache().context().wal();
> WALPointer fromPtr = null;
> int recordsCnt = WAL_SEGMENT_SIZE / 8 /* record size */ * 5;
> for (int i = 0; i < recordsCnt; i++) {
> WALPointer ptr = walMgr.log(new PartitionDestroyRecord(i, i));
> if (i == 100)
> fromPtr = ptr;
> }
> assertNotNull(fromPtr);
> cache.put(1, 1);
> forceCheckpoint();
> // Generate WAL segments for filling WAL archive folder.
> for (int i = 0; i < 2 * ig.configuration().getDataStorageConfiguration().getWalSegments(); i++) {
> sharedMgr.checkpointReadLock();
> try {
> walMgr.log(new SnapshotRecord(i, false), RolloverType.NEXT_SEGMENT);
> }
> finally {
> sharedMgr.checkpointReadUnlock();
> }
> }
> cache.put(2, 2);
> forceCheckpoint();
> U.sleep(5000);
> stopGrid();
> WALIterator it = new IgniteWalIteratorFactory(log)
> .iterator(new IteratorParametersBuilder().from((FileWALPointer)fromPtr).filesOrDirs(archiveWalDir));
> TreeSet<Integer> foundCounters = new TreeSet<>();
> it.forEach(x -> {
> WALRecord rec = x.get2();
> if (rec instanceof PartitionDestroyRecord)
> foundCounters.add(((WalRecordCacheGroupAware)rec).groupId());
> });
> assertEquals(new Integer(100), foundCounters.first());
> assertEquals(new Integer(recordsCnt - 1), foundCounters.last());
> assertEquals(recordsCnt - 100, foundCounters.size());
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)