You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2022/07/04 18:11:35 UTC
[GitHub] [ignite-3] tkalkirill opened a new pull request, #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
tkalkirill opened a new pull request, #920:
URL: https://github.com/apache/ignite-3/pull/920
https://issues.apache.org/jira/browse/IGNITE-17295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924404080
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -250,7 +215,7 @@ private CheckpointDirtyPages owner() {
}
private boolean needsNextRegion() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.length;
Review Comment:
I already explained myself. `dirtyPages.get(regionIndex).dirtyPages` - this extra lookup is unnecessary. You're calculating the same object every time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov merged pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov merged PR #920:
URL: https://github.com/apache/ignite-3/pull/920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924469348
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -182,23 +198,36 @@ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws Ign
* @param pageBuf Page buffer to read into.
* @param checkCrc Check CRC on page.
* @param keepCrc By default reading zeroes CRC which was on file, but you can keep it in pageBuf if set keepCrc.
+ * @param checkPageOffsetLogically Check page offset by {@link #allocatedBytes} or {@link #size}.
* @return {@code true} if page has been read successfully, {@code false} if page hasn't been written yet.
* @throws IgniteInternalCheckedException If reading failed (IO error occurred).
*/
- private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean keepCrc) throws IgniteInternalCheckedException {
+ private boolean read0(
+ long pageId,
+ ByteBuffer pageBuf,
+ boolean checkCrc,
+ boolean keepCrc,
+ boolean checkPageOffsetLogically
+ ) throws IgniteInternalCheckedException {
ensure();
try {
- long off = pageOffset(pageId);
-
assert pageBuf.capacity() == pageSize : pageBuf.capacity();
assert pageBuf.remaining() == pageSize : pageBuf.remaining();
assert pageBuf.position() == 0 : pageBuf.position();
assert pageBuf.order() == nativeOrder() : pageBuf.order();
- assert off <= allocatedBytes() : "calculatedOffset=" + off
- + ", allocated=" + allocatedBytes() + ", headerSize=" + headerSize + ", filePath=" + filePath;
- int n = readWithFailover(pageBuf, off);
+ long pageOff = pageOffset(pageId);
+
+ if (checkPageOffsetLogically) {
+ assert pageOff <= allocatedBytes() : "calculatedOffset=" + pageOff
Review Comment:
I analyzed the code and realized that the check should be left as is, we should not catch an error if the page has not yet been written.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924797463
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -232,8 +230,6 @@ private boolean read0(
// If page was not written yet, nothing to read.
if (n < 0) {
pageBuf.put(new byte[pageBuf.remaining()]);
Review Comment:
I tried to analyze the code associated with this place and it seems that there may be a moment when we want to write the page to disk, but we only wrote it to a temporary buffer for writing it, and another thread wants to read it.
I would not touch this place yet, maybe come back to it later, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921225536
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/PartitionFilePageStore.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * File page store with partition meta.
+ */
+public class PartitionFilePageStore implements PageStore {
+ private final FilePageStore filePageStore;
+
+ private final PartitionMeta partitionMeta;
+
+ /**
+ * Constructor.
+ *
+ * @param filePageStore File page store.
+ * @param partitionMeta Partition meta.
+ */
+ public PartitionFilePageStore(FilePageStore filePageStore, PartitionMeta partitionMeta) {
+ this.filePageStore = filePageStore;
+ this.partitionMeta = partitionMeta;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void addWriteListener(PageWriteListener listener) {
+ filePageStore.addWriteListener(listener);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void removeWriteListener(PageWriteListener listener) {
+ filePageStore.removeWriteListener(listener);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void stop(boolean clean) throws IgniteInternalCheckedException {
+ filePageStore.stop(clean);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public long allocatePage() throws IgniteInternalCheckedException {
+ return filePageStore.allocatePage();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public int pages() {
+ return filePageStore.pages();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws IgniteInternalCheckedException {
+ return filePageStore.read(pageId, pageBuf, keepCrc);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void write(long pageId, ByteBuffer pageBuf, int tag, boolean calculateCrc) throws IgniteInternalCheckedException {
+ filePageStore.write(pageId, pageBuf, tag, calculateCrc);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void sync() throws IgniteInternalCheckedException {
+ filePageStore.sync();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean exists() {
+ return filePageStore.exists();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void ensure() throws IgniteInternalCheckedException {
+ filePageStore.ensure();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void close() throws IOException {
+ filePageStore.close();
+ }
+
+ /**
+ * Returns partition meta.
+ */
+ public PartitionMeta meta() {
Review Comment:
I'll change it later.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/PartitionFilePageStoreFactory.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link PartitionFilePageStore}.
+ */
+class PartitionFilePageStoreFactory {
+ /** Latest file page store version. */
+ public final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ private PageIoRegistry ioRegistry;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param ioRegistry Page IO registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionFilePageStoreFactory(
+ FileIoFactory fileIoFactory,
+ PageIoRegistry ioRegistry,
+ int pageSize
+ ) {
+ this.fileIoFactory = fileIoFactory;
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link PartitionFilePageStore}.
+ *
+ * <p>If the partition file exists, it will read its {@link FilePageStoreHeader header} and {@link PartitionMetaIo meta} ({@code
+ * pageIdx == 0}) to create the {@link PartitionFilePageStore}.
+ *
+ * @param filePath File page store path.
+ * @param readIntoBuffer Direct byte buffer for reading {@link PartitionMetaIo meta} from {@code filePath}.
+ * @return File page store.
+ * @throws IgniteInternalCheckedException if failed
+ */
+ public PartitionFilePageStore createPageStore(
+ Path filePath,
+ ByteBuffer readIntoBuffer
+ ) throws IgniteInternalCheckedException {
+ assert readIntoBuffer.remaining() == pageSize : "Expected pageSize=" + pageSize + ", bufferSize=" + readIntoBuffer.remaining();
+
+ if (!Files.exists(filePath)) {
+ return createPageStore(filePath, new FilePageStoreHeader(latestVersion, pageSize), newPartitionMeta());
+ }
+
+ try (FileIo fileIo = fileIoFactory.create(filePath)) {
+ FilePageStoreHeader header = readHeader(fileIo);
+
+ if (header == null) {
+ header = new FilePageStoreHeader(latestVersion, pageSize);
+ }
+
+ PartitionMeta meta = readPartitionMeta(fileIo, header.headerSize(), readIntoBuffer);
+
+ return createPageStore(filePath, header, meta);
+ } catch (IOException e) {
+ throw new IgniteInternalCheckedException("Error while creating file page store [file=" + filePath + "]:", e);
+ }
+ }
+
+ private PartitionFilePageStore createPageStore(
+ Path filePath,
+ FilePageStoreHeader header,
+ PartitionMeta partitionMeta
+ ) throws IgniteInternalCheckedException {
+ if (header.version() == FilePageStore.VERSION_1) {
+ return new PartitionFilePageStore(
+ new FilePageStore(
+ header.version(),
+ header.pageSize(),
+ header.headerSize(),
+ partitionMeta.pageCount(),
+ filePath,
+ fileIoFactory
+ ),
+ partitionMeta
+ );
+ }
+
+ throw new IgniteInternalCheckedException(String.format(
+ "Unknown version of file page store [version=%s, file=%s]",
+ header.version(),
+ filePath
+ ));
+ }
+
+ private PartitionMeta readPartitionMeta(FileIo fileIo, int headerSize, ByteBuffer readIntoBuffer) throws IOException {
+ if (fileIo.size() <= headerSize) {
+ return newPartitionMeta();
+ }
+
+ try {
+ fileIo.readFully(readIntoBuffer, headerSize);
+
+ long pageAddr = bufferAddress(readIntoBuffer);
+
+ if (PartitionMetaIo.getType(pageAddr) <= 0) {
+ return newPartitionMeta();
+ }
+
+ PartitionMetaIo partitionMetaIo = ioRegistry.resolve(pageAddr);
+
+ return new PartitionMeta(
Review Comment:
I'll change it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921289031
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PageMemoryStorageIoModule.java:
##########
@@ -38,8 +37,7 @@ public Collection<IoVersions<?>> ioVersions() {
TableMetaIo.VERSIONS,
TableInnerIo.VERSIONS,
TableLeafIo.VERSIONS,
- TableDataIo.VERSIONS,
- PartitionMetaIo.VERSIONS
+ TableDataIo.VERSIONS
Review Comment:
We should assign a very clear responsibility to each module and finally put everything where it belongs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921933526
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -161,109 +164,24 @@ public CheckpointDirtyPagesQueue toQueue() {
return null;
}
- List<FullPageId> pageIds = dirtyPages.get(regionIndex).getValue();
+ FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923232427
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
+ Loggers.forClass(FilePageStoreManager.class),
igniteInstanceName,
storagePath,
fileIoFactory,
- ioRegistry,
pageSize
);
- partitionFilePageStoreManager.start();
+ filePageStoreManager.start();
} catch (IgniteInternalCheckedException e) {
throw new StorageException("Error starting file page store manager", e);
}
+ partitionMetaManager = new PartitionMetaManager(ioRegistry, pageSize);
Review Comment:
Configuration of what and from where?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924387879
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,27 +17,29 @@
package org.apache.ignite.internal.pagememory.persistence;
+import static org.apache.ignite.internal.pagememory.PageIdAllocator.FLAG_AUX;
+import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
+
import java.lang.invoke.MethodHandles;
import java.lang.invoke.VarHandle;
import java.util.UUID;
-import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
import org.apache.ignite.internal.tostring.S;
import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
- private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
- PartitionMeta.class,
- "pageCount"
- );
+ private static final VarHandle PAGE_COUNT;
Review Comment:
Are you serious?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923168659
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
Review Comment:
Okay, what name should I use? I don’t know yet whether both methods will be needed or only one, do you insist on deleting the method for the general case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924393933
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/Checkpointer.java:
##########
@@ -381,14 +382,13 @@ boolean writePages(
IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds = checkpointDirtyPages.toDirtyPageIdQueue();
- IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds =
- checkpointDirtyPages.toDirtyPartitionIdQueue();
+ Set<GroupPartitionId> savedPartitionMetas = ConcurrentHashMap.newKeySet();
Review Comment:
If each writer has his own set, then we will write the meta several times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921143950
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
}
/**
- * Returns a queue of dirty pages to be written to a checkpoint.
+ * Returns a queue of dirty page IDs to be written to a checkpoint.
*/
- public CheckpointDirtyPagesQueue toQueue() {
- return new CheckpointDirtyPagesQueue();
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> toDirtyPageIdQueue() {
+ List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> dirtyPageIds = dirtyPages.stream()
+ .map(pages -> new IgniteBiTuple<>(pages.pageMemory, pages.dirtyPages.pageIds))
+ .collect(toList());
+
+ return new IgniteConcurrentMultiPairQueue<>(dirtyPageIds);
Review Comment:
Discussed personally, no need to change.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
}
/**
- * Returns a queue of dirty pages to be written to a checkpoint.
+ * Returns a queue of dirty page IDs to be written to a checkpoint.
*/
- public CheckpointDirtyPagesQueue toQueue() {
- return new CheckpointDirtyPagesQueue();
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> toDirtyPageIdQueue() {
+ List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> dirtyPageIds = dirtyPages.stream()
+ .map(pages -> new IgniteBiTuple<>(pages.pageMemory, pages.dirtyPages.pageIds))
+ .collect(toList());
+
+ return new IgniteConcurrentMultiPairQueue<>(dirtyPageIds);
}
/**
- * Looks for dirty page views for a specific group and partition.
+ * Returns a queue of dirty partition IDs to be modified to a checkpoint.
+ */
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> toDirtyPartitionIdQueue() {
Review Comment:
Discussed personally, no need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921210044
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
+
+ io.setTreeRootPageId(pageAddr, meta.treeRootPageId());
+ io.setReuseListRootPageId(pageAddr, meta.reuseListRootPageId());
+ io.setPageCount(pageAddr, meta.pageCount());
+
+ FullPageId fullPageId = new FullPageId(partitionMetaPageId, partitionId.getGroupId());
+
+ PageStore store = pageWriter.write(
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921207427
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
Review Comment:
Try fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921924127
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -59,7 +66,10 @@ public class CheckpointPagesWriter implements Runnable {
* <p>Overall pages to write may be greater than this queue, since it may be necessary to retire write some pages due to unsuccessful
* page write lock acquisition
*/
- private final CheckpointDirtyPagesQueue writePageIds;
+ private final IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds;
+
+ /** Queue of dirty partition IDs to update {@link PartitionMetaIo} under this task. */
Review Comment:
Yes, this is better
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921143503
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921145149
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -87,57 +103,66 @@ public class CheckpointPagesWriter implements Runnable {
*
* @param tracker Checkpoint metrics tracker.
* @param writePageIds Queue of dirty page IDs to write.
+ * @param updatePartitionIds Queue of dirty partition IDs to update.
* @param updStores Updating storage.
* @param doneFut Done future.
* @param beforePageWrite Action to be performed before every page write.
* @param log Logger.
* @param threadBuf Thread local byte buffer.
* @param checkpointProgress Checkpoint progress.
* @param pageWriter File page store manager.
+ * @param ioRegistry Page IO registry.
+ * @param partitionFilePageStoreManager Partition file page store manager.
* @param shutdownNow Shutdown supplier.
*/
CheckpointPagesWriter(
IgniteLogger log,
CheckpointMetricsTracker tracker,
- CheckpointDirtyPagesQueue writePageIds,
+ IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds,
+ IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds,
ConcurrentMap<PageStore, LongAdder> updStores,
CompletableFuture<?> doneFut,
Runnable beforePageWrite,
ThreadLocal<ByteBuffer> threadBuf,
CheckpointProgressImpl checkpointProgress,
CheckpointPageWriter pageWriter,
+ PageIoRegistry ioRegistry,
+ PartitionFilePageStoreManager partitionFilePageStoreManager,
BooleanSupplier shutdownNow
) {
this.log = log;
this.tracker = tracker;
this.writePageIds = writePageIds;
+ this.updatePartitionIds = updatePartitionIds;
this.updStores = updStores;
this.doneFut = doneFut;
this.beforePageWrite = beforePageWrite;
this.threadBuf = threadBuf;
this.checkpointProgress = checkpointProgress;
this.pageWriter = pageWriter;
+ this.ioRegistry = ioRegistry;
+ this.partitionFilePageStoreManager = partitionFilePageStoreManager;
this.shutdownNow = shutdownNow;
}
/** {@inheritDoc} */
@Override
public void run() {
try {
- CheckpointDirtyPagesQueue pagesToRetry = writePages(writePageIds);
-
- if (pagesToRetry.isEmpty()) {
- doneFut.complete(null);
- } else {
- log.info("Checkpoint pages were not written yet due to "
- + "unsuccessful page write lock acquisition and will be retried [pageCount={}]", pagesToRetry.size());
+ IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> pageIdsToRetry = writePages(writePageIds);
- while (!pagesToRetry.isEmpty()) {
- pagesToRetry = writePages(pagesToRetry);
+ while (!pageIdsToRetry.isEmpty()) {
+ if (log.isInfoEnabled()) {
Review Comment:
Discussed personally, no need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921200619
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -161,109 +164,24 @@ public CheckpointDirtyPagesQueue toQueue() {
return null;
}
- List<FullPageId> pageIds = dirtyPages.get(regionIndex).getValue();
+ FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
Review Comment:
Tried to rename.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923176296
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
Review Comment:
Could this affect performance if we ask the operating system for the file size when reading pages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923222411
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -167,15 +180,15 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
if (fromPosition == pageIds.length - 1 || !equalsByGroupAndPartition(pageIds[fromPosition], pageIds[fromPosition + 1])) {
Review Comment:
Okay, we're afraid of more complex code, removed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924402646
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -240,35 +247,25 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
};
}
- /**
- * Updates the partitions meta page.
- *
- * @throws IgniteInternalCheckedException If failed.
- */
- private void updatePartitions() throws IgniteInternalCheckedException {
- Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+ private void writePartitionMeta(GroupPartitionId partitionId, ByteBuffer buffer) throws IgniteInternalCheckedException {
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
- while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
- GroupPartitionId partitionId = queueResult.getValue();
+ partitionMetaManager.writeMetaToBuffer(partitionId, partitionMetaSnapshot, buffer.rewind());
- ByteBuffer buffer = threadBuf.get();
+ FullPageId fullPageId = new FullPageId(partitionMetaPageId(partitionId.getPartitionId()), partitionId.getGroupId());
- PersistentPageMemory pageMemory = queueResult.getKey();
-
- PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
-
- partitionMetaManager.writeMetaToBuffer(partitionId, pageMemory, partitionMetaSnapshot, buffer.rewind());
+ PageStore store = pageWriter.write(fullPageId, buffer.rewind());
- FullPageId fullPageId = new FullPageId(
- pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId()),
- partitionId.getGroupId()
- );
+ checkpointProgress.writtenPagesCounter().incrementAndGet();
- PageStore store = pageWriter.write(fullPageId, buffer.rewind());
+ updStores.computeIfAbsent(store, k -> new LongAdder()).increment();
+ }
- checkpointProgress.writtenPagesCounter().incrementAndGet();
+ private static boolean isPartitionChange(@Nullable GroupPartitionId partitionId, FullPageId pageId) {
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921923843
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -161,109 +164,24 @@ public CheckpointDirtyPagesQueue toQueue() {
return null;
}
- List<FullPageId> pageIds = dirtyPages.get(regionIndex).getValue();
+ FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
Review Comment:
Why do you spell "is need" when you can use "needs"? Just "needsNextRegion" or "isLastPartinionInRegion" or anything similar will be just fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923195915
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -253,38 +251,21 @@ private void updatePartitions() throws IgniteInternalCheckedException {
while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
GroupPartitionId partitionId = queueResult.getValue();
- PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
- partitionId.getGroupId(),
- partitionId.getPartitionId()
- );
-
ByteBuffer buffer = threadBuf.get();
PersistentPageMemory pageMemory = queueResult.getKey();
- long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
-
- long pageAddr = GridUnsafe.bufferAddress(buffer);
-
- // TODO: IGNITE-16657 Merger will put the page into a partition file.
- PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
Review Comment:
Yes, I understand you, it can only create contention in some situations and use more memory due to the arrangement of the **ConcurrentHashSet**, but in general it is reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923209324
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -159,11 +165,11 @@ public void stop() throws StorageException {
Stream<AutoCloseable> closeRegions = regions.values().stream().map(region -> region::stop);
CheckpointManager checkpointManager = this.checkpointManager;
- PartitionFilePageStoreManager partitionFilePageStoreManager = this.partitionFilePageStoreManager;
+ FilePageStoreManager filePageStoreManager = this.filePageStoreManager;
Stream<AutoCloseable> closeManagers = Stream.of(
checkpointManager == null ? null : (AutoCloseable) checkpointManager::stop,
- partitionFilePageStoreManager == null ? null : (AutoCloseable) partitionFilePageStoreManager::stop
+ filePageStoreManager == null ? null : (AutoCloseable) filePageStoreManager::stop
Review Comment:
Is it so? Can't it be that the **stop** method will be called before the **start** method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921807582
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PageMemoryStorageIoModule.java:
##########
@@ -38,8 +37,7 @@ public Collection<IoVersions<?>> ioVersions() {
TableMetaIo.VERSIONS,
TableInnerIo.VERSIONS,
TableLeafIo.VERSIONS,
- TableDataIo.VERSIONS,
- PartitionMetaIo.VERSIONS
+ TableDataIo.VERSIONS
Review Comment:
Agree, do I need to do it in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923270859
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r925320903
##########
modules/page-memory/src/test/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemoryNoLoadTest.java:
##########
@@ -277,11 +309,47 @@ private static FilePageStoreManager createFilePageStoreManager(Path storagePath)
return new FilePageStoreManager(log, "test", storagePath, new RandomAccessFileIoFactory(), PAGE_SIZE);
}
- private static void initGroupFilePageStores(FilePageStoreManager filePageStoreManager) throws Exception {
- filePageStoreManager.initialize("Test", GRP_ID, PARTITION_ID + 1);
+ private static void initGroupFilePageStores(
+ FilePageStoreManager filePageStoreManager,
+ PartitionMetaManager partitionMetaManager,
+ CheckpointManager checkpointManager
+ ) throws Exception {
+ int partitions = PARTITION_ID + 1;
+
+ filePageStoreManager.initialize("Test", GRP_ID, partitions);
+
+ checkpointManager.checkpointTimeoutLock().checkpointReadLock();
+
+ try {
+ for (int i = 0; i < partitions; i++) {
+ FilePageStore filePageStore = filePageStoreManager.getStore(GRP_ID, i);
+
+ filePageStore.ensure();
+
+ GroupPartitionId groupPartitionId = new GroupPartitionId(GRP_ID, i);
- for (FilePageStore filePageStore : filePageStoreManager.getStores(GRP_ID)) {
- filePageStore.ensure();
+ CheckpointProgress currentProgress = checkpointManager.currentProgress();
+
+ PartitionMeta partitionMeta = partitionMetaManager.readOrCreateMeta(
+ currentProgress == null ? null : currentProgress.id(),
+ groupPartitionId,
+ filePageStore
+ );
+
+ partitionMetaManager.addMeta(groupPartitionId, partitionMeta);
+
+ filePageStore.setPageAllocationListener(pageIdx -> {
+ assert checkpointManager.checkpointTimeoutLock().checkpointLockIsHeldByThread();
+
+ CheckpointProgress curr = checkpointManager.currentProgress();
Review Comment:
fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924392133
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -117,7 +120,7 @@ public class CheckpointPagesWriter implements Runnable {
IgniteLogger log,
CheckpointMetricsTracker tracker,
IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds,
- IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds,
+ Set<GroupPartitionId> savedPartitionMetas,
Review Comment:
So that all writers (this is several threads) do not write the same meta.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923052401
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,33 +17,54 @@
package org.apache.ignite.internal.pagememory.persistence;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import org.apache.ignite.internal.tostring.S;
+import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
+ private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
Review Comment:
You can use VarHandle here as well
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -86,24 +113,89 @@ public int pageCount() {
}
/**
- * Sets count of pages in the partition.
- *
- * @param pageCount Count of pages in the partition.
+ * Increases the number of pages in a partition.
*/
- public void pageCount(int pageCount) {
- this.pageCount = pageCount;
+ public void incrementPageCount(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ PAGE_COUNT_UPDATER.incrementAndGet(this);
}
/**
- * Returns {@code true} if the partition meta was created, {@code false} if it was read from a partition file.
+ * Returns the latest snapshot of the partition meta.
+ *
+ * @param checkpointId Checkpoint ID.
*/
- public boolean isCreated() {
- return created;
+ public PartitionMetaSnapshot metaSnapshot(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ return metaSnapshot;
+ }
+
+ private void updateSnapshot(@Nullable UUID checkpointId) {
Review Comment:
This method deserves a comment as well
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -86,24 +113,89 @@ public int pageCount() {
}
/**
- * Sets count of pages in the partition.
- *
- * @param pageCount Count of pages in the partition.
+ * Increases the number of pages in a partition.
*/
- public void pageCount(int pageCount) {
- this.pageCount = pageCount;
+ public void incrementPageCount(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ PAGE_COUNT_UPDATER.incrementAndGet(this);
}
/**
- * Returns {@code true} if the partition meta was created, {@code false} if it was read from a partition file.
+ * Returns the latest snapshot of the partition meta.
+ *
+ * @param checkpointId Checkpoint ID.
*/
- public boolean isCreated() {
- return created;
+ public PartitionMetaSnapshot metaSnapshot(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ return metaSnapshot;
+ }
+
+ private void updateSnapshot(@Nullable UUID checkpointId) {
+ PartitionMetaSnapshot current = this.metaSnapshot;
+
+ if (current.checkpointId != checkpointId) {
+ META_SNAPSHOT_VH.compareAndSet(this, current, new PartitionMetaSnapshot(checkpointId, this));
Review Comment:
I believe that conventional name for var handle would be "META_SNAPSHOT", please rename it
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,33 +17,54 @@
package org.apache.ignite.internal.pagememory.persistence;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import org.apache.ignite.internal.tostring.S;
+import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
+ private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
+ PartitionMeta.class,
+ "pageCount"
+ );
+
+ private static final VarHandle META_SNAPSHOT_VH;
+
+ static {
+ try {
+ META_SNAPSHOT_VH = MethodHandles.lookup().findVarHandle(PartitionMeta.class, "metaSnapshot", PartitionMetaSnapshot.class);
+ } catch (ReflectiveOperationException e) {
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
private volatile long treeRootPageId;
private volatile long reuseListRootPageId;
private volatile int pageCount;
- private final boolean created;
+ private volatile PartitionMetaSnapshot metaSnapshot;
/**
* Constructor.
*
+ * @param checkpointId Checkpoint ID.
* @param treeRootPageId Tree root page ID.
* @param reuseListRootPageId Reuse list root page ID.
* @param pageCount Count of pages in the partition.
- * @param created {@code True} if the partition meta was created, {@code false} if it was read from a partition file.
*/
- public PartitionMeta(long treeRootPageId, long reuseListRootPageId, int pageCount, boolean created) {
+ public PartitionMeta(@Nullable UUID checkpointId, long treeRootPageId, long reuseListRootPageId, int pageCount) {
Review Comment:
Why is "checkpointId" nullable? It can be null only on initialization, right? We could just use (0, 0) in that case
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
Review Comment:
I don't get it, what's happening here? We need a better comment or even changing this code. Looks like a dangerous workaround
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
Review Comment:
I see that you have not renamed this method. "View" is a very general term
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
+ filePageStore.pages(1);
+ }
+
+ boolean read = filePageStore.read(partitionMetaPageId, buffer, false);
+
+ assert read : filePageStore.filePath();
+
+ return createMeta(ioRegistry.resolve(bufferAddr), bufferAddr, checkpointId);
+ } else {
+ // Creates and writes a partition meta.
+ PartitionMetaIo partitionMetaIo = PartitionMetaIo.VERSIONS.latest();
+ partitionMetaIo.initNewPage(bufferAddr, partitionMetaPageId, pageSize);
+
+ // Because we will now write this page.
+ partitionMetaIo.setPageCount(bufferAddr, 1);
+
+ filePageStore.pages(1);
+
+ filePageStore.write(partitionMetaPageId, buffer.rewind(), true);
+
+ filePageStore.sync();
+
+ return createMeta(partitionMetaIo, bufferAddr, checkpointId);
+ }
+ } finally {
+ freeBuffer(buffer);
+ }
+ }
+
+ /**
+ * Writes the partition meta to the buffer.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param partitionMeta Snapshot of the partition meta.
+ * @param writeToBuffer Direct byte buffer to write partition meta.
+ */
+ public void writeMetaToBuffer(
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ PartitionMetaSnapshot partitionMeta,
+ ByteBuffer writeToBuffer
+ ) {
+ assert writeToBuffer.remaining() == pageSize : writeToBuffer.remaining();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ long pageAddr = bufferAddress(writeToBuffer);
+
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, pageSize);
+
+ io.setTreeRootPageId(pageAddr, partitionMeta.treeRootPageId());
Review Comment:
I already asked you to encapsulate such lists of calls
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
+ filePageStore.pages(1);
+ }
+
+ boolean read = filePageStore.read(partitionMetaPageId, buffer, false);
+
+ assert read : filePageStore.filePath();
+
+ return createMeta(ioRegistry.resolve(bufferAddr), bufferAddr, checkpointId);
+ } else {
+ // Creates and writes a partition meta.
+ PartitionMetaIo partitionMetaIo = PartitionMetaIo.VERSIONS.latest();
+ partitionMetaIo.initNewPage(bufferAddr, partitionMetaPageId, pageSize);
+
+ // Because we will now write this page.
+ partitionMetaIo.setPageCount(bufferAddr, 1);
+
+ filePageStore.pages(1);
+
+ filePageStore.write(partitionMetaPageId, buffer.rewind(), true);
+
+ filePageStore.sync();
+
+ return createMeta(partitionMetaIo, bufferAddr, checkpointId);
+ }
+ } finally {
+ freeBuffer(buffer);
+ }
+ }
+
+ /**
+ * Writes the partition meta to the buffer.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param partitionMeta Snapshot of the partition meta.
+ * @param writeToBuffer Direct byte buffer to write partition meta.
+ */
+ public void writeMetaToBuffer(
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ PartitionMetaSnapshot partitionMeta,
+ ByteBuffer writeToBuffer
+ ) {
+ assert writeToBuffer.remaining() == pageSize : writeToBuffer.remaining();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ long pageAddr = bufferAddress(writeToBuffer);
+
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, pageSize);
+
+ io.setTreeRootPageId(pageAddr, partitionMeta.treeRootPageId());
+ io.setReuseListRootPageId(pageAddr, partitionMeta.reuseListRootPageId());
+ io.setPageCount(pageAddr, partitionMeta.pageCount());
+ }
+
+ private PartitionMeta createMeta(PartitionMetaIo metaIo, long pageAddr, @Nullable UUID checkpointId) {
+ return new PartitionMeta(
Review Comment:
Same here
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
+ filePageStore.pages(1);
+ }
+
+ boolean read = filePageStore.read(partitionMetaPageId, buffer, false);
+
+ assert read : filePageStore.filePath();
+
+ return createMeta(ioRegistry.resolve(bufferAddr), bufferAddr, checkpointId);
+ } else {
+ // Creates and writes a partition meta.
+ PartitionMetaIo partitionMetaIo = PartitionMetaIo.VERSIONS.latest();
+ partitionMetaIo.initNewPage(bufferAddr, partitionMetaPageId, pageSize);
+
+ // Because we will now write this page.
+ partitionMetaIo.setPageCount(bufferAddr, 1);
+
+ filePageStore.pages(1);
+
+ filePageStore.write(partitionMetaPageId, buffer.rewind(), true);
+
+ filePageStore.sync();
+
+ return createMeta(partitionMetaIo, bufferAddr, checkpointId);
+ }
+ } finally {
+ freeBuffer(buffer);
+ }
+ }
+
+ /**
+ * Writes the partition meta to the buffer.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param partitionMeta Snapshot of the partition meta.
+ * @param writeToBuffer Direct byte buffer to write partition meta.
+ */
+ public void writeMetaToBuffer(
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ PartitionMetaSnapshot partitionMeta,
+ ByteBuffer writeToBuffer
+ ) {
+ assert writeToBuffer.remaining() == pageSize : writeToBuffer.remaining();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
Review Comment:
So, the only reason of passing page memory instance is to construct meta page id? Are there helper methods for it somewhere? I'd ask you to remove page memory parameter in such places.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
- if (dirtyPages.isEmpty()) {
- return null;
+ for (int i = 0; i < dirtyPages.size(); i++) {
+ CheckpointDirtyPagesView view = findView(i, grpId, partId);
+
+ if (view != null) {
+ return view;
+ }
}
- FullPageId startPageId = new FullPageId(pageId(partId, (byte) 0, 0), grpId);
- FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
+ return null;
+ }
+ /**
+ * Looks for dirty page IDs views for a specific group and partition.
+ *
+ * @param pageMemory Page memory.
+ * @param grpId Group ID.
+ * @param partId Partition ID.
+ */
+ public @Nullable CheckpointDirtyPagesView findView(PersistentPageMemory pageMemory, int grpId, int partId) {
for (int i = 0; i < dirtyPages.size(); i++) {
- FullPageId[] pageIds = dirtyPages.get(i).dirtyPages.pageIds;
+ if (dirtyPages.get(i).pageMemory == pageMemory) {
+ return findView(i, grpId, partId);
+ }
+ }
- int fromIndex = binarySearch(pageIds, startPageId, DIRTY_PAGE_COMPARATOR);
+ return null;
+ }
- fromIndex = fromIndex >= 0 ? fromIndex : Math.min(pageIds.length - 1, -fromIndex - 1);
+ private @Nullable CheckpointDirtyPagesView findView(int dirtyPagesIdx, int grpId, int partId) {
+ FullPageId startPageId = new FullPageId(pageId(partId, (byte) 0, 0), grpId);
+ FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
- if (!equalsByGroupAndPartition(startPageId, pageIds[fromIndex])) {
- continue;
- }
+ FullPageId[] pageIds = dirtyPages.get(dirtyPagesIdx).dirtyPages.pageIds;
- int toIndex = binarySearch(pageIds, fromIndex, pageIds.length, endPageId, DIRTY_PAGE_COMPARATOR);
+ int fromIndex = binarySearch(pageIds, startPageId, DIRTY_PAGE_COMPARATOR);
- // toIndex cannot be 0 because endPageId is greater than startPageId by DIRTY_PAGE_COMPARATOR.
- toIndex = toIndex > 0 ? toIndex - 1 : -toIndex - 2;
+ fromIndex = fromIndex >= 0 ? fromIndex : Math.min(pageIds.length - 1, -fromIndex - 1);
- return new CheckpointDirtyPagesView(i, fromIndex, toIndex);
+ if (!equalsByGroupAndPartition(startPageId, pageIds[fromIndex])) {
+ return null;
}
- return null;
+ int toIndex = binarySearch(pageIds, fromIndex, pageIds.length, endPageId, DIRTY_PAGE_COMPARATOR);
+
+ // toIndex cannot be 0 because endPageId is greater than startPageId by DIRTY_PAGE_COMPARATOR.
+ toIndex = toIndex > 0 ? toIndex : -toIndex - 1;
Review Comment:
This comment wouldn't be required if you wrote ">= 0". Why does it have to be more complicated?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -229,15 +242,15 @@ public PersistentPageMemory pageMemory() {
* Returns the size of the view.
*/
public int size() {
- return toPosition - fromPosition + 1;
+ return toPosition - fromPosition;
}
private CheckpointDirtyPages owner() {
return CheckpointDirtyPages.this;
}
- private boolean isToPositionLast() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length - 1;
+ private boolean needsNextRegion() {
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
Review Comment:
Why don't we store `dirtyPages.get(regionIndex)` precalculated? No one needs this extra lookup on every access
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -167,15 +180,15 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
if (fromPosition == pageIds.length - 1 || !equalsByGroupAndPartition(pageIds[fromPosition], pageIds[fromPosition + 1])) {
Review Comment:
This is weirdly specific case, why did you add this code? Does it break with regular binary searches?
Again, this makes code more complicated. That's why I'm asking. If we can just remove it then let's remove it
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -156,10 +138,18 @@ public void stop(boolean clean) throws IgniteInternalCheckedException {
/** {@inheritDoc} */
@Override
- public long allocatePage() throws IgniteInternalCheckedException {
+ public int allocatePage() throws IgniteInternalCheckedException {
ensure();
- return pageCount.getAndIncrement();
+ int pageIdx = pageCount.getAndIncrement();
+
+ PageAllocationListener listener = this.pageAllocationListener;
+
+ if (listener != null) {
+ listener.onAllocationPage(pageIdx);
Review Comment:
"onAllocationPage" is grammatically incorrect. Please use better names, like "onPageAllocated", for example. Don't ignore proper English in naming
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointWorkflow.java:
##########
@@ -289,31 +282,26 @@ private DataRegionsDirtyPages beginCheckpoint(
Collection<? extends DataRegion<PersistentPageMemory>> dataRegions,
CompletableFuture<?> allowToReplace
) throws IgniteInternalCheckedException {
- Collection<DataRegionDirtyPages<CollectionDirtyPages>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
+ Collection<DataRegionDirtyPages<DirtyPagesCollection>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
Review Comment:
Can I ask you to add assertions that write lock is locked in every method that needs it? Currently it's impossible to understand it by just looking at method names or javadocs
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
+ *
+ * @param filePath File page store path.
+ * @param readIntoBuffer Buffer for reading {@link FilePageStoreHeader header} from {@code filePath}.
Review Comment:
Can you just call it "headerBuffer"?
Phrase like "read-into buffer" is very unusual imo. Usually people use word "output" for this purpose
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
+ *
+ * @param filePath File page store path.
+ * @param readIntoBuffer Buffer for reading {@link FilePageStoreHeader header} from {@code filePath}.
+ * @return File page store.
+ * @throws IgniteInternalCheckedException if failed
+ */
+ public FilePageStore createPageStore(Path filePath, ByteBuffer readIntoBuffer) throws IgniteInternalCheckedException {
+ assert readIntoBuffer.remaining() == pageSize : readIntoBuffer.remaining();
+
+ if (!Files.exists(filePath)) {
+ return createPageStore(filePath, new FilePageStoreHeader(latestVersion, pageSize));
+ }
+
+ try (FileIo fileIo = fileIoFactory.create(filePath)) {
+ FilePageStoreHeader header = readHeader(fileIo, readIntoBuffer);
+
+ if (header == null) {
+ header = new FilePageStoreHeader(latestVersion, pageSize);
+ }
+
+ return createPageStore(filePath, header);
+ } catch (IOException e) {
+ throw new IgniteInternalCheckedException("Error while creating file page store [file=" + filePath + "]:", e);
Review Comment:
What's up with the colon at the end?
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
Review Comment:
Can these fields be instantiated in constructor? Then they wouldn't have to be nullable
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreHeader.java:
##########
@@ -97,18 +97,19 @@ ByteBuffer toByteBuffer() {
* Reads the header of a file page store.
*
* @param fileIo File page store fileIo.
+ * @param readIntoBuffer Buffer for reading {@link FilePageStoreHeader header} from {@code fileIo}.
Review Comment:
Same here, strange naming
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
+ Loggers.forClass(FilePageStoreManager.class),
igniteInstanceName,
storagePath,
fileIoFactory,
- ioRegistry,
pageSize
);
- partitionFilePageStoreManager.start();
+ filePageStoreManager.start();
} catch (IgniteInternalCheckedException e) {
throw new StorageException("Error starting file page store manager", e);
}
+ partitionMetaManager = new PartitionMetaManager(ioRegistry, pageSize);
Review Comment:
Same here. Constructor initializes object, "start" starts all sub-components. This is how it was intended to work
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -159,11 +165,11 @@ public void stop() throws StorageException {
Stream<AutoCloseable> closeRegions = regions.values().stream().map(region -> region::stop);
CheckpointManager checkpointManager = this.checkpointManager;
- PartitionFilePageStoreManager partitionFilePageStoreManager = this.partitionFilePageStoreManager;
+ FilePageStoreManager filePageStoreManager = this.filePageStoreManager;
Stream<AutoCloseable> closeManagers = Stream.of(
checkpointManager == null ? null : (AutoCloseable) checkpointManager::stop,
- partitionFilePageStoreManager == null ? null : (AutoCloseable) partitionFilePageStoreManager::stop
+ filePageStoreManager == null ? null : (AutoCloseable) filePageStoreManager::stop
Review Comment:
All these null checks won't be required anymore
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
Review Comment:
Why do we need two versions of the method now? Don't we always have a page memory instance when we need a "view"?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
- if (dirtyPages.isEmpty()) {
- return null;
+ for (int i = 0; i < dirtyPages.size(); i++) {
+ CheckpointDirtyPagesView view = findView(i, grpId, partId);
+
+ if (view != null) {
+ return view;
+ }
}
- FullPageId startPageId = new FullPageId(pageId(partId, (byte) 0, 0), grpId);
- FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
+ return null;
+ }
+ /**
+ * Looks for dirty page IDs views for a specific group and partition.
+ *
+ * @param pageMemory Page memory.
+ * @param grpId Group ID.
+ * @param partId Partition ID.
+ */
+ public @Nullable CheckpointDirtyPagesView findView(PersistentPageMemory pageMemory, int grpId, int partId) {
for (int i = 0; i < dirtyPages.size(); i++) {
- FullPageId[] pageIds = dirtyPages.get(i).dirtyPages.pageIds;
+ if (dirtyPages.get(i).pageMemory == pageMemory) {
+ return findView(i, grpId, partId);
+ }
+ }
- int fromIndex = binarySearch(pageIds, startPageId, DIRTY_PAGE_COMPARATOR);
+ return null;
Review Comment:
Technically, this must be impossible, right? Maybe we should add an assertion or something?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -253,38 +251,21 @@ private void updatePartitions() throws IgniteInternalCheckedException {
while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
GroupPartitionId partitionId = queueResult.getValue();
- PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
- partitionId.getGroupId(),
- partitionId.getPartitionId()
- );
-
ByteBuffer buffer = threadBuf.get();
PersistentPageMemory pageMemory = queueResult.getKey();
- long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
-
- long pageAddr = GridUnsafe.bufferAddress(buffer);
-
- // TODO: IGNITE-16657 Merger will put the page into a partition file.
- PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
Review Comment:
Ok, you're not gonna like what I want to say. But, a collection of dirty partition ids is not required anymore once you implemented "async" meta snapshots.
Here's what you can do: for each page you need to write you additionally check if its partition has its meta saved. This can be done with a simple `if (saveMetaPage.put(groupPartitionId))`. Then, if current thread is the one that needs to write meta, you just do it by acquiring meta object the same way as it is in the code right now.
The point is, a regular list of dirty pages already contains all information about dirty partitions.
Please ask questions if I didn't manage to explain myself here.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
Review Comment:
What did you try to say with "an attempt will be reads its"? I don't understand it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923214030
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
Review Comment:
I don't know actually. I wouldn't risk it for every page. Especially considering that in the future file size will not match the logical pages count if you had failed checkpoint.
BTW, I think we should double-check fault tolerance of creating a new partition file. It might not work if we failed to write meta page I think. Also, please check that you didn't forget to fsync file after you write meta page into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923211115
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointWorkflow.java:
##########
@@ -289,31 +282,26 @@ private DataRegionsDirtyPages beginCheckpoint(
Collection<? extends DataRegion<PersistentPageMemory>> dataRegions,
CompletableFuture<?> allowToReplace
) throws IgniteInternalCheckedException {
- Collection<DataRegionDirtyPages<CollectionDirtyPages>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
+ Collection<DataRegionDirtyPages<DirtyPagesCollection>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
Review Comment:
I'll try to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923220731
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
Review Comment:
If you don't like the name, suggest a suitable one:
- getPartitionView - it's about partitions and not dirty page ids, does it really fit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r922198013
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/PartitionFilePageStoreFactory.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link PartitionFilePageStore}.
+ */
+class PartitionFilePageStoreFactory {
+ /** Latest file page store version. */
+ public final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ private PageIoRegistry ioRegistry;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param ioRegistry Page IO registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionFilePageStoreFactory(
+ FileIoFactory fileIoFactory,
+ PageIoRegistry ioRegistry,
+ int pageSize
+ ) {
+ this.fileIoFactory = fileIoFactory;
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link PartitionFilePageStore}.
+ *
+ * <p>If the partition file exists, it will read its {@link FilePageStoreHeader header} and {@link PartitionMetaIo meta} ({@code
+ * pageIdx == 0}) to create the {@link PartitionFilePageStore}.
+ *
+ * @param filePath File page store path.
+ * @param readIntoBuffer Direct byte buffer for reading {@link PartitionMetaIo meta} from {@code filePath}.
+ * @return File page store.
+ * @throws IgniteInternalCheckedException if failed
+ */
+ public PartitionFilePageStore createPageStore(
+ Path filePath,
+ ByteBuffer readIntoBuffer
+ ) throws IgniteInternalCheckedException {
+ assert readIntoBuffer.remaining() == pageSize : "Expected pageSize=" + pageSize + ", bufferSize=" + readIntoBuffer.remaining();
+
+ if (!Files.exists(filePath)) {
+ return createPageStore(filePath, new FilePageStoreHeader(latestVersion, pageSize), newPartitionMeta());
+ }
+
+ try (FileIo fileIo = fileIoFactory.create(filePath)) {
+ FilePageStoreHeader header = readHeader(fileIo);
+
+ if (header == null) {
+ header = new FilePageStoreHeader(latestVersion, pageSize);
+ }
+
+ PartitionMeta meta = readPartitionMeta(fileIo, header.headerSize(), readIntoBuffer);
+
+ return createPageStore(filePath, header, meta);
+ } catch (IOException e) {
+ throw new IgniteInternalCheckedException("Error while creating file page store [file=" + filePath + "]:", e);
+ }
+ }
+
+ private PartitionFilePageStore createPageStore(
+ Path filePath,
+ FilePageStoreHeader header,
+ PartitionMeta partitionMeta
+ ) throws IgniteInternalCheckedException {
+ if (header.version() == FilePageStore.VERSION_1) {
+ return new PartitionFilePageStore(
+ new FilePageStore(
+ header.version(),
+ header.pageSize(),
+ header.headerSize(),
+ partitionMeta.pageCount(),
+ filePath,
+ fileIoFactory
+ ),
+ partitionMeta
+ );
+ }
+
+ throw new IgniteInternalCheckedException(String.format(
+ "Unknown version of file page store [version=%s, file=%s]",
+ header.version(),
+ filePath
+ ));
+ }
+
+ private PartitionMeta readPartitionMeta(FileIo fileIo, int headerSize, ByteBuffer readIntoBuffer) throws IOException {
+ if (fileIo.size() <= headerSize) {
+ return newPartitionMeta();
+ }
+
+ try {
+ fileIo.readFully(readIntoBuffer, headerSize);
+
+ long pageAddr = bufferAddress(readIntoBuffer);
+
+ if (PartitionMetaIo.getType(pageAddr) <= 0) {
+ return newPartitionMeta();
+ }
+
+ PartitionMetaIo partitionMetaIo = ioRegistry.resolve(pageAddr);
+
+ return new PartitionMeta(
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924389571
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -132,11 +150,17 @@ public PartitionMetaSnapshot metaSnapshot(@Nullable UUID checkpointId) {
return metaSnapshot;
}
+ /**
+ * Takes a snapshot of the meta partition if the {@code checkpointId} is different from the {@link #metaSnapshot last snapshot} {@link
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924658018
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -232,8 +230,6 @@ private boolean read0(
// If page was not written yet, nothing to read.
if (n < 0) {
pageBuf.put(new byte[pageBuf.remaining()]);
Review Comment:
Is this even possible? Maybe we should remove this part as well. What's the point of returning an empty page.
I have a hypothesis that this is part of the code of partition initialization from Ignite 2.x that's also obsolete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923122045
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -86,24 +113,89 @@ public int pageCount() {
}
/**
- * Sets count of pages in the partition.
- *
- * @param pageCount Count of pages in the partition.
+ * Increases the number of pages in a partition.
*/
- public void pageCount(int pageCount) {
- this.pageCount = pageCount;
+ public void incrementPageCount(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ PAGE_COUNT_UPDATER.incrementAndGet(this);
}
/**
- * Returns {@code true} if the partition meta was created, {@code false} if it was read from a partition file.
+ * Returns the latest snapshot of the partition meta.
+ *
+ * @param checkpointId Checkpoint ID.
*/
- public boolean isCreated() {
- return created;
+ public PartitionMetaSnapshot metaSnapshot(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ return metaSnapshot;
+ }
+
+ private void updateSnapshot(@Nullable UUID checkpointId) {
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923230955
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -253,38 +251,21 @@ private void updatePartitions() throws IgniteInternalCheckedException {
while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
GroupPartitionId partitionId = queueResult.getValue();
- PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
- partitionId.getGroupId(),
- partitionId.getPartitionId()
- );
-
ByteBuffer buffer = threadBuf.get();
PersistentPageMemory pageMemory = queueResult.getKey();
- long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
-
- long pageAddr = GridUnsafe.bufferAddress(buffer);
-
- // TODO: IGNITE-16657 Merger will put the page into a partition file.
- PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
Review Comment:
Page id consist of more than just grpId and partId.
So objects are problems or not? Above you write that this is not a problem.
Of course put into a set, since not one thread writes to one partition, but many threads write to a partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920950426
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryTableStorage.java:
##########
@@ -81,22 +80,36 @@ public void start() throws StorageException {
protected PersistentPageMemoryPartitionStorage createPartitionStorage(int partId) throws StorageException {
TableView tableView = tableCfg.value();
- FilePageStore partitionFilePageStore = ensurePartitionFilePageStore(tableView, partId);
+ PartitionFilePageStore partitionFilePageStore = ensurePartitionFilePageStore(tableView, partId);
CheckpointTimeoutLock checkpointTimeoutLock = dataRegion.checkpointManager().checkpointTimeoutLock();
checkpointTimeoutLock.checkpointReadLock();
try {
- PartitionMeta partitionMeta = getOrCreatePartitionMeta(tableView, partId, partitionFilePageStore);
+ PersistentPageMemory persistentPageMemory = dataRegion.pageMemory();
- TableFreeList tableFreeList = createTableFreeList(tableView, partId, partitionMeta);
+ int grpId = groupId(tableView);
+
+ PartitionMeta meta = partitionFilePageStore.meta();
+
+ if (meta.isCreated()) {
Review Comment:
Why do we need a "isCreated" flag? Makes absolutely no sense. You can compare stored ids with default values instead.
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryTableStorage.java:
##########
@@ -226,8 +169,8 @@ TableFreeList createTableFreeList(
partId,
dataRegion.pageMemory(),
PageLockListenerNoOp.INSTANCE,
- partitionMeta.reuseListRoot.pageId(),
- partitionMeta.allocated,
+ partitionMeta.reuseListRootPageId(),
+ partitionMeta.isCreated(),
Review Comment:
What's the purpose of this flag? Can we remove it as well?
I'd prefer not having "isCreated" method at all.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/PartitionFilePageStore.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * File page store with partition meta.
+ */
+public class PartitionFilePageStore implements PageStore {
+ private final FilePageStore filePageStore;
+
+ private final PartitionMeta partitionMeta;
+
+ /**
+ * Constructor.
+ *
+ * @param filePageStore File page store.
+ * @param partitionMeta Partition meta.
+ */
+ public PartitionFilePageStore(FilePageStore filePageStore, PartitionMeta partitionMeta) {
+ this.filePageStore = filePageStore;
+ this.partitionMeta = partitionMeta;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void addWriteListener(PageWriteListener listener) {
+ filePageStore.addWriteListener(listener);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void removeWriteListener(PageWriteListener listener) {
+ filePageStore.removeWriteListener(listener);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void stop(boolean clean) throws IgniteInternalCheckedException {
+ filePageStore.stop(clean);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public long allocatePage() throws IgniteInternalCheckedException {
+ return filePageStore.allocatePage();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public int pages() {
+ return filePageStore.pages();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws IgniteInternalCheckedException {
+ return filePageStore.read(pageId, pageBuf, keepCrc);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void write(long pageId, ByteBuffer pageBuf, int tag, boolean calculateCrc) throws IgniteInternalCheckedException {
+ filePageStore.write(pageId, pageBuf, tag, calculateCrc);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void sync() throws IgniteInternalCheckedException {
+ filePageStore.sync();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean exists() {
+ return filePageStore.exists();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void ensure() throws IgniteInternalCheckedException {
+ filePageStore.ensure();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void close() throws IOException {
+ filePageStore.close();
+ }
+
+ /**
+ * Returns partition meta.
+ */
+ public PartitionMeta meta() {
Review Comment:
This method is called at the and of checkpoint, right? How can you be sure that the object is not modified?
I don't see any comments about meta snapshots.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/PartitionFilePageStoreFactory.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link PartitionFilePageStore}.
+ */
+class PartitionFilePageStoreFactory {
+ /** Latest file page store version. */
+ public final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ private PageIoRegistry ioRegistry;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param ioRegistry Page IO registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionFilePageStoreFactory(
+ FileIoFactory fileIoFactory,
+ PageIoRegistry ioRegistry,
+ int pageSize
+ ) {
+ this.fileIoFactory = fileIoFactory;
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link PartitionFilePageStore}.
+ *
+ * <p>If the partition file exists, it will read its {@link FilePageStoreHeader header} and {@link PartitionMetaIo meta} ({@code
+ * pageIdx == 0}) to create the {@link PartitionFilePageStore}.
+ *
+ * @param filePath File page store path.
+ * @param readIntoBuffer Direct byte buffer for reading {@link PartitionMetaIo meta} from {@code filePath}.
+ * @return File page store.
+ * @throws IgniteInternalCheckedException if failed
+ */
+ public PartitionFilePageStore createPageStore(
+ Path filePath,
+ ByteBuffer readIntoBuffer
+ ) throws IgniteInternalCheckedException {
+ assert readIntoBuffer.remaining() == pageSize : "Expected pageSize=" + pageSize + ", bufferSize=" + readIntoBuffer.remaining();
+
+ if (!Files.exists(filePath)) {
+ return createPageStore(filePath, new FilePageStoreHeader(latestVersion, pageSize), newPartitionMeta());
+ }
+
+ try (FileIo fileIo = fileIoFactory.create(filePath)) {
+ FilePageStoreHeader header = readHeader(fileIo);
+
+ if (header == null) {
+ header = new FilePageStoreHeader(latestVersion, pageSize);
+ }
+
+ PartitionMeta meta = readPartitionMeta(fileIo, header.headerSize(), readIntoBuffer);
+
+ return createPageStore(filePath, header, meta);
+ } catch (IOException e) {
+ throw new IgniteInternalCheckedException("Error while creating file page store [file=" + filePath + "]:", e);
+ }
+ }
+
+ private PartitionFilePageStore createPageStore(
+ Path filePath,
+ FilePageStoreHeader header,
+ PartitionMeta partitionMeta
+ ) throws IgniteInternalCheckedException {
+ if (header.version() == FilePageStore.VERSION_1) {
+ return new PartitionFilePageStore(
+ new FilePageStore(
+ header.version(),
+ header.pageSize(),
+ header.headerSize(),
+ partitionMeta.pageCount(),
+ filePath,
+ fileIoFactory
+ ),
+ partitionMeta
+ );
+ }
+
+ throw new IgniteInternalCheckedException(String.format(
+ "Unknown version of file page store [version=%s, file=%s]",
+ header.version(),
+ filePath
+ ));
+ }
+
+ private PartitionMeta readPartitionMeta(FileIo fileIo, int headerSize, ByteBuffer readIntoBuffer) throws IOException {
+ if (fileIo.size() <= headerSize) {
+ return newPartitionMeta();
+ }
+
+ try {
+ fileIo.readFully(readIntoBuffer, headerSize);
+
+ long pageAddr = bufferAddress(readIntoBuffer);
+
+ if (PartitionMetaIo.getType(pageAddr) <= 0) {
+ return newPartitionMeta();
+ }
+
+ PartitionMetaIo partitionMetaIo = ioRegistry.resolve(pageAddr);
+
+ return new PartitionMeta(
Review Comment:
Can't we encapsulate this as well? I don't like that all methods of IO are explicitly listed somewhere outside of IO. Someone will forget to add line here while modifying Meta class, I promise you.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
Review Comment:
"Page IO" is not a synonym for "Page"
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -161,109 +164,24 @@ public CheckpointDirtyPagesQueue toQueue() {
return null;
}
- List<FullPageId> pageIds = dirtyPages.get(regionIndex).getValue();
+ FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
Review Comment:
Comment for the code above. Name "isToPositionLast" is very confusing
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -59,7 +66,10 @@ public class CheckpointPagesWriter implements Runnable {
* <p>Overall pages to write may be greater than this queue, since it may be necessary to retire write some pages due to unsuccessful
* page write lock acquisition
*/
- private final CheckpointDirtyPagesQueue writePageIds;
+ private final IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds;
+
+ /** Queue of dirty partition IDs to update {@link PartitionMetaIo} under this task. */
Review Comment:
What do you mean by "update PartitionMetaIo"? IO classes are stateless, they can't be updated
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -2056,39 +2076,39 @@ public FullPageId pullPageFromCpBuffer() {
}
/**
- * Gets a collection of dirty page IDs since the last checkpoint. If a dirty page is being written after the checkpointing operation
- * begun, the modifications will be written to a temporary buffer which will be flushed to the main memory after the checkpointing
- * finished. This method must be called when no concurrent operations on pages are performed.
+ * Returns the container of dirty page IDs and partition IDs since the last checkpoint. If a dirty page is being written after the
+ * checkpointing operation begun, the modifications will be written to a temporary buffer which will be flushed to the main memory after
+ * the checkpointing finished. This method must be called when no concurrent operations on pages are performed.
*
* @param allowToReplace The sign which allows replacing pages from a checkpoint by page replacer.
- * @return Collection view of dirty page IDs.
* @throws IgniteInternalException If checkpoint has been already started and was not finished.
*/
- public Collection<FullPageId> beginCheckpoint(CompletableFuture<?> allowToReplace) throws IgniteInternalException {
+ public CollectionDirtyPages beginCheckpoint(CompletableFuture<?> allowToReplace) throws IgniteInternalException {
if (segments == null) {
- return List.of();
+ return new CollectionDirtyPages(List.of(), List.of());
}
- Collection<FullPageId>[] collections = new Collection[segments.length];
+ Set<FullPageId>[] dirtyPageIds = new Set[segments.length];
+
+ Set<GroupPartitionId> dirtyPartitionIds = new HashSet<>();
for (int i = 0; i < segments.length; i++) {
- Segment seg = segments[i];
+ Segment segment = segments[i];
- if (seg.checkpointPages != null) {
- throw new IgniteInternalException("Failed to begin checkpoint (it is already in progress).");
- }
+ assert segment.checkpointPages == null : "Failed to begin checkpoint (it is already in progress)";
+
+ Set<FullPageId> segmentDirtyPages = (dirtyPageIds[i] = segment.dirtyPages);
- Set<FullPageId> dirtyPages = seg.dirtyPages;
- collections[i] = dirtyPages;
+ dirtyPartitionIds.addAll(segment.dirtyPartitions);
Review Comment:
You see, now it's required to manually merge these collections while holding a write lock. Maybe it's not a big deal.
Am I right that this collection is temporary and we have plans to remove it once we implement a "faster" way of taking partitions meta snapshots? If so, then please reflect this in a comment somewhere.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
Review Comment:
Please rename method "meta", it's not very clear. Also, I thought that we will not store meta directly in page store, why did you decide to do this anyway?
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PageMemoryStorageIoModule.java:
##########
@@ -38,8 +37,7 @@ public Collection<IoVersions<?>> ioVersions() {
TableMetaIo.VERSIONS,
TableInnerIo.VERSIONS,
TableLeafIo.VERSIONS,
- TableDataIo.VERSIONS,
- PartitionMetaIo.VERSIONS
+ TableDataIo.VERSIONS
Review Comment:
So, partition is now outside of table module, but it's aware about "rootId", weird
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointWorkflow.java:
##########
@@ -279,56 +288,69 @@ public List<CheckpointListener> collectCheckpointListeners(Collection<? extends
private DataRegionsDirtyPages beginCheckpoint(
Collection<? extends DataRegion<PersistentPageMemory>> dataRegions,
CompletableFuture<?> allowToReplace
- ) {
- Collection<IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>>> pages = new ArrayList<>(dataRegions.size());
+ ) throws IgniteInternalCheckedException {
+ Collection<DataRegionDirtyPages<CollectionDirtyPages>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
for (DataRegion<PersistentPageMemory> dataRegion : dataRegions) {
- Collection<FullPageId> dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
+ CollectionDirtyPages dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
- pages.add(new IgniteBiTuple<>(dataRegion.pageMemory(), dirtyPages));
+ dataRegionsDirtyPages.add(new DataRegionDirtyPages<>(dataRegion.pageMemory(), dirtyPages));
}
- return new DataRegionsDirtyPages(pages);
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages) {
+ for (GroupPartitionId dirtyPartition : dataRegionDirtyPages.dirtyPages.partitionIds()) {
+ partitionFilePageStoreManager.getStore(dirtyPartition.getGroupId(), dirtyPartition.getPartitionId()).updateMetaPageCount();
+ }
+ }
+
+ return new DataRegionsDirtyPages(dataRegionsDirtyPages);
}
CheckpointDirtyPages createAndSortCheckpointDirtyPages(
DataRegionsDirtyPages dataRegionsDirtyPages
) throws IgniteInternalCheckedException {
- List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> checkpointPages = new ArrayList<>();
+ List<DataRegionDirtyPages<ArrayDirtyPages>> checkpointDirtyPages = new ArrayList<>();
int realPagesArrSize = 0;
- for (IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>> regionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
- FullPageId[] checkpointRegionDirtyPages = new FullPageId[regionDirtyPages.getValue().size()];
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
+ FullPageId[] pageIds = new FullPageId[dataRegionDirtyPages.dirtyPages.pageIds().size()];
int pagePos = 0;
- for (FullPageId dirtyPage : regionDirtyPages.getValue()) {
+ for (FullPageId dirtyPage : dataRegionDirtyPages.dirtyPages.pageIds()) {
assert realPagesArrSize++ != dataRegionsDirtyPages.dirtyPageCount :
"Incorrect estimated dirty pages number: " + dataRegionsDirtyPages.dirtyPageCount;
- checkpointRegionDirtyPages[pagePos++] = dirtyPage;
+ pageIds[pagePos++] = dirtyPage;
Review Comment:
What if we use arraycopy here somehow? :) Or maybe we should just believe in JIT
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
+
+ io.setTreeRootPageId(pageAddr, meta.treeRootPageId());
Review Comment:
Dude, what is happening. This must be encapsulated!
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreHeader.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static java.nio.ByteOrder.nativeOrder;
+import static org.apache.ignite.internal.util.IgniteUtils.hexLong;
+import static org.apache.ignite.internal.util.IgniteUtils.readableSize;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * {@link FilePageStore} header.
+ *
+ * <p>Total length in bytes {@link #headerSize()}.
+ *
+ * <ul>
+ * <li>{@link #SIGNATURE signature} (8 byte)</li>
Review Comment:
8 byte**s**
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointWorkflow.java:
##########
@@ -279,56 +288,69 @@ public List<CheckpointListener> collectCheckpointListeners(Collection<? extends
private DataRegionsDirtyPages beginCheckpoint(
Collection<? extends DataRegion<PersistentPageMemory>> dataRegions,
CompletableFuture<?> allowToReplace
- ) {
- Collection<IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>>> pages = new ArrayList<>(dataRegions.size());
+ ) throws IgniteInternalCheckedException {
+ Collection<DataRegionDirtyPages<CollectionDirtyPages>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
for (DataRegion<PersistentPageMemory> dataRegion : dataRegions) {
- Collection<FullPageId> dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
+ CollectionDirtyPages dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
- pages.add(new IgniteBiTuple<>(dataRegion.pageMemory(), dirtyPages));
+ dataRegionsDirtyPages.add(new DataRegionDirtyPages<>(dataRegion.pageMemory(), dirtyPages));
}
- return new DataRegionsDirtyPages(pages);
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages) {
+ for (GroupPartitionId dirtyPartition : dataRegionDirtyPages.dirtyPages.partitionIds()) {
+ partitionFilePageStoreManager.getStore(dirtyPartition.getGroupId(), dirtyPartition.getPartitionId()).updateMetaPageCount();
+ }
+ }
+
+ return new DataRegionsDirtyPages(dataRegionsDirtyPages);
}
CheckpointDirtyPages createAndSortCheckpointDirtyPages(
DataRegionsDirtyPages dataRegionsDirtyPages
) throws IgniteInternalCheckedException {
- List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> checkpointPages = new ArrayList<>();
+ List<DataRegionDirtyPages<ArrayDirtyPages>> checkpointDirtyPages = new ArrayList<>();
int realPagesArrSize = 0;
- for (IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>> regionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
- FullPageId[] checkpointRegionDirtyPages = new FullPageId[regionDirtyPages.getValue().size()];
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
+ FullPageId[] pageIds = new FullPageId[dataRegionDirtyPages.dirtyPages.pageIds().size()];
int pagePos = 0;
- for (FullPageId dirtyPage : regionDirtyPages.getValue()) {
+ for (FullPageId dirtyPage : dataRegionDirtyPages.dirtyPages.pageIds()) {
assert realPagesArrSize++ != dataRegionsDirtyPages.dirtyPageCount :
"Incorrect estimated dirty pages number: " + dataRegionsDirtyPages.dirtyPageCount;
- checkpointRegionDirtyPages[pagePos++] = dirtyPage;
+ pageIds[pagePos++] = dirtyPage;
}
// Some pages may have been already replaced.
if (pagePos == 0) {
continue;
- } else if (pagePos != checkpointRegionDirtyPages.length) {
- checkpointPages.add(new IgniteBiTuple<>(regionDirtyPages.getKey(), Arrays.copyOf(checkpointRegionDirtyPages, pagePos)));
- } else {
- checkpointPages.add(new IgniteBiTuple<>(regionDirtyPages.getKey(), checkpointRegionDirtyPages));
+ } else if (pagePos != pageIds.length) {
+ pageIds = Arrays.copyOf(pageIds, pagePos);
}
+
+ assert !dataRegionDirtyPages.dirtyPages.partitionIds().isEmpty();
+
+ GroupPartitionId[] partitionIds = dataRegionDirtyPages.dirtyPages.partitionIds().toArray(new GroupPartitionId[0]);
+
+ checkpointDirtyPages.add(new DataRegionDirtyPages<>(
+ dataRegionDirtyPages.pageMemory,
+ new ArrayDirtyPages(pageIds, partitionIds)
+ ));
}
- List<ForkJoinTask<?>> parallelSortTasks = checkpointPages.stream()
- .map(IgniteBiTuple::getValue)
- .filter(pages -> pages.length >= PARALLEL_SORT_THRESHOLD)
- .map(pages -> parallelSortThreadPool.submit(() -> Arrays.parallelSort(pages, DIRTY_PAGE_COMPARATOR)))
+ List<ForkJoinTask<?>> parallelSortTasks = checkpointDirtyPages.stream()
Review Comment:
I wish you would add more comments here and there, this core is rather complicated and data just moves from one place to the other. It's very hard to track.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
+
+ io.setTreeRootPageId(pageAddr, meta.treeRootPageId());
+ io.setReuseListRootPageId(pageAddr, meta.reuseListRootPageId());
+ io.setPageCount(pageAddr, meta.pageCount());
+
+ FullPageId fullPageId = new FullPageId(partitionMetaPageId, partitionId.getGroupId());
+
+ PageStore store = pageWriter.write(
Review Comment:
So, for regular pages you're using the result of "createPageStoreWriter", which doesn't require partition generation tag. But here we have "pageWriter" that does require partition generation tag.
Something's messed up here, it shouldn't be this complicated. Why don't we have a single way of writing pages? Why do we need to know tag here?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -87,57 +103,66 @@ public class CheckpointPagesWriter implements Runnable {
*
* @param tracker Checkpoint metrics tracker.
* @param writePageIds Queue of dirty page IDs to write.
+ * @param updatePartitionIds Queue of dirty partition IDs to update.
* @param updStores Updating storage.
* @param doneFut Done future.
* @param beforePageWrite Action to be performed before every page write.
* @param log Logger.
* @param threadBuf Thread local byte buffer.
* @param checkpointProgress Checkpoint progress.
* @param pageWriter File page store manager.
+ * @param ioRegistry Page IO registry.
+ * @param partitionFilePageStoreManager Partition file page store manager.
* @param shutdownNow Shutdown supplier.
*/
CheckpointPagesWriter(
IgniteLogger log,
CheckpointMetricsTracker tracker,
- CheckpointDirtyPagesQueue writePageIds,
+ IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds,
+ IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds,
ConcurrentMap<PageStore, LongAdder> updStores,
CompletableFuture<?> doneFut,
Runnable beforePageWrite,
ThreadLocal<ByteBuffer> threadBuf,
CheckpointProgressImpl checkpointProgress,
CheckpointPageWriter pageWriter,
+ PageIoRegistry ioRegistry,
+ PartitionFilePageStoreManager partitionFilePageStoreManager,
BooleanSupplier shutdownNow
) {
this.log = log;
this.tracker = tracker;
this.writePageIds = writePageIds;
+ this.updatePartitionIds = updatePartitionIds;
this.updStores = updStores;
this.doneFut = doneFut;
this.beforePageWrite = beforePageWrite;
this.threadBuf = threadBuf;
this.checkpointProgress = checkpointProgress;
this.pageWriter = pageWriter;
+ this.ioRegistry = ioRegistry;
+ this.partitionFilePageStoreManager = partitionFilePageStoreManager;
this.shutdownNow = shutdownNow;
}
/** {@inheritDoc} */
@Override
public void run() {
try {
- CheckpointDirtyPagesQueue pagesToRetry = writePages(writePageIds);
-
- if (pagesToRetry.isEmpty()) {
- doneFut.complete(null);
- } else {
- log.info("Checkpoint pages were not written yet due to "
- + "unsuccessful page write lock acquisition and will be retried [pageCount={}]", pagesToRetry.size());
+ IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> pageIdsToRetry = writePages(writePageIds);
- while (!pagesToRetry.isEmpty()) {
- pagesToRetry = writePages(pagesToRetry);
+ while (!pageIdsToRetry.isEmpty()) {
+ if (log.isInfoEnabled()) {
Review Comment:
Oh my god, you print this message on first iteration. I consider this a bug, please change it
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -161,109 +164,24 @@ public CheckpointDirtyPagesQueue toQueue() {
return null;
}
- List<FullPageId> pageIds = dirtyPages.get(regionIndex).getValue();
+ FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
- if (fromPosition == pageIds.size() - 1 || !equalsByGroupAndPartition(pageIds.get(fromPosition), pageIds.get(fromPosition + 1))) {
+ if (fromPosition == pageIds.length - 1 || !equalsByGroupAndPartition(pageIds[fromPosition], pageIds[fromPosition + 1])) {
return new CheckpointDirtyPagesView(regionIndex, fromPosition, fromPosition);
}
- FullPageId startPageId = pageIds.get(fromPosition);
+ FullPageId startPageId = pageIds[fromPosition];
FullPageId endPageId = new FullPageId(pageId(partitionId(startPageId.pageId()) + 1, (byte) 0, 0), startPageId.groupId());
- int toPosition = binarySearch(pageIds.subList(fromPosition, pageIds.size()), endPageId, DIRTY_PAGE_COMPARATOR);
+ int toPosition = binarySearch(pageIds, fromPosition, pageIds.length, endPageId, DIRTY_PAGE_COMPARATOR);
toPosition = toPosition > 0 ? toPosition - 1 : -toPosition - 2;
Review Comment:
Is "toPosition" an "inclusive" index? I don't understand, why you subtract 1 when it's positive.
Usually, lower bound is inclusive and upper bound is exclusive. It looks like you violated this rule. If so, why?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/ArrayDirtyPages.java:
##########
@@ -15,37 +15,29 @@
* limitations under the License.
*/
-package org.apache.ignite.internal.storage.pagememory;
+package org.apache.ignite.internal.pagememory.persistence.checkpoint;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
/**
- * Class for storing {@link TableTree} partition metadata.
+ * Dirty page array container.
*/
-class PartitionMeta {
- /** {@link TableTree} root. */
- final FullPageId treeRoot;
+class ArrayDirtyPages {
Review Comment:
Again, I don't get the order of words here. Was is supposed to be "ArrayOfDirtyPages"?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/CollectionDirtyPages.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import java.util.Collection;
+import org.apache.ignite.internal.pagememory.FullPageId;
+
+/**
+ * Dirty pages collection container.
+ */
+public class CollectionDirtyPages {
+ private final Collection<FullPageId> pageIds;
Review Comment:
Should we explicitly use "Set" here? Or in the next collection?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -1549,18 +1573,14 @@ public long refreshOutdatedPage(int grpId, long pageId, boolean rmv) {
loadedPages.remove(grpId, effectivePageId(pageId));
}
- CheckpointPages cpPages = checkpointPages;
+ CheckpointPages checkpointPages = this.checkpointPages;
- if (cpPages != null) {
- cpPages.markAsSaved(new FullPageId(pageId, grpId));
+ if (checkpointPages != null) {
+ checkpointPages.markAsSaved(new FullPageId(pageId, grpId));
Review Comment:
I still don't get it. A comment is required here, why are we modifying these collections if there's an outdated page.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
Review Comment:
How often is this method called? Pre-calculated value is fast. Counting a count every time is slow. So I'm wondering, why did you do this?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -874,19 +871,37 @@ private void copyInBuffer(long absPtr, ByteBuffer tmpBuf) {
}
/**
- * Get current prartition generation tag.
+ * Get current partition generation tag.
*
* @param seg Segment.
- * @param fullId Full page id.
+ * @param fullPageId Full page id.
* @return Current partition generation tag.
*/
- private int generationTag(Segment seg, FullPageId fullId) {
+ private int generationTag(Segment seg, FullPageId fullPageId) {
return seg.partGeneration(
- fullId.groupId(),
- partitionId(fullId.pageId())
+ fullPageId.groupId(),
+ partitionId(fullPageId.pageId())
);
}
+ /**
+ * Get current partition generation tag.
+ *
+ * @param fullPageId Full page id.
+ * @return Current partition generation tag.
+ */
+ public int generationTag(FullPageId fullPageId) {
Review Comment:
Why is this public now? Partition generation is internal information, isn't it?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/CollectionDirtyPages.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import java.util.Collection;
+import org.apache.ignite.internal.pagememory.FullPageId;
+
+/**
+ * Dirty pages collection container.
+ */
+public class CollectionDirtyPages {
Review Comment:
Why not "DirtyPagesCollection"? I don't quite get the order of words here
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
}
/**
- * Returns a queue of dirty pages to be written to a checkpoint.
+ * Returns a queue of dirty page IDs to be written to a checkpoint.
*/
- public CheckpointDirtyPagesQueue toQueue() {
- return new CheckpointDirtyPagesQueue();
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> toDirtyPageIdQueue() {
+ List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> dirtyPageIds = dirtyPages.stream()
+ .map(pages -> new IgniteBiTuple<>(pages.pageMemory, pages.dirtyPages.pageIds))
+ .collect(toList());
+
+ return new IgniteConcurrentMultiPairQueue<>(dirtyPageIds);
}
/**
- * Looks for dirty page views for a specific group and partition.
+ * Returns a queue of dirty partition IDs to be modified to a checkpoint.
+ */
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> toDirtyPartitionIdQueue() {
Review Comment:
Why is this a queue? Do we process it in parallel while holding checkpoint write lock?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
Review Comment:
I suppose that DataRegionDirtyPages is defined as
`class DataRegionDirtyPages<T>` and T doesn't have to be a dirty pages collection of any sort, is this correct?
This type is pretty mouthful, would it be more convenient to have DataRegionData<ArrayDirtyPages>? Or something else that's less redundant?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -920,19 +935,21 @@ public int invalidate(int grpId, int partId) {
int tag = 0;
- for (Segment seg : segments) {
- seg.writeLock().lock();
+ for (Segment segment : segments) {
+ segment.writeLock().lock();
try {
- int newTag = seg.incrementPartGeneration(grpId, partId);
+ int newTag = segment.incrementPartGeneration(grpId, partId);
if (tag == 0) {
tag = newTag;
}
assert tag == newTag;
+
+ segment.dirtyPartitions.remove(new GroupPartitionId(grpId, partId));
Review Comment:
This implies that each segment has independent set of partitions. I believe that this set will be the same for all segments (on avarage). Was this a conscious decision?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPages.java:
##########
@@ -30,7 +30,7 @@
* View of pages which should be stored during current checkpoint.
*/
public class CheckpointPages {
- private final Set<FullPageId> segCheckpointPages;
+ private final Set<FullPageId> segmentDirtyPages;
Review Comment:
Ok, these can just be "segmentPages" then. There's no way that clean page can get here, right?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -113,29 +116,29 @@ public CheckpointDirtyPagesQueue toQueue() {
FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
for (int i = 0; i < dirtyPages.size(); i++) {
- List<FullPageId> pageIds = dirtyPages.get(i).getValue();
+ FullPageId[] pageIds = dirtyPages.get(i).dirtyPages.pageIds;
Review Comment:
This comment is not for this particular line. I find it strange that CheckpointDirtyPagesView is always used for a single partition, yet it's not reflected in its name. Name "findView" is also very technical, it doesn't reflect a meaning - here we find all pages that belong to a given partition, right?
Also, passing a DataRegion or PageMemory instance here would help you avoiding unnecessary binary searches.
Also, why exactly do we have a list of arrays instead of the Map, as in the original constructor?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
}
/**
- * Returns a queue of dirty pages to be written to a checkpoint.
+ * Returns a queue of dirty page IDs to be written to a checkpoint.
*/
- public CheckpointDirtyPagesQueue toQueue() {
- return new CheckpointDirtyPagesQueue();
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> toDirtyPageIdQueue() {
+ List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> dirtyPageIds = dirtyPages.stream()
+ .map(pages -> new IgniteBiTuple<>(pages.pageMemory, pages.dirtyPages.pageIds))
+ .collect(toList());
+
+ return new IgniteConcurrentMultiPairQueue<>(dirtyPageIds);
Review Comment:
This looks like a very non-optimal step. Can we consider/discuss other implementation? Why does it have to be a list of pairs and not a wrapper of existing list of arrays? Data copying could be avoided.
Plus, that list of arrays was itself created from the map, we just convert data back and forth, we should probably decrease a number of conversions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920968007
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -920,19 +935,21 @@ public int invalidate(int grpId, int partId) {
int tag = 0;
- for (Segment seg : segments) {
- seg.writeLock().lock();
+ for (Segment segment : segments) {
+ segment.writeLock().lock();
try {
- int newTag = seg.incrementPartGeneration(grpId, partId);
+ int newTag = segment.incrementPartGeneration(grpId, partId);
if (tag == 0) {
tag = newTag;
}
assert tag == newTag;
+
+ segment.dirtyPartitions.remove(new GroupPartitionId(grpId, partId));
Review Comment:
In general, I can make a general one for all segments, but do I need to do the same with **PersistentPageMemory.Segment#partGenerationMap**? It seems that we can reduce the contention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921243949
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -1549,18 +1573,14 @@ public long refreshOutdatedPage(int grpId, long pageId, boolean rmv) {
loadedPages.remove(grpId, effectivePageId(pageId));
}
- CheckpointPages cpPages = checkpointPages;
+ CheckpointPages checkpointPages = this.checkpointPages;
- if (cpPages != null) {
- cpPages.markAsSaved(new FullPageId(pageId, grpId));
+ if (checkpointPages != null) {
+ checkpointPages.markAsSaved(new FullPageId(pageId, grpId));
Review Comment:
No need to modify removed this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920957316
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/CollectionDirtyPages.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import java.util.Collection;
+import org.apache.ignite.internal.pagememory.FullPageId;
+
+/**
+ * Dirty pages collection container.
+ */
+public class CollectionDirtyPages {
Review Comment:
Got it, I'll change the class name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921137045
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -2056,39 +2076,39 @@ public FullPageId pullPageFromCpBuffer() {
}
/**
- * Gets a collection of dirty page IDs since the last checkpoint. If a dirty page is being written after the checkpointing operation
- * begun, the modifications will be written to a temporary buffer which will be flushed to the main memory after the checkpointing
- * finished. This method must be called when no concurrent operations on pages are performed.
+ * Returns the container of dirty page IDs and partition IDs since the last checkpoint. If a dirty page is being written after the
+ * checkpointing operation begun, the modifications will be written to a temporary buffer which will be flushed to the main memory after
+ * the checkpointing finished. This method must be called when no concurrent operations on pages are performed.
*
* @param allowToReplace The sign which allows replacing pages from a checkpoint by page replacer.
- * @return Collection view of dirty page IDs.
* @throws IgniteInternalException If checkpoint has been already started and was not finished.
*/
- public Collection<FullPageId> beginCheckpoint(CompletableFuture<?> allowToReplace) throws IgniteInternalException {
+ public CollectionDirtyPages beginCheckpoint(CompletableFuture<?> allowToReplace) throws IgniteInternalException {
if (segments == null) {
- return List.of();
+ return new CollectionDirtyPages(List.of(), List.of());
}
- Collection<FullPageId>[] collections = new Collection[segments.length];
+ Set<FullPageId>[] dirtyPageIds = new Set[segments.length];
+
+ Set<GroupPartitionId> dirtyPartitionIds = new HashSet<>();
for (int i = 0; i < segments.length; i++) {
- Segment seg = segments[i];
+ Segment segment = segments[i];
- if (seg.checkpointPages != null) {
- throw new IgniteInternalException("Failed to begin checkpoint (it is already in progress).");
- }
+ assert segment.checkpointPages == null : "Failed to begin checkpoint (it is already in progress)";
+
+ Set<FullPageId> segmentDirtyPages = (dirtyPageIds[i] = segment.dirtyPages);
- Set<FullPageId> dirtyPages = seg.dirtyPages;
- collections[i] = dirtyPages;
+ dirtyPartitionIds.addAll(segment.dirtyPartitions);
Review Comment:
Discussed personally, no need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921140543
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
Review Comment:
Discussed personally, no need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921147237
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreHeader.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static java.nio.ByteOrder.nativeOrder;
+import static org.apache.ignite.internal.util.IgniteUtils.hexLong;
+import static org.apache.ignite.internal.util.IgniteUtils.readableSize;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * {@link FilePageStore} header.
+ *
+ * <p>Total length in bytes {@link #headerSize()}.
+ *
+ * <ul>
+ * <li>{@link #SIGNATURE signature} (8 byte)</li>
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921136765
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -920,19 +935,21 @@ public int invalidate(int grpId, int partId) {
int tag = 0;
- for (Segment seg : segments) {
- seg.writeLock().lock();
+ for (Segment segment : segments) {
+ segment.writeLock().lock();
try {
- int newTag = seg.incrementPartGeneration(grpId, partId);
+ int newTag = segment.incrementPartGeneration(grpId, partId);
if (tag == 0) {
tag = newTag;
}
assert tag == newTag;
+
+ segment.dirtyPartitions.remove(new GroupPartitionId(grpId, partId));
Review Comment:
Discussed personally, no need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921224621
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PageMemoryStorageIoModule.java:
##########
@@ -38,8 +37,7 @@ public Collection<IoVersions<?>> ioVersions() {
TableMetaIo.VERSIONS,
TableInnerIo.VERSIONS,
TableLeafIo.VERSIONS,
- TableDataIo.VERSIONS,
- PartitionMetaIo.VERSIONS
+ TableDataIo.VERSIONS
Review Comment:
I agree that it is strange, but what to do now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924309236
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -229,15 +242,15 @@ public PersistentPageMemory pageMemory() {
* Returns the size of the view.
*/
public int size() {
- return toPosition - fromPosition + 1;
+ return toPosition - fromPosition;
}
private CheckpointDirtyPages owner() {
return CheckpointDirtyPages.this;
}
- private boolean isToPositionLast() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length - 1;
+ private boolean needsNextRegion() {
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
Review Comment:
You can have both index and collection, it's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924390811
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -250,7 +215,7 @@ private CheckpointDirtyPages owner() {
}
private boolean needsNextRegion() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.length;
Review Comment:
Yes, the **regionIndex** is needed to search for the **nextPartitionView**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924329610
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/FullPageId.java:
##########
@@ -185,4 +185,6 @@ public int partitionId() {
public int pageIdx() {
return PageIdUtils.pageIndex(pageId);
}
+
Review Comment:
What is this?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -117,7 +120,7 @@ public class CheckpointPagesWriter implements Runnable {
IgniteLogger log,
CheckpointMetricsTracker tracker,
IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds,
- IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds,
+ Set<GroupPartitionId> savedPartitionMetas,
Review Comment:
Interesting, why does it have to be a parameter? I don't think I get it, are we use this collection somewhere outside of the writer?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -183,18 +184,24 @@ private IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePa
Result<PersistentPageMemory, FullPageId> queueResult = new Result<>();
+ GroupPartitionId partitionId = null;
+
while (!shutdownNow.getAsBoolean() && writePageIds.next(queueResult)) {
beforePageWrite.run();
- FullPageId fullId = queueResult.getValue();
+ FullPageId pageId = queueResult.getValue();
Review Comment:
"fullId" was better. When I see a pageId, I think that it lacks groupId, as it should, because usually it's a "long" value.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -240,35 +247,25 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
};
}
- /**
- * Updates the partitions meta page.
- *
- * @throws IgniteInternalCheckedException If failed.
- */
- private void updatePartitions() throws IgniteInternalCheckedException {
- Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+ private void writePartitionMeta(GroupPartitionId partitionId, ByteBuffer buffer) throws IgniteInternalCheckedException {
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
- while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
- GroupPartitionId partitionId = queueResult.getValue();
+ partitionMetaManager.writeMetaToBuffer(partitionId, partitionMetaSnapshot, buffer.rewind());
- ByteBuffer buffer = threadBuf.get();
+ FullPageId fullPageId = new FullPageId(partitionMetaPageId(partitionId.getPartitionId()), partitionId.getGroupId());
- PersistentPageMemory pageMemory = queueResult.getKey();
-
- PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
-
- partitionMetaManager.writeMetaToBuffer(partitionId, pageMemory, partitionMetaSnapshot, buffer.rewind());
+ PageStore store = pageWriter.write(fullPageId, buffer.rewind());
- FullPageId fullPageId = new FullPageId(
- pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId()),
- partitionId.getGroupId()
- );
+ checkpointProgress.writtenPagesCounter().incrementAndGet();
- PageStore store = pageWriter.write(fullPageId, buffer.rewind());
+ updStores.computeIfAbsent(store, k -> new LongAdder()).increment();
+ }
- checkpointProgress.writtenPagesCounter().incrementAndGet();
+ private static boolean isPartitionChange(@Nullable GroupPartitionId partitionId, FullPageId pageId) {
Review Comment:
Sorry for making comments about naming, but how should I interpret this name?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -237,9 +266,41 @@ private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean
}
}
+ /**
+ * Writes a page unlike {@link #write(long, ByteBuffer, boolean)}, checks the page offset in the file not logically (pageOffset <=
+ * {@link #pages()} * {@link #pageSize}) but physically (pageOffset <= {@link #size()}), which can affect performance when used in
+ * production code.
+ *
+ * @param pageId Page ID.
+ * @param pageBuf Page buffer to write from.
+ * @param calculateCrc If {@code false} crc calculation will be forcibly skipped.
+ * @throws IgniteInternalCheckedException If page writing failed (IO error occurred).
+ */
+ public void writeByPhysicalOffset(long pageId, ByteBuffer pageBuf, boolean calculateCrc) throws IgniteInternalCheckedException {
Review Comment:
This one seems very non-intuitive for me. We could allocate meta page before writing it, it shouldn't contradict anything. I don't really understand the necessity of 2 separate write methods in this case
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -132,11 +150,17 @@ public PartitionMetaSnapshot metaSnapshot(@Nullable UUID checkpointId) {
return metaSnapshot;
}
+ /**
+ * Takes a snapshot of the meta partition if the {@code checkpointId} is different from the {@link #metaSnapshot last snapshot} {@link
Review Comment:
"of the partition meta"
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -250,7 +215,7 @@ private CheckpointDirtyPages owner() {
}
private boolean needsNextRegion() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.length;
Review Comment:
You decided to leave this extra lookup, I see
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,27 +17,29 @@
package org.apache.ignite.internal.pagememory.persistence;
+import static org.apache.ignite.internal.pagememory.PageIdAllocator.FLAG_AUX;
+import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
+
import java.lang.invoke.MethodHandles;
import java.lang.invoke.VarHandle;
import java.util.UUID;
-import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
import org.apache.ignite.internal.tostring.S;
import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
- private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
- PartitionMeta.class,
- "pageCount"
- );
+ private static final VarHandle PAGE_COUNT;
Review Comment:
I know that comments for private stuff are not mandatory, but you could at least add a link to a field. Just saying
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/Checkpointer.java:
##########
@@ -381,14 +382,13 @@ boolean writePages(
IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds = checkpointDirtyPages.toDirtyPageIdQueue();
- IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds =
- checkpointDirtyPages.toDirtyPartitionIdQueue();
+ Set<GroupPartitionId> savedPartitionMetas = ConcurrentHashMap.newKeySet();
Review Comment:
So, I believe that this object can be instantiated somewhere inside of checkpoint writer.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -182,23 +198,36 @@ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws Ign
* @param pageBuf Page buffer to read into.
* @param checkCrc Check CRC on page.
* @param keepCrc By default reading zeroes CRC which was on file, but you can keep it in pageBuf if set keepCrc.
+ * @param checkPageOffsetLogically Check page offset by {@link #allocatedBytes} or {@link #size}.
* @return {@code true} if page has been read successfully, {@code false} if page hasn't been written yet.
* @throws IgniteInternalCheckedException If reading failed (IO error occurred).
*/
- private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean keepCrc) throws IgniteInternalCheckedException {
+ private boolean read0(
+ long pageId,
+ ByteBuffer pageBuf,
+ boolean checkCrc,
+ boolean keepCrc,
+ boolean checkPageOffsetLogically
+ ) throws IgniteInternalCheckedException {
ensure();
try {
- long off = pageOffset(pageId);
-
assert pageBuf.capacity() == pageSize : pageBuf.capacity();
assert pageBuf.remaining() == pageSize : pageBuf.remaining();
assert pageBuf.position() == 0 : pageBuf.position();
assert pageBuf.order() == nativeOrder() : pageBuf.order();
- assert off <= allocatedBytes() : "calculatedOffset=" + off
- + ", allocated=" + allocatedBytes() + ", headerSize=" + headerSize + ", filePath=" + filePath;
- int n = readWithFailover(pageBuf, off);
+ long pageOff = pageOffset(pageId);
+
+ if (checkPageOffsetLogically) {
+ assert pageOff <= allocatedBytes() : "calculatedOffset=" + pageOff
Review Comment:
Technically speaking, we could add "pageSize" to "pageOff" in both assertions, because "pageOff" points to a beginning of the page, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r925308112
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -232,8 +230,6 @@ private boolean read0(
// If page was not written yet, nothing to read.
if (n < 0) {
pageBuf.put(new byte[pageBuf.remaining()]);
Review Comment:
Created IGNITE-17397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923169668
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
- if (dirtyPages.isEmpty()) {
- return null;
+ for (int i = 0; i < dirtyPages.size(); i++) {
+ CheckpointDirtyPagesView view = findView(i, grpId, partId);
+
+ if (view != null) {
+ return view;
+ }
}
- FullPageId startPageId = new FullPageId(pageId(partId, (byte) 0, 0), grpId);
- FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
+ return null;
+ }
+ /**
+ * Looks for dirty page IDs views for a specific group and partition.
+ *
+ * @param pageMemory Page memory.
+ * @param grpId Group ID.
+ * @param partId Partition ID.
+ */
+ public @Nullable CheckpointDirtyPagesView findView(PersistentPageMemory pageMemory, int grpId, int partId) {
for (int i = 0; i < dirtyPages.size(); i++) {
- FullPageId[] pageIds = dirtyPages.get(i).dirtyPages.pageIds;
+ if (dirtyPages.get(i).pageMemory == pageMemory) {
+ return findView(i, grpId, partId);
+ }
+ }
- int fromIndex = binarySearch(pageIds, startPageId, DIRTY_PAGE_COMPARATOR);
+ return null;
Review Comment:
Technically possible, I can add a check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923182250
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -167,15 +180,15 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
if (fromPosition == pageIds.length - 1 || !equalsByGroupAndPartition(pageIds[fromPosition], pageIds[fromPosition + 1])) {
Review Comment:
In general, this is a small optimization, but if you insist, I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920979802
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
}
/**
- * Returns a queue of dirty pages to be written to a checkpoint.
+ * Returns a queue of dirty page IDs to be written to a checkpoint.
*/
- public CheckpointDirtyPagesQueue toQueue() {
- return new CheckpointDirtyPagesQueue();
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> toDirtyPageIdQueue() {
+ List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> dirtyPageIds = dirtyPages.stream()
+ .map(pages -> new IgniteBiTuple<>(pages.pageMemory, pages.dirtyPages.pageIds))
+ .collect(toList());
+
+ return new IgniteConcurrentMultiPairQueue<>(dirtyPageIds);
Review Comment:
Okay, let's talk in person.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920977739
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
Review Comment:
**DataRegionData** - sounds like data region data, not dirty pages. While I don't know how the name is better, the current name at least says that there are dirty pages and not something else.
This class is package private, I think it's not scary that it's just **T**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920960486
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/CollectionDirtyPages.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import java.util.Collection;
+import org.apache.ignite.internal.pagememory.FullPageId;
+
+/**
+ * Dirty pages collection container.
+ */
+public class CollectionDirtyPages {
+ private final Collection<FullPageId> pageIds;
Review Comment:
I don’t think it’s necessary, since we don’t need the features of the set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921200619
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -161,109 +164,24 @@ public CheckpointDirtyPagesQueue toQueue() {
return null;
}
- List<FullPageId> pageIds = dirtyPages.get(regionIndex).getValue();
+ FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
Review Comment:
**isNeedNextDirtyPagesRegion** is fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920973299
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -2056,39 +2076,39 @@ public FullPageId pullPageFromCpBuffer() {
}
/**
- * Gets a collection of dirty page IDs since the last checkpoint. If a dirty page is being written after the checkpointing operation
- * begun, the modifications will be written to a temporary buffer which will be flushed to the main memory after the checkpointing
- * finished. This method must be called when no concurrent operations on pages are performed.
+ * Returns the container of dirty page IDs and partition IDs since the last checkpoint. If a dirty page is being written after the
+ * checkpointing operation begun, the modifications will be written to a temporary buffer which will be flushed to the main memory after
+ * the checkpointing finished. This method must be called when no concurrent operations on pages are performed.
*
* @param allowToReplace The sign which allows replacing pages from a checkpoint by page replacer.
- * @return Collection view of dirty page IDs.
* @throws IgniteInternalException If checkpoint has been already started and was not finished.
*/
- public Collection<FullPageId> beginCheckpoint(CompletableFuture<?> allowToReplace) throws IgniteInternalException {
+ public CollectionDirtyPages beginCheckpoint(CompletableFuture<?> allowToReplace) throws IgniteInternalException {
if (segments == null) {
- return List.of();
+ return new CollectionDirtyPages(List.of(), List.of());
}
- Collection<FullPageId>[] collections = new Collection[segments.length];
+ Set<FullPageId>[] dirtyPageIds = new Set[segments.length];
+
+ Set<GroupPartitionId> dirtyPartitionIds = new HashSet<>();
for (int i = 0; i < segments.length; i++) {
- Segment seg = segments[i];
+ Segment segment = segments[i];
- if (seg.checkpointPages != null) {
- throw new IgniteInternalException("Failed to begin checkpoint (it is already in progress).");
- }
+ assert segment.checkpointPages == null : "Failed to begin checkpoint (it is already in progress)";
+
+ Set<FullPageId> segmentDirtyPages = (dirtyPageIds[i] = segment.dirtyPages);
- Set<FullPageId> dirtyPages = seg.dirtyPages;
- collections[i] = dirtyPages;
+ dirtyPartitionIds.addAll(segment.dirtyPartitions);
Review Comment:
I don't know how to do it faster yet, we will have to choose between memory usage and contention anyway, while I chose to use more memory. I think that the merging of these collections should not be long in time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r922198393
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
+
+ io.setTreeRootPageId(pageAddr, meta.treeRootPageId());
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923177981
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
- if (dirtyPages.isEmpty()) {
- return null;
+ for (int i = 0; i < dirtyPages.size(); i++) {
+ CheckpointDirtyPagesView view = findView(i, grpId, partId);
+
+ if (view != null) {
+ return view;
+ }
}
- FullPageId startPageId = new FullPageId(pageId(partId, (byte) 0, 0), grpId);
- FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
+ return null;
+ }
+ /**
+ * Looks for dirty page IDs views for a specific group and partition.
+ *
+ * @param pageMemory Page memory.
+ * @param grpId Group ID.
+ * @param partId Partition ID.
+ */
+ public @Nullable CheckpointDirtyPagesView findView(PersistentPageMemory pageMemory, int grpId, int partId) {
for (int i = 0; i < dirtyPages.size(); i++) {
- FullPageId[] pageIds = dirtyPages.get(i).dirtyPages.pageIds;
+ if (dirtyPages.get(i).pageMemory == pageMemory) {
+ return findView(i, grpId, partId);
+ }
+ }
- int fromIndex = binarySearch(pageIds, startPageId, DIRTY_PAGE_COMPARATOR);
+ return null;
+ }
- fromIndex = fromIndex >= 0 ? fromIndex : Math.min(pageIds.length - 1, -fromIndex - 1);
+ private @Nullable CheckpointDirtyPagesView findView(int dirtyPagesIdx, int grpId, int partId) {
+ FullPageId startPageId = new FullPageId(pageId(partId, (byte) 0, 0), grpId);
+ FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
- if (!equalsByGroupAndPartition(startPageId, pageIds[fromIndex])) {
- continue;
- }
+ FullPageId[] pageIds = dirtyPages.get(dirtyPagesIdx).dirtyPages.pageIds;
- int toIndex = binarySearch(pageIds, fromIndex, pageIds.length, endPageId, DIRTY_PAGE_COMPARATOR);
+ int fromIndex = binarySearch(pageIds, startPageId, DIRTY_PAGE_COMPARATOR);
- // toIndex cannot be 0 because endPageId is greater than startPageId by DIRTY_PAGE_COMPARATOR.
- toIndex = toIndex > 0 ? toIndex - 1 : -toIndex - 2;
+ fromIndex = fromIndex >= 0 ? fromIndex : Math.min(pageIds.length - 1, -fromIndex - 1);
- return new CheckpointDirtyPagesView(i, fromIndex, toIndex);
+ if (!equalsByGroupAndPartition(startPageId, pageIds[fromIndex])) {
+ return null;
}
- return null;
+ int toIndex = binarySearch(pageIds, fromIndex, pageIds.length, endPageId, DIRTY_PAGE_COMPARATOR);
+
+ // toIndex cannot be 0 because endPageId is greater than startPageId by DIRTY_PAGE_COMPARATOR.
+ toIndex = toIndex > 0 ? toIndex : -toIndex - 1;
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924400629
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -182,23 +198,36 @@ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws Ign
* @param pageBuf Page buffer to read into.
* @param checkCrc Check CRC on page.
* @param keepCrc By default reading zeroes CRC which was on file, but you can keep it in pageBuf if set keepCrc.
+ * @param checkPageOffsetLogically Check page offset by {@link #allocatedBytes} or {@link #size}.
* @return {@code true} if page has been read successfully, {@code false} if page hasn't been written yet.
* @throws IgniteInternalCheckedException If reading failed (IO error occurred).
*/
- private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean keepCrc) throws IgniteInternalCheckedException {
+ private boolean read0(
+ long pageId,
+ ByteBuffer pageBuf,
+ boolean checkCrc,
+ boolean keepCrc,
+ boolean checkPageOffsetLogically
+ ) throws IgniteInternalCheckedException {
ensure();
try {
- long off = pageOffset(pageId);
-
assert pageBuf.capacity() == pageSize : pageBuf.capacity();
assert pageBuf.remaining() == pageSize : pageBuf.remaining();
assert pageBuf.position() == 0 : pageBuf.position();
assert pageBuf.order() == nativeOrder() : pageBuf.order();
- assert off <= allocatedBytes() : "calculatedOffset=" + off
- + ", allocated=" + allocatedBytes() + ", headerSize=" + headerSize + ", filePath=" + filePath;
- int n = readWithFailover(pageBuf, off);
+ long pageOff = pageOffset(pageId);
+
+ if (checkPageOffsetLogically) {
+ assert pageOff <= allocatedBytes() : "calculatedOffset=" + pageOff
Review Comment:
This wasn't about the naming, it was about the fact that the assertion is not strict enough, at least if looks like it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924400095
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/Checkpointer.java:
##########
@@ -381,14 +382,13 @@ boolean writePages(
IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds = checkpointDirtyPages.toDirtyPageIdQueue();
- IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds =
- checkpointDirtyPages.toDirtyPartitionIdQueue();
+ Set<GroupPartitionId> savedPartitionMetas = ConcurrentHashMap.newKeySet();
Review Comment:
Yes, you are right, I missed that part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923118797
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,33 +17,54 @@
package org.apache.ignite.internal.pagememory.persistence;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import org.apache.ignite.internal.tostring.S;
+import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
+ private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
+ PartitionMeta.class,
+ "pageCount"
+ );
+
+ private static final VarHandle META_SNAPSHOT_VH;
+
+ static {
+ try {
+ META_SNAPSHOT_VH = MethodHandles.lookup().findVarHandle(PartitionMeta.class, "metaSnapshot", PartitionMetaSnapshot.class);
+ } catch (ReflectiveOperationException e) {
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
private volatile long treeRootPageId;
private volatile long reuseListRootPageId;
private volatile int pageCount;
- private final boolean created;
+ private volatile PartitionMetaSnapshot metaSnapshot;
/**
* Constructor.
*
+ * @param checkpointId Checkpoint ID.
* @param treeRootPageId Tree root page ID.
* @param reuseListRootPageId Reuse list root page ID.
* @param pageCount Count of pages in the partition.
- * @param created {@code True} if the partition meta was created, {@code false} if it was read from a partition file.
*/
- public PartitionMeta(long treeRootPageId, long reuseListRootPageId, int pageCount, boolean created) {
+ public PartitionMeta(@Nullable UUID checkpointId, long treeRootPageId, long reuseListRootPageId, int pageCount) {
Review Comment:
It will be **null** until the first breakpoint occurs. Do you think this can be ignored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923123074
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,33 +17,54 @@
package org.apache.ignite.internal.pagememory.persistence;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import org.apache.ignite.internal.tostring.S;
+import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
+ private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
+ PartitionMeta.class,
+ "pageCount"
+ );
+
+ private static final VarHandle META_SNAPSHOT_VH;
+
+ static {
+ try {
+ META_SNAPSHOT_VH = MethodHandles.lookup().findVarHandle(PartitionMeta.class, "metaSnapshot", PartitionMetaSnapshot.class);
+ } catch (ReflectiveOperationException e) {
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
private volatile long treeRootPageId;
private volatile long reuseListRootPageId;
private volatile int pageCount;
- private final boolean created;
+ private volatile PartitionMetaSnapshot metaSnapshot;
/**
* Constructor.
*
+ * @param checkpointId Checkpoint ID.
* @param treeRootPageId Tree root page ID.
* @param reuseListRootPageId Reuse list root page ID.
* @param pageCount Count of pages in the partition.
- * @param created {@code True} if the partition meta was created, {@code false} if it was read from a partition file.
*/
- public PartitionMeta(long treeRootPageId, long reuseListRootPageId, int pageCount, boolean created) {
+ public PartitionMeta(@Nullable UUID checkpointId, long treeRootPageId, long reuseListRootPageId, int pageCount) {
Review Comment:
I don't mind it, to be honest. Let's leave it nullable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923268461
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -159,11 +165,11 @@ public void stop() throws StorageException {
Stream<AutoCloseable> closeRegions = regions.values().stream().map(region -> region::stop);
CheckpointManager checkpointManager = this.checkpointManager;
- PartitionFilePageStoreManager partitionFilePageStoreManager = this.partitionFilePageStoreManager;
+ FilePageStoreManager filePageStoreManager = this.filePageStoreManager;
Stream<AutoCloseable> closeManagers = Stream.of(
checkpointManager == null ? null : (AutoCloseable) checkpointManager::stop,
- partitionFilePageStoreManager == null ? null : (AutoCloseable) partitionFilePageStoreManager::stop
+ filePageStoreManager == null ? null : (AutoCloseable) filePageStoreManager::stop
Review Comment:
Decided to leave it as it is for now.
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
+ Loggers.forClass(FilePageStoreManager.class),
igniteInstanceName,
storagePath,
fileIoFactory,
- ioRegistry,
pageSize
);
- partitionFilePageStoreManager.start();
+ filePageStoreManager.start();
} catch (IgniteInternalCheckedException e) {
throw new StorageException("Error starting file page store manager", e);
}
+ partitionMetaManager = new PartitionMetaManager(ioRegistry, pageSize);
Review Comment:
Decided to leave it as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924389228
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,27 +17,29 @@
package org.apache.ignite.internal.pagememory.persistence;
+import static org.apache.ignite.internal.pagememory.PageIdAllocator.FLAG_AUX;
+import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
+
import java.lang.invoke.MethodHandles;
import java.lang.invoke.VarHandle;
import java.util.UUID;
-import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
import org.apache.ignite.internal.tostring.S;
import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
- private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
- PartitionMeta.class,
- "pageCount"
- );
+ private static final VarHandle PAGE_COUNT;
Review Comment:
I said "could"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921214714
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointWorkflow.java:
##########
@@ -279,56 +288,69 @@ public List<CheckpointListener> collectCheckpointListeners(Collection<? extends
private DataRegionsDirtyPages beginCheckpoint(
Collection<? extends DataRegion<PersistentPageMemory>> dataRegions,
CompletableFuture<?> allowToReplace
- ) {
- Collection<IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>>> pages = new ArrayList<>(dataRegions.size());
+ ) throws IgniteInternalCheckedException {
+ Collection<DataRegionDirtyPages<CollectionDirtyPages>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
for (DataRegion<PersistentPageMemory> dataRegion : dataRegions) {
- Collection<FullPageId> dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
+ CollectionDirtyPages dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
- pages.add(new IgniteBiTuple<>(dataRegion.pageMemory(), dirtyPages));
+ dataRegionsDirtyPages.add(new DataRegionDirtyPages<>(dataRegion.pageMemory(), dirtyPages));
}
- return new DataRegionsDirtyPages(pages);
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages) {
+ for (GroupPartitionId dirtyPartition : dataRegionDirtyPages.dirtyPages.partitionIds()) {
+ partitionFilePageStoreManager.getStore(dirtyPartition.getGroupId(), dirtyPartition.getPartitionId()).updateMetaPageCount();
+ }
+ }
+
+ return new DataRegionsDirtyPages(dataRegionsDirtyPages);
}
CheckpointDirtyPages createAndSortCheckpointDirtyPages(
DataRegionsDirtyPages dataRegionsDirtyPages
) throws IgniteInternalCheckedException {
- List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> checkpointPages = new ArrayList<>();
+ List<DataRegionDirtyPages<ArrayDirtyPages>> checkpointDirtyPages = new ArrayList<>();
int realPagesArrSize = 0;
- for (IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>> regionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
- FullPageId[] checkpointRegionDirtyPages = new FullPageId[regionDirtyPages.getValue().size()];
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
+ FullPageId[] pageIds = new FullPageId[dataRegionDirtyPages.dirtyPages.pageIds().size()];
int pagePos = 0;
- for (FullPageId dirtyPage : regionDirtyPages.getValue()) {
+ for (FullPageId dirtyPage : dataRegionDirtyPages.dirtyPages.pageIds()) {
assert realPagesArrSize++ != dataRegionsDirtyPages.dirtyPageCount :
"Incorrect estimated dirty pages number: " + dataRegionsDirtyPages.dirtyPageCount;
- checkpointRegionDirtyPages[pagePos++] = dirtyPage;
+ pageIds[pagePos++] = dirtyPage;
Review Comment:
It will not work here because we are iterating over several collections, or rather, by concatenating sets of dirty pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921217997
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryTableStorage.java:
##########
@@ -81,22 +80,36 @@ public void start() throws StorageException {
protected PersistentPageMemoryPartitionStorage createPartitionStorage(int partId) throws StorageException {
TableView tableView = tableCfg.value();
- FilePageStore partitionFilePageStore = ensurePartitionFilePageStore(tableView, partId);
+ PartitionFilePageStore partitionFilePageStore = ensurePartitionFilePageStore(tableView, partId);
CheckpointTimeoutLock checkpointTimeoutLock = dataRegion.checkpointManager().checkpointTimeoutLock();
checkpointTimeoutLock.checkpointReadLock();
try {
- PartitionMeta partitionMeta = getOrCreatePartitionMeta(tableView, partId, partitionFilePageStore);
+ PersistentPageMemory persistentPageMemory = dataRegion.pageMemory();
- TableFreeList tableFreeList = createTableFreeList(tableView, partId, partitionMeta);
+ int grpId = groupId(tableView);
+
+ PartitionMeta meta = partitionFilePageStore.meta();
+
+ if (meta.isCreated()) {
Review Comment:
Fix it
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryTableStorage.java:
##########
@@ -226,8 +169,8 @@ TableFreeList createTableFreeList(
partId,
dataRegion.pageMemory(),
PageLockListenerNoOp.INSTANCE,
- partitionMeta.reuseListRoot.pageId(),
- partitionMeta.allocated,
+ partitionMeta.reuseListRootPageId(),
+ partitionMeta.isCreated(),
Review Comment:
Remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920980123
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
}
/**
- * Returns a queue of dirty pages to be written to a checkpoint.
+ * Returns a queue of dirty page IDs to be written to a checkpoint.
*/
- public CheckpointDirtyPagesQueue toQueue() {
- return new CheckpointDirtyPagesQueue();
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> toDirtyPageIdQueue() {
+ List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> dirtyPageIds = dirtyPages.stream()
+ .map(pages -> new IgniteBiTuple<>(pages.pageMemory, pages.dirtyPages.pageIds))
+ .collect(toList());
+
+ return new IgniteConcurrentMultiPairQueue<>(dirtyPageIds);
}
/**
- * Looks for dirty page views for a specific group and partition.
+ * Returns a queue of dirty partition IDs to be modified to a checkpoint.
+ */
+ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> toDirtyPartitionIdQueue() {
Review Comment:
Let's talk in person.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921126340
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/CollectionDirtyPages.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import java.util.Collection;
+import org.apache.ignite.internal.pagememory.FullPageId;
+
+/**
+ * Dirty pages collection container.
+ */
+public class CollectionDirtyPages {
+ private final Collection<FullPageId> pageIds;
Review Comment:
Discussed personally, no need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921924987
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PageMemoryStorageIoModule.java:
##########
@@ -38,8 +37,7 @@ public Collection<IoVersions<?>> ioVersions() {
TableMetaIo.VERSIONS,
TableInnerIo.VERSIONS,
TableLeafIo.VERSIONS,
- TableDataIo.VERSIONS,
- PartitionMetaIo.VERSIONS
+ TableDataIo.VERSIONS
Review Comment:
Please no, it's huge already
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920962866
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -874,19 +871,37 @@ private void copyInBuffer(long absPtr, ByteBuffer tmpBuf) {
}
/**
- * Get current prartition generation tag.
+ * Get current partition generation tag.
*
* @param seg Segment.
- * @param fullId Full page id.
+ * @param fullPageId Full page id.
* @return Current partition generation tag.
*/
- private int generationTag(Segment seg, FullPageId fullId) {
+ private int generationTag(Segment seg, FullPageId fullPageId) {
return seg.partGeneration(
- fullId.groupId(),
- partitionId(fullId.pageId())
+ fullPageId.groupId(),
+ partitionId(fullPageId.pageId())
);
}
+ /**
+ * Get current partition generation tag.
+ *
+ * @param fullPageId Full page id.
+ * @return Current partition generation tag.
+ */
+ public int generationTag(FullPageId fullPageId) {
Review Comment:
Since we are writing page **PartitionMetaIo** out of **PersistentPageMemory** and directly to a file, does it seem like we have to write the same generation for it, or can we omit that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921201820
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -161,109 +164,24 @@ public CheckpointDirtyPagesQueue toQueue() {
return null;
}
- List<FullPageId> pageIds = dirtyPages.get(regionIndex).getValue();
+ FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
- if (fromPosition == pageIds.size() - 1 || !equalsByGroupAndPartition(pageIds.get(fromPosition), pageIds.get(fromPosition + 1))) {
+ if (fromPosition == pageIds.length - 1 || !equalsByGroupAndPartition(pageIds[fromPosition], pageIds[fromPosition + 1])) {
return new CheckpointDirtyPagesView(regionIndex, fromPosition, fromPosition);
}
- FullPageId startPageId = pageIds.get(fromPosition);
+ FullPageId startPageId = pageIds[fromPosition];
FullPageId endPageId = new FullPageId(pageId(partitionId(startPageId.pageId()) + 1, (byte) 0, 0), startPageId.groupId());
- int toPosition = binarySearch(pageIds.subList(fromPosition, pageIds.size()), endPageId, DIRTY_PAGE_COMPARATOR);
+ int toPosition = binarySearch(pageIds, fromPosition, pageIds.length, endPageId, DIRTY_PAGE_COMPARATOR);
toPosition = toPosition > 0 ? toPosition - 1 : -toPosition - 2;
Review Comment:
Now **toPosition** is not included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923200521
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
Review Comment:
My english is bad. Wanted to say:
If the file exists (not empty), then at the beginning it will read the header and create a FilePageStore based on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923218808
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -229,15 +242,15 @@ public PersistentPageMemory pageMemory() {
* Returns the size of the view.
*/
public int size() {
- return toPosition - fromPosition + 1;
+ return toPosition - fromPosition;
}
private CheckpointDirtyPages owner() {
return CheckpointDirtyPages.this;
}
- private boolean isToPositionLast() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length - 1;
+ private boolean needsNextRegion() {
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
Review Comment:
What does it have to do with GC? Your object will be 4 bytes bigger. No new objects required. You can store both index and dirty pages collection, it's not a problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923201819
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
+ *
+ * @param filePath File page store path.
+ * @param readIntoBuffer Buffer for reading {@link FilePageStoreHeader header} from {@code filePath}.
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923215741
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
Review Comment:
I think the general method is unnecessary. It takes space, it requires tests and so on. Name - anything meaningful. Like "getPartitionView" or something. Anyways, why is it "view" and not "slice"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923226994
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -229,15 +242,15 @@ public PersistentPageMemory pageMemory() {
* Returns the size of the view.
*/
public int size() {
- return toPosition - fromPosition + 1;
+ return toPosition - fromPosition;
}
private CheckpointDirtyPages owner() {
return CheckpointDirtyPages.this;
}
- private boolean isToPositionLast() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length - 1;
+ private boolean needsNextRegion() {
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
Review Comment:
But why? In general objects are not a problem worth going down the path of pure OOP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r925257664
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -232,8 +230,6 @@ private boolean read0(
// If page was not written yet, nothing to read.
if (n < 0) {
pageBuf.put(new byte[pageBuf.remaining()]);
Review Comment:
Maybe later, ok.
>we want to write the page to disk, but we only wrote it to a temporary buffer for writing it, and another thread wants to read it.
Other thread should read from buffer then. We need a page content, not an empty page in this case, right? Looks kinda buggy to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923118797
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,33 +17,54 @@
package org.apache.ignite.internal.pagememory.persistence;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import org.apache.ignite.internal.tostring.S;
+import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
+ private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
+ PartitionMeta.class,
+ "pageCount"
+ );
+
+ private static final VarHandle META_SNAPSHOT_VH;
+
+ static {
+ try {
+ META_SNAPSHOT_VH = MethodHandles.lookup().findVarHandle(PartitionMeta.class, "metaSnapshot", PartitionMetaSnapshot.class);
+ } catch (ReflectiveOperationException e) {
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
private volatile long treeRootPageId;
private volatile long reuseListRootPageId;
private volatile int pageCount;
- private final boolean created;
+ private volatile PartitionMetaSnapshot metaSnapshot;
/**
* Constructor.
*
+ * @param checkpointId Checkpoint ID.
* @param treeRootPageId Tree root page ID.
* @param reuseListRootPageId Reuse list root page ID.
* @param pageCount Count of pages in the partition.
- * @param created {@code True} if the partition meta was created, {@code false} if it was read from a partition file.
*/
- public PartitionMeta(long treeRootPageId, long reuseListRootPageId, int pageCount, boolean created) {
+ public PartitionMeta(@Nullable UUID checkpointId, long treeRootPageId, long reuseListRootPageId, int pageCount) {
Review Comment:
It will be **null** until the first checkpoint occurs. Do you think this can be ignored?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,33 +17,54 @@
package org.apache.ignite.internal.pagememory.persistence;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import org.apache.ignite.internal.tostring.S;
+import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
+ private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
+ PartitionMeta.class,
+ "pageCount"
+ );
+
+ private static final VarHandle META_SNAPSHOT_VH;
+
+ static {
+ try {
+ META_SNAPSHOT_VH = MethodHandles.lookup().findVarHandle(PartitionMeta.class, "metaSnapshot", PartitionMetaSnapshot.class);
+ } catch (ReflectiveOperationException e) {
+ throw new ExceptionInInitializerError(e);
+ }
+ }
+
private volatile long treeRootPageId;
private volatile long reuseListRootPageId;
private volatile int pageCount;
- private final boolean created;
+ private volatile PartitionMetaSnapshot metaSnapshot;
/**
* Constructor.
*
+ * @param checkpointId Checkpoint ID.
* @param treeRootPageId Tree root page ID.
* @param reuseListRootPageId Reuse list root page ID.
* @param pageCount Count of pages in the partition.
- * @param created {@code True} if the partition meta was created, {@code false} if it was read from a partition file.
*/
- public PartitionMeta(long treeRootPageId, long reuseListRootPageId, int pageCount, boolean created) {
+ public PartitionMeta(@Nullable UUID checkpointId, long treeRootPageId, long reuseListRootPageId, int pageCount) {
Review Comment:
Ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923169668
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -108,33 +100,54 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
* @param partId Partition ID.
*/
public @Nullable CheckpointDirtyPagesView findView(int grpId, int partId) {
- if (dirtyPages.isEmpty()) {
- return null;
+ for (int i = 0; i < dirtyPages.size(); i++) {
+ CheckpointDirtyPagesView view = findView(i, grpId, partId);
+
+ if (view != null) {
+ return view;
+ }
}
- FullPageId startPageId = new FullPageId(pageId(partId, (byte) 0, 0), grpId);
- FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
+ return null;
+ }
+ /**
+ * Looks for dirty page IDs views for a specific group and partition.
+ *
+ * @param pageMemory Page memory.
+ * @param grpId Group ID.
+ * @param partId Partition ID.
+ */
+ public @Nullable CheckpointDirtyPagesView findView(PersistentPageMemory pageMemory, int grpId, int partId) {
for (int i = 0; i < dirtyPages.size(); i++) {
- FullPageId[] pageIds = dirtyPages.get(i).dirtyPages.pageIds;
+ if (dirtyPages.get(i).pageMemory == pageMemory) {
+ return findView(i, grpId, partId);
+ }
+ }
- int fromIndex = binarySearch(pageIds, startPageId, DIRTY_PAGE_COMPARATOR);
+ return null;
Review Comment:
Purely technically possible, I can add a check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924393384
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -240,35 +247,25 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
};
}
- /**
- * Updates the partitions meta page.
- *
- * @throws IgniteInternalCheckedException If failed.
- */
- private void updatePartitions() throws IgniteInternalCheckedException {
- Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+ private void writePartitionMeta(GroupPartitionId partitionId, ByteBuffer buffer) throws IgniteInternalCheckedException {
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
- while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
- GroupPartitionId partitionId = queueResult.getValue();
+ partitionMetaManager.writeMetaToBuffer(partitionId, partitionMetaSnapshot, buffer.rewind());
- ByteBuffer buffer = threadBuf.get();
+ FullPageId fullPageId = new FullPageId(partitionMetaPageId(partitionId.getPartitionId()), partitionId.getGroupId());
- PersistentPageMemory pageMemory = queueResult.getKey();
-
- PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
-
- partitionMetaManager.writeMetaToBuffer(partitionId, pageMemory, partitionMetaSnapshot, buffer.rewind());
+ PageStore store = pageWriter.write(fullPageId, buffer.rewind());
- FullPageId fullPageId = new FullPageId(
- pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId()),
- partitionId.getGroupId()
- );
+ checkpointProgress.writtenPagesCounter().incrementAndGet();
- PageStore store = pageWriter.write(fullPageId, buffer.rewind());
+ updStores.computeIfAbsent(store, k -> new LongAdder()).increment();
+ }
- checkpointProgress.writtenPagesCounter().incrementAndGet();
+ private static boolean isPartitionChange(@Nullable GroupPartitionId partitionId, FullPageId pageId) {
Review Comment:
Has the partition changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r922296663
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/PartitionFilePageStore.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * File page store with partition meta.
+ */
+public class PartitionFilePageStore implements PageStore {
+ private final FilePageStore filePageStore;
+
+ private final PartitionMeta partitionMeta;
+
+ /**
+ * Constructor.
+ *
+ * @param filePageStore File page store.
+ * @param partitionMeta Partition meta.
+ */
+ public PartitionFilePageStore(FilePageStore filePageStore, PartitionMeta partitionMeta) {
+ this.filePageStore = filePageStore;
+ this.partitionMeta = partitionMeta;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void addWriteListener(PageWriteListener listener) {
+ filePageStore.addWriteListener(listener);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void removeWriteListener(PageWriteListener listener) {
+ filePageStore.removeWriteListener(listener);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void stop(boolean clean) throws IgniteInternalCheckedException {
+ filePageStore.stop(clean);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public long allocatePage() throws IgniteInternalCheckedException {
+ return filePageStore.allocatePage();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public int pages() {
+ return filePageStore.pages();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws IgniteInternalCheckedException {
+ return filePageStore.read(pageId, pageBuf, keepCrc);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void write(long pageId, ByteBuffer pageBuf, int tag, boolean calculateCrc) throws IgniteInternalCheckedException {
+ filePageStore.write(pageId, pageBuf, tag, calculateCrc);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void sync() throws IgniteInternalCheckedException {
+ filePageStore.sync();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean exists() {
+ return filePageStore.exists();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void ensure() throws IgniteInternalCheckedException {
+ filePageStore.ensure();
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void close() throws IOException {
+ filePageStore.close();
+ }
+
+ /**
+ * Returns partition meta.
+ */
+ public PartitionMeta meta() {
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923122218
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -86,24 +113,89 @@ public int pageCount() {
}
/**
- * Sets count of pages in the partition.
- *
- * @param pageCount Count of pages in the partition.
+ * Increases the number of pages in a partition.
*/
- public void pageCount(int pageCount) {
- this.pageCount = pageCount;
+ public void incrementPageCount(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ PAGE_COUNT_UPDATER.incrementAndGet(this);
}
/**
- * Returns {@code true} if the partition meta was created, {@code false} if it was read from a partition file.
+ * Returns the latest snapshot of the partition meta.
+ *
+ * @param checkpointId Checkpoint ID.
*/
- public boolean isCreated() {
- return created;
+ public PartitionMetaSnapshot metaSnapshot(@Nullable UUID checkpointId) {
+ updateSnapshot(checkpointId);
+
+ return metaSnapshot;
+ }
+
+ private void updateSnapshot(@Nullable UUID checkpointId) {
+ PartitionMetaSnapshot current = this.metaSnapshot;
+
+ if (current.checkpointId != checkpointId) {
+ META_SNAPSHOT_VH.compareAndSet(this, current, new PartitionMetaSnapshot(checkpointId, this));
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923231805
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
Review Comment:
I don't understand what you're suggesting, please show me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923117537
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,33 +17,54 @@
package org.apache.ignite.internal.pagememory.persistence;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import org.apache.ignite.internal.tostring.S;
+import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
+ private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923202336
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
+ *
+ * @param filePath File page store path.
+ * @param readIntoBuffer Buffer for reading {@link FilePageStoreHeader header} from {@code filePath}.
+ * @return File page store.
+ * @throws IgniteInternalCheckedException if failed
+ */
+ public FilePageStore createPageStore(Path filePath, ByteBuffer readIntoBuffer) throws IgniteInternalCheckedException {
+ assert readIntoBuffer.remaining() == pageSize : readIntoBuffer.remaining();
+
+ if (!Files.exists(filePath)) {
+ return createPageStore(filePath, new FilePageStoreHeader(latestVersion, pageSize));
+ }
+
+ try (FileIo fileIo = fileIoFactory.create(filePath)) {
+ FilePageStoreHeader header = readHeader(fileIo, readIntoBuffer);
+
+ if (header == null) {
+ header = new FilePageStoreHeader(latestVersion, pageSize);
+ }
+
+ return createPageStore(filePath, header);
+ } catch (IOException e) {
+ throw new IgniteInternalCheckedException("Error while creating file page store [file=" + filePath + "]:", e);
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921206632
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -59,7 +66,10 @@ public class CheckpointPagesWriter implements Runnable {
* <p>Overall pages to write may be greater than this queue, since it may be necessary to retire write some pages due to unsuccessful
* page write lock acquisition
*/
- private final CheckpointDirtyPagesQueue writePageIds;
+ private final IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds;
+
+ /** Queue of dirty partition IDs to update {@link PartitionMetaIo} under this task. */
Review Comment:
Try to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923170221
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
Review Comment:
This is a tricky question. I think that a specific method for reading pages is required, that will check file size instead of logical pages count. So it's not 1 or 3, it's more like 2.1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920979005
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -17,89 +17,92 @@
package org.apache.ignite.internal.pagememory.persistence.checkpoint;
-import static java.util.Collections.binarySearch;
+import static java.util.Arrays.binarySearch;
import static java.util.stream.Collectors.toList;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
import static org.apache.ignite.internal.pagememory.util.PageIdUtils.partitionId;
-import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.RandomAccess;
-import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
import org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory;
+import org.apache.ignite.internal.util.IgniteConcurrentMultiPairQueue;
import org.apache.ignite.lang.IgniteBiTuple;
import org.jetbrains.annotations.Nullable;
/**
- * Sorted dirty pages from data regions that should be checkpointed.
- *
- * <p>Dirty pages should be sorted by groupId -> partitionId -> pageIdx.
+ * Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs that should be
+ * checkpointed.
*/
class CheckpointDirtyPages {
- /** Dirty page ID comparator. */
+ /** Dirty page ID comparator by groupId -> partitionId -> pageIdx. */
static final Comparator<FullPageId> DIRTY_PAGE_COMPARATOR = Comparator
.comparingInt(FullPageId::groupId)
.thenComparingLong(FullPageId::effectivePageId);
/** Empty checkpoint dirty pages. */
static final CheckpointDirtyPages EMPTY = new CheckpointDirtyPages(List.of());
- /** Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx. */
- private final List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages;
-
- /** Total number of dirty pages. */
- private final int dirtyPagesCount;
+ /** Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs. */
+ private final List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages;
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(Map<PersistentPageMemory, List<FullPageId>> dirtyPages) {
- this(dirtyPages.entrySet().stream().map(e -> new IgniteBiTuple<>(e.getKey(), e.getValue())).collect(toList()));
+ public CheckpointDirtyPages(Map<PersistentPageMemory, ArrayDirtyPages> dirtyPages) {
+ this(dirtyPages.entrySet().stream()
+ .map(e -> new DataRegionDirtyPages<>(e.getKey(), e.getValue()))
+ .collect(toList())
+ );
}
/**
* Constructor.
*
- * @param dirtyPages Sorted dirty pages from data regions by groupId -> partitionId -> pageIdx.
+ * @param dirtyPages Dirty pages of data regions, with sorted page IDs by {@link #DIRTY_PAGE_COMPARATOR} and unsorted partition IDs.
*/
- public CheckpointDirtyPages(List<IgniteBiTuple<PersistentPageMemory, List<FullPageId>>> dirtyPages) {
+ public CheckpointDirtyPages(List<DataRegionDirtyPages<ArrayDirtyPages>> dirtyPages) {
assert dirtyPages instanceof RandomAccess : dirtyPages;
this.dirtyPages = dirtyPages;
-
- int count = 0;
-
- for (IgniteBiTuple<PersistentPageMemory, List<FullPageId>> pages : dirtyPages) {
- assert !pages.getValue().isEmpty() : pages.getKey();
- assert pages.getValue() instanceof RandomAccess : pages.getValue();
-
- count += pages.getValue().size();
- }
-
- dirtyPagesCount = count;
}
/**
- * Returns total number of dirty pages.
+ * Returns total number of dirty page IDs.
*/
public int dirtyPagesCount() {
- return dirtyPagesCount;
+ return dirtyPages.stream().mapToInt(pages -> pages.dirtyPages.pageIds.length).sum();
Review Comment:
It is called only once, and the field can be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924402125
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -237,9 +266,41 @@ private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean
}
}
+ /**
+ * Writes a page unlike {@link #write(long, ByteBuffer, boolean)}, checks the page offset in the file not logically (pageOffset <=
+ * {@link #pages()} * {@link #pageSize}) but physically (pageOffset <= {@link #size()}), which can affect performance when used in
+ * production code.
+ *
+ * @param pageId Page ID.
+ * @param pageBuf Page buffer to write from.
+ * @param calculateCrc If {@code false} crc calculation will be forcibly skipped.
+ * @throws IgniteInternalCheckedException If page writing failed (IO error occurred).
+ */
+ public void writeByPhysicalOffset(long pageId, ByteBuffer pageBuf, boolean calculateCrc) throws IgniteInternalCheckedException {
Review Comment:
That's correct.
I meant that a place for a meta page should be allocated before writing that meta page into a file. This way we wouldn't violate any internal assertions. And we _probably_ wouldn't need to manually set a page count to 1. Which is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924402000
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -250,7 +215,7 @@ private CheckpointDirtyPages owner() {
}
private boolean needsNextRegion() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.length;
Review Comment:
But for what? according to me, you need to add only what you need, do you insist that it be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924399186
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -117,7 +120,7 @@ public class CheckpointPagesWriter implements Runnable {
IgniteLogger log,
CheckpointMetricsTracker tracker,
IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds,
- IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> updatePartitionIds,
+ Set<GroupPartitionId> savedPartitionMetas,
Review Comment:
Ok, makes sense now, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924398652
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -250,7 +215,7 @@ private CheckpointDirtyPages owner() {
}
private boolean needsNextRegion() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.length;
Review Comment:
Does this imply that you can't add another field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924415744
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -237,9 +266,41 @@ private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean
}
}
+ /**
+ * Writes a page unlike {@link #write(long, ByteBuffer, boolean)}, checks the page offset in the file not logically (pageOffset <=
+ * {@link #pages()} * {@link #pageSize}) but physically (pageOffset <= {@link #size()}), which can affect performance when used in
+ * production code.
+ *
+ * @param pageId Page ID.
+ * @param pageBuf Page buffer to write from.
+ * @param calculateCrc If {@code false} crc calculation will be forcibly skipped.
+ * @throws IgniteInternalCheckedException If page writing failed (IO error occurred).
+ */
+ public void writeByPhysicalOffset(long pageId, ByteBuffer pageBuf, boolean calculateCrc) throws IgniteInternalCheckedException {
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r925271105
##########
modules/page-memory/src/test/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemoryNoLoadTest.java:
##########
@@ -277,11 +309,47 @@ private static FilePageStoreManager createFilePageStoreManager(Path storagePath)
return new FilePageStoreManager(log, "test", storagePath, new RandomAccessFileIoFactory(), PAGE_SIZE);
}
- private static void initGroupFilePageStores(FilePageStoreManager filePageStoreManager) throws Exception {
- filePageStoreManager.initialize("Test", GRP_ID, PARTITION_ID + 1);
+ private static void initGroupFilePageStores(
+ FilePageStoreManager filePageStoreManager,
+ PartitionMetaManager partitionMetaManager,
+ CheckpointManager checkpointManager
+ ) throws Exception {
+ int partitions = PARTITION_ID + 1;
+
+ filePageStoreManager.initialize("Test", GRP_ID, partitions);
+
+ checkpointManager.checkpointTimeoutLock().checkpointReadLock();
+
+ try {
+ for (int i = 0; i < partitions; i++) {
+ FilePageStore filePageStore = filePageStoreManager.getStore(GRP_ID, i);
+
+ filePageStore.ensure();
+
+ GroupPartitionId groupPartitionId = new GroupPartitionId(GRP_ID, i);
- for (FilePageStore filePageStore : filePageStoreManager.getStores(GRP_ID)) {
- filePageStore.ensure();
+ CheckpointProgress currentProgress = checkpointManager.currentProgress();
+
+ PartitionMeta partitionMeta = partitionMetaManager.readOrCreateMeta(
+ currentProgress == null ? null : currentProgress.id(),
+ groupPartitionId,
+ filePageStore
+ );
+
+ partitionMetaManager.addMeta(groupPartitionId, partitionMeta);
+
+ filePageStore.setPageAllocationListener(pageIdx -> {
+ assert checkpointManager.checkpointTimeoutLock().checkpointLockIsHeldByThread();
+
+ CheckpointProgress curr = checkpointManager.currentProgress();
Review Comment:
What does "currentProgress()" return if checkpoint is already finished?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointManager.java:
##########
@@ -191,6 +199,13 @@ public CheckpointProgress forceCheckpoint(String reason) {
return checkpointer.scheduleCheckpoint(0, reason);
}
+ /**
+ * Returns progress of current checkpoint, last finished one or {@code null}, if checkpoint has never started.
+ */
+ public @Nullable CheckpointProgress currentProgress() {
Review Comment:
Now I see an answer to my previous question. Can you please rename it to "lastCheckpointProgress" so that no one would have the same question?
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -199,17 +231,41 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
assert getType(buf) != 0 : "Invalid state. Type is 0! pageId = " + hexLong(pageId);
assert getVersion(buf) != 0 : "Invalid state. Version is 0! pageId = " + hexLong(pageId);
+ assert fullPageId.pageIdx() != 0 : "Invalid pageIdx. Index is 0! pageId = " + hexLong(pageId);
+ assert !(ioRegistry.resolve(buf) instanceof PartitionMetaIo) : "Invalid IO type. pageId = " + hexLong(pageId);
if (flag(pageId) == FLAG_DATA) {
tracker.onDataPageWritten();
}
checkpointProgress.writtenPagesCounter().incrementAndGet();
- PageStore store = pageWriter.write(fullPageId, buf, tag);
+ PageStore store = pageWriter.write(fullPageId, buf);
updStores.computeIfAbsent(store, k -> new LongAdder()).increment();
}
};
}
+
+ private void writePartitionMeta(GroupPartitionId partitionId, ByteBuffer buffer) throws IgniteInternalCheckedException {
Review Comment:
Note for the future: I don't like a complete lack of comments in your code. You're writing complicated things. You and I know what these things mean. Other people don't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924390811
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -250,7 +215,7 @@ private CheckpointDirtyPages owner() {
}
private boolean needsNextRegion() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.length;
Review Comment:
Yes, the **regionIndex** is needed to search for the **getNextPartitionView**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924386997
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/FullPageId.java:
##########
@@ -185,4 +185,6 @@ public int partitionId() {
public int pageIdx() {
return PageIdUtils.pageIndex(pageId);
}
+
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921170910
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -113,29 +116,29 @@ public CheckpointDirtyPagesQueue toQueue() {
FullPageId endPageId = new FullPageId(pageId(partId + 1, (byte) 0, 0), grpId);
for (int i = 0; i < dirtyPages.size(); i++) {
- List<FullPageId> pageIds = dirtyPages.get(i).getValue();
+ FullPageId[] pageIds = dirtyPages.get(i).dirtyPages.pageIds;
Review Comment:
There is a description for the method that says that we get a view of all pages for a specific partition and group.
Added a method with one more method.
> Also, passing a DataRegion or PageMemory instance here would help you avoiding unnecessary binary searches.
Removed constructor.
> Also, why exactly do we have a list of arrays instead of the Map, as in the original constructor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921135808
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PersistentPageMemory.java:
##########
@@ -874,19 +871,37 @@ private void copyInBuffer(long absPtr, ByteBuffer tmpBuf) {
}
/**
- * Get current prartition generation tag.
+ * Get current partition generation tag.
*
* @param seg Segment.
- * @param fullId Full page id.
+ * @param fullPageId Full page id.
* @return Current partition generation tag.
*/
- private int generationTag(Segment seg, FullPageId fullId) {
+ private int generationTag(Segment seg, FullPageId fullPageId) {
return seg.partGeneration(
- fullId.groupId(),
- partitionId(fullId.pageId())
+ fullPageId.groupId(),
+ partitionId(fullPageId.pageId())
);
}
+ /**
+ * Get current partition generation tag.
+ *
+ * @param fullPageId Full page id.
+ * @return Current partition generation tag.
+ */
+ public int generationTag(FullPageId fullPageId) {
Review Comment:
Got rid of the method and associated code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921207427
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
Review Comment:
is fine?
`Updates the partitions meta page.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921144823
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPages.java:
##########
@@ -30,7 +30,7 @@
* View of pages which should be stored during current checkpoint.
*/
public class CheckpointPages {
- private final Set<FullPageId> segCheckpointPages;
+ private final Set<FullPageId> segmentDirtyPages;
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921235136
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointWorkflow.java:
##########
@@ -279,56 +288,69 @@ public List<CheckpointListener> collectCheckpointListeners(Collection<? extends
private DataRegionsDirtyPages beginCheckpoint(
Collection<? extends DataRegion<PersistentPageMemory>> dataRegions,
CompletableFuture<?> allowToReplace
- ) {
- Collection<IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>>> pages = new ArrayList<>(dataRegions.size());
+ ) throws IgniteInternalCheckedException {
+ Collection<DataRegionDirtyPages<CollectionDirtyPages>> dataRegionsDirtyPages = new ArrayList<>(dataRegions.size());
for (DataRegion<PersistentPageMemory> dataRegion : dataRegions) {
- Collection<FullPageId> dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
+ CollectionDirtyPages dirtyPages = dataRegion.pageMemory().beginCheckpoint(allowToReplace);
- pages.add(new IgniteBiTuple<>(dataRegion.pageMemory(), dirtyPages));
+ dataRegionsDirtyPages.add(new DataRegionDirtyPages<>(dataRegion.pageMemory(), dirtyPages));
}
- return new DataRegionsDirtyPages(pages);
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages) {
+ for (GroupPartitionId dirtyPartition : dataRegionDirtyPages.dirtyPages.partitionIds()) {
+ partitionFilePageStoreManager.getStore(dirtyPartition.getGroupId(), dirtyPartition.getPartitionId()).updateMetaPageCount();
+ }
+ }
+
+ return new DataRegionsDirtyPages(dataRegionsDirtyPages);
}
CheckpointDirtyPages createAndSortCheckpointDirtyPages(
DataRegionsDirtyPages dataRegionsDirtyPages
) throws IgniteInternalCheckedException {
- List<IgniteBiTuple<PersistentPageMemory, FullPageId[]>> checkpointPages = new ArrayList<>();
+ List<DataRegionDirtyPages<ArrayDirtyPages>> checkpointDirtyPages = new ArrayList<>();
int realPagesArrSize = 0;
- for (IgniteBiTuple<PersistentPageMemory, Collection<FullPageId>> regionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
- FullPageId[] checkpointRegionDirtyPages = new FullPageId[regionDirtyPages.getValue().size()];
+ for (DataRegionDirtyPages<CollectionDirtyPages> dataRegionDirtyPages : dataRegionsDirtyPages.dirtyPages) {
+ FullPageId[] pageIds = new FullPageId[dataRegionDirtyPages.dirtyPages.pageIds().size()];
int pagePos = 0;
- for (FullPageId dirtyPage : regionDirtyPages.getValue()) {
+ for (FullPageId dirtyPage : dataRegionDirtyPages.dirtyPages.pageIds()) {
assert realPagesArrSize++ != dataRegionsDirtyPages.dirtyPageCount :
"Incorrect estimated dirty pages number: " + dataRegionsDirtyPages.dirtyPageCount;
- checkpointRegionDirtyPages[pagePos++] = dirtyPage;
+ pageIds[pagePos++] = dirtyPage;
}
// Some pages may have been already replaced.
if (pagePos == 0) {
continue;
- } else if (pagePos != checkpointRegionDirtyPages.length) {
- checkpointPages.add(new IgniteBiTuple<>(regionDirtyPages.getKey(), Arrays.copyOf(checkpointRegionDirtyPages, pagePos)));
- } else {
- checkpointPages.add(new IgniteBiTuple<>(regionDirtyPages.getKey(), checkpointRegionDirtyPages));
+ } else if (pagePos != pageIds.length) {
+ pageIds = Arrays.copyOf(pageIds, pagePos);
}
+
+ assert !dataRegionDirtyPages.dirtyPages.partitionIds().isEmpty();
+
+ GroupPartitionId[] partitionIds = dataRegionDirtyPages.dirtyPages.partitionIds().toArray(new GroupPartitionId[0]);
+
+ checkpointDirtyPages.add(new DataRegionDirtyPages<>(
+ dataRegionDirtyPages.pageMemory,
+ new ArrayDirtyPages(pageIds, partitionIds)
+ ));
}
- List<ForkJoinTask<?>> parallelSortTasks = checkpointPages.stream()
- .map(IgniteBiTuple::getValue)
- .filter(pages -> pages.length >= PARALLEL_SORT_THRESHOLD)
- .map(pages -> parallelSortThreadPool.submit(() -> Arrays.parallelSort(pages, DIRTY_PAGE_COMPARATOR)))
+ List<ForkJoinTask<?>> parallelSortTasks = checkpointDirtyPages.stream()
Review Comment:
I added a few comments, I hope they are enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r925264419
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -232,8 +230,6 @@ private boolean read0(
// If page was not written yet, nothing to read.
if (n < 0) {
pageBuf.put(new byte[pageBuf.remaining()]);
Review Comment:
Let's investigate it later, ok? Please create a separate JIRA issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923137432
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
+ filePageStore.pages(1);
+ }
+
+ boolean read = filePageStore.read(partitionMetaPageId, buffer, false);
+
+ assert read : filePageStore.filePath();
+
+ return createMeta(ioRegistry.resolve(bufferAddr), bufferAddr, checkpointId);
+ } else {
+ // Creates and writes a partition meta.
+ PartitionMetaIo partitionMetaIo = PartitionMetaIo.VERSIONS.latest();
+ partitionMetaIo.initNewPage(bufferAddr, partitionMetaPageId, pageSize);
+
+ // Because we will now write this page.
+ partitionMetaIo.setPageCount(bufferAddr, 1);
+
+ filePageStore.pages(1);
+
+ filePageStore.write(partitionMetaPageId, buffer.rewind(), true);
+
+ filePageStore.sync();
+
+ return createMeta(partitionMetaIo, bufferAddr, checkpointId);
+ }
+ } finally {
+ freeBuffer(buffer);
+ }
+ }
+
+ /**
+ * Writes the partition meta to the buffer.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param partitionMeta Snapshot of the partition meta.
+ * @param writeToBuffer Direct byte buffer to write partition meta.
+ */
+ public void writeMetaToBuffer(
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ PartitionMetaSnapshot partitionMeta,
+ ByteBuffer writeToBuffer
+ ) {
+ assert writeToBuffer.remaining() == pageSize : writeToBuffer.remaining();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
Review Comment:
Sounds reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923268846
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
Review Comment:
Decided to leave it as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924395866
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -182,23 +198,36 @@ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws Ign
* @param pageBuf Page buffer to read into.
* @param checkCrc Check CRC on page.
* @param keepCrc By default reading zeroes CRC which was on file, but you can keep it in pageBuf if set keepCrc.
+ * @param checkPageOffsetLogically Check page offset by {@link #allocatedBytes} or {@link #size}.
* @return {@code true} if page has been read successfully, {@code false} if page hasn't been written yet.
* @throws IgniteInternalCheckedException If reading failed (IO error occurred).
*/
- private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean keepCrc) throws IgniteInternalCheckedException {
+ private boolean read0(
+ long pageId,
+ ByteBuffer pageBuf,
+ boolean checkCrc,
+ boolean keepCrc,
+ boolean checkPageOffsetLogically
+ ) throws IgniteInternalCheckedException {
ensure();
try {
- long off = pageOffset(pageId);
-
assert pageBuf.capacity() == pageSize : pageBuf.capacity();
assert pageBuf.remaining() == pageSize : pageBuf.remaining();
assert pageBuf.position() == 0 : pageBuf.position();
assert pageBuf.order() == nativeOrder() : pageBuf.order();
- assert off <= allocatedBytes() : "calculatedOffset=" + off
- + ", allocated=" + allocatedBytes() + ", headerSize=" + headerSize + ", filePath=" + filePath;
- int n = readWithFailover(pageBuf, off);
+ long pageOff = pageOffset(pageId);
+
+ if (checkPageOffsetLogically) {
+ assert pageOff <= allocatedBytes() : "calculatedOffset=" + pageOff
Review Comment:
**fileOff** sounds better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924400517
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMeta.java:
##########
@@ -17,27 +17,29 @@
package org.apache.ignite.internal.pagememory.persistence;
+import static org.apache.ignite.internal.pagememory.PageIdAllocator.FLAG_AUX;
+import static org.apache.ignite.internal.pagememory.util.PageIdUtils.pageId;
+
import java.lang.invoke.MethodHandles;
import java.lang.invoke.VarHandle;
import java.util.UUID;
-import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
import org.apache.ignite.internal.tostring.S;
import org.jetbrains.annotations.Nullable;
/**
* Partition meta information.
*/
public class PartitionMeta {
- private static final AtomicIntegerFieldUpdater<PartitionMeta> PAGE_COUNT_UPDATER = AtomicIntegerFieldUpdater.newUpdater(
- PartitionMeta.class,
- "pageCount"
- );
+ private static final VarHandle PAGE_COUNT;
Review Comment:
Then I would leave it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924392700
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -183,18 +184,24 @@ private IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePa
Result<PersistentPageMemory, FullPageId> queueResult = new Result<>();
+ GroupPartitionId partitionId = null;
+
while (!shutdownNow.getAsBoolean() && writePageIds.next(queueResult)) {
beforePageWrite.run();
- FullPageId fullId = queueResult.getValue();
+ FullPageId pageId = queueResult.getValue();
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923223851
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
Review Comment:
It's not that bad, I guess that current line was born out of several conflicting sentences and you forgot to clean it up.
I think it supposed to be "... an attempt will be **made to read** its ..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923226885
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -159,11 +165,11 @@ public void stop() throws StorageException {
Stream<AutoCloseable> closeRegions = regions.values().stream().map(region -> region::stop);
CheckpointManager checkpointManager = this.checkpointManager;
- PartitionFilePageStoreManager partitionFilePageStoreManager = this.partitionFilePageStoreManager;
+ FilePageStoreManager filePageStoreManager = this.filePageStoreManager;
Stream<AutoCloseable> closeManagers = Stream.of(
checkpointManager == null ? null : (AutoCloseable) checkpointManager::stop,
- partitionFilePageStoreManager == null ? null : (AutoCloseable) partitionFilePageStoreManager::stop
+ filePageStoreManager == null ? null : (AutoCloseable) filePageStoreManager::stop
Review Comment:
"stop" can't be called before constructor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923224684
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
Review Comment:
filePageStoreManager, meta manager as well. Fields that you instantiate in "start()"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r925256663
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -182,23 +198,36 @@ public boolean read(long pageId, ByteBuffer pageBuf, boolean keepCrc) throws Ign
* @param pageBuf Page buffer to read into.
* @param checkCrc Check CRC on page.
* @param keepCrc By default reading zeroes CRC which was on file, but you can keep it in pageBuf if set keepCrc.
+ * @param checkPageOffsetLogically Check page offset by {@link #allocatedBytes} or {@link #size}.
* @return {@code true} if page has been read successfully, {@code false} if page hasn't been written yet.
* @throws IgniteInternalCheckedException If reading failed (IO error occurred).
*/
- private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean keepCrc) throws IgniteInternalCheckedException {
+ private boolean read0(
+ long pageId,
+ ByteBuffer pageBuf,
+ boolean checkCrc,
+ boolean keepCrc,
+ boolean checkPageOffsetLogically
+ ) throws IgniteInternalCheckedException {
ensure();
try {
- long off = pageOffset(pageId);
-
assert pageBuf.capacity() == pageSize : pageBuf.capacity();
assert pageBuf.remaining() == pageSize : pageBuf.remaining();
assert pageBuf.position() == 0 : pageBuf.position();
assert pageBuf.order() == nativeOrder() : pageBuf.order();
- assert off <= allocatedBytes() : "calculatedOffset=" + off
- + ", allocated=" + allocatedBytes() + ", headerSize=" + headerSize + ", filePath=" + filePath;
- int n = readWithFailover(pageBuf, off);
+ long pageOff = pageOffset(pageId);
+
+ if (checkPageOffsetLogically) {
+ assert pageOff <= allocatedBytes() : "calculatedOffset=" + pageOff
Review Comment:
Ok, but why? Do we have a code that relies on reading empty pages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923233299
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -159,11 +165,11 @@ public void stop() throws StorageException {
Stream<AutoCloseable> closeRegions = regions.values().stream().map(region -> region::stop);
CheckpointManager checkpointManager = this.checkpointManager;
- PartitionFilePageStoreManager partitionFilePageStoreManager = this.partitionFilePageStoreManager;
+ FilePageStoreManager filePageStoreManager = this.filePageStoreManager;
Stream<AutoCloseable> closeManagers = Stream.of(
checkpointManager == null ? null : (AutoCloseable) checkpointManager::stop,
- partitionFilePageStoreManager == null ? null : (AutoCloseable) partitionFilePageStoreManager::stop
+ filePageStoreManager == null ? null : (AutoCloseable) filePageStoreManager::stop
Review Comment:
Then why can't there be null's here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923279863
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -229,15 +242,15 @@ public PersistentPageMemory pageMemory() {
* Returns the size of the view.
*/
public int size() {
- return toPosition - fromPosition + 1;
+ return toPosition - fromPosition;
}
private CheckpointDirtyPages owner() {
return CheckpointDirtyPages.this;
}
- private boolean isToPositionLast() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length - 1;
+ private boolean needsNextRegion() {
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
Review Comment:
nextPartitionView - here I need this index, maybe I shouldn't remove it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923218391
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
Review Comment:
I don't understand what you are suggesting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923216745
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -167,15 +180,15 @@ public IgniteConcurrentMultiPairQueue<PersistentPageMemory, GroupPartitionId> to
FullPageId[] pageIds = dirtyPages.get(regionIndex).dirtyPages.pageIds;
if (fromPosition == pageIds.length - 1 || !equalsByGroupAndPartition(pageIds[fromPosition], pageIds[fromPosition + 1])) {
Review Comment:
I don't think that it'll have any noticeable effect, while making code more complicated. Yes, I insist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921225230
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
+
+ io.setTreeRootPageId(pageAddr, meta.treeRootPageId());
Review Comment:
I'll change it later.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
Review Comment:
I'll change it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r922199008
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -212,4 +241,53 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
}
};
}
+
+ /**
+ * Updates the partition {@link PartitionMetaIo} (pageIdx == 0).
+ *
+ * @throws IgniteInternalCheckedException If failed.
+ */
+ private void updatePartitions() throws IgniteInternalCheckedException {
+ Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+
+ while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
+ GroupPartitionId partitionId = queueResult.getValue();
+
+ PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
+ partitionId.getGroupId(),
+ partitionId.getPartitionId()
+ );
+
+ ByteBuffer buffer = threadBuf.get();
+
+ PersistentPageMemory pageMemory = queueResult.getKey();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
+
+ long pageAddr = GridUnsafe.bufferAddress(buffer);
+
+ // TODO: IGNITE-16657 Merger will put the page into a partition file.
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, buffer.capacity());
+
+ PartitionMeta meta = partitionFilePageStore.meta();
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r920973759
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/ArrayDirtyPages.java:
##########
@@ -15,37 +15,29 @@
* limitations under the License.
*/
-package org.apache.ignite.internal.storage.pagememory;
+package org.apache.ignite.internal.pagememory.persistence.checkpoint;
import org.apache.ignite.internal.pagememory.FullPageId;
+import org.apache.ignite.internal.pagememory.persistence.GroupPartitionId;
/**
- * Class for storing {@link TableTree} partition metadata.
+ * Dirty page array container.
*/
-class PartitionMeta {
- /** {@link TableTree} root. */
- final FullPageId treeRoot;
+class ArrayDirtyPages {
Review Comment:
Got it, I'll change the class name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r921208552
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -59,7 +66,10 @@ public class CheckpointPagesWriter implements Runnable {
* <p>Overall pages to write may be greater than this queue, since it may be necessary to retire write some pages due to unsuccessful
* page write lock acquisition
*/
- private final CheckpointDirtyPagesQueue writePageIds;
+ private final IgniteConcurrentMultiPairQueue<PersistentPageMemory, FullPageId> writePageIds;
+
+ /** Queue of dirty partition IDs to update {@link PartitionMetaIo} under this task. */
Review Comment:
is fine?
`/** Queue of dirty partition IDs to update partition meta page under this task. */`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923288112
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreFactory.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence.store;
+
+import static org.apache.ignite.internal.pagememory.persistence.store.FilePageStoreHeader.readHeader;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.ignite.internal.fileio.FileIo;
+import org.apache.ignite.internal.fileio.FileIoFactory;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+
+/**
+ * Factory for creating {@link FilePageStore}.
+ */
+class FilePageStoreFactory {
+ /** Latest file page store version. */
+ private final int latestVersion = FilePageStore.VERSION_1;
+
+ private final FileIoFactory fileIoFactory;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param fileIoFactory File IO factory.
+ * @param pageSize Page size in bytes.
+ */
+ public FilePageStoreFactory(FileIoFactory fileIoFactory, int pageSize) {
+ this.fileIoFactory = fileIoFactory;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Creates instance of {@link FilePageStore}.
+ *
+ * <p>If the file exists, an attempt will be reads its {@link FilePageStoreHeader header} and create the {@link FilePageStore}.
Review Comment:
fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923134619
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
Review Comment:
If we just initialized the **FilePageStore** (with `pageCount == 0`) and try to read the **PartitionMetaIo** (`pageIdx == 0`), we will get an error that we are trying to read the page beyond the number of pages.
We can go the following ways:
1. leave as is, but add comment `FilePageStore has not yet updated 'pageCount' so as not to get an error when reading PartitionMetaIo (pageIdx == 0) let's update its 'pageCount'.`;
2. change the logic of reading from a **FilePageStore** and for example:
2.1 do not check page read boundaries;
2.2 for the first page (`pageIdx == 0`) do not check borders;
3. always do `filePageStore.pages(1)`.
I am for the 1 and 3 option, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923501294
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -253,38 +251,21 @@ private void updatePartitions() throws IgniteInternalCheckedException {
while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
GroupPartitionId partitionId = queueResult.getValue();
- PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
- partitionId.getGroupId(),
- partitionId.getPartitionId()
- );
-
ByteBuffer buffer = threadBuf.get();
PersistentPageMemory pageMemory = queueResult.getKey();
- long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
-
- long pageAddr = GridUnsafe.bufferAddress(buffer);
-
- // TODO: IGNITE-16657 Merger will put the page into a partition file.
- PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923165330
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
+ filePageStore.pages(1);
+ }
+
+ boolean read = filePageStore.read(partitionMetaPageId, buffer, false);
+
+ assert read : filePageStore.filePath();
+
+ return createMeta(ioRegistry.resolve(bufferAddr), bufferAddr, checkpointId);
+ } else {
+ // Creates and writes a partition meta.
+ PartitionMetaIo partitionMetaIo = PartitionMetaIo.VERSIONS.latest();
+ partitionMetaIo.initNewPage(bufferAddr, partitionMetaPageId, pageSize);
+
+ // Because we will now write this page.
+ partitionMetaIo.setPageCount(bufferAddr, 1);
+
+ filePageStore.pages(1);
+
+ filePageStore.write(partitionMetaPageId, buffer.rewind(), true);
+
+ filePageStore.sync();
+
+ return createMeta(partitionMetaIo, bufferAddr, checkpointId);
+ }
+ } finally {
+ freeBuffer(buffer);
+ }
+ }
+
+ /**
+ * Writes the partition meta to the buffer.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param partitionMeta Snapshot of the partition meta.
+ * @param writeToBuffer Direct byte buffer to write partition meta.
+ */
+ public void writeMetaToBuffer(
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ PartitionMetaSnapshot partitionMeta,
+ ByteBuffer writeToBuffer
+ ) {
+ assert writeToBuffer.remaining() == pageSize : writeToBuffer.remaining();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ long pageAddr = bufferAddress(writeToBuffer);
+
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, pageSize);
+
+ io.setTreeRootPageId(pageAddr, partitionMeta.treeRootPageId());
Review Comment:
Fix it.
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/PartitionMetaManager.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.pagememory.persistence;
+
+import static org.apache.ignite.internal.util.GridUnsafe.allocateBuffer;
+import static org.apache.ignite.internal.util.GridUnsafe.bufferAddress;
+import static org.apache.ignite.internal.util.GridUnsafe.freeBuffer;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.internal.pagememory.io.PageIoRegistry;
+import org.apache.ignite.internal.pagememory.persistence.PartitionMeta.PartitionMetaSnapshot;
+import org.apache.ignite.internal.pagememory.persistence.io.PartitionMetaIo;
+import org.apache.ignite.internal.pagememory.persistence.store.FilePageStore;
+import org.apache.ignite.lang.IgniteInternalCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Partition meta information manager.
+ */
+// TODO: IGNITE-17132 Do not forget about deleting the partition meta information
+public class PartitionMetaManager {
+ private final Map<GroupPartitionId, PartitionMeta> metas = new ConcurrentHashMap<>();
+
+ private final PageIoRegistry ioRegistry;
+
+ private final int pageSize;
+
+ /**
+ * Constructor.
+ *
+ * @param ioRegistry Page IO Registry.
+ * @param pageSize Page size in bytes.
+ */
+ public PartitionMetaManager(PageIoRegistry ioRegistry, int pageSize) {
+ this.ioRegistry = ioRegistry;
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Returns the partition's meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ */
+ public @Nullable PartitionMeta getMeta(GroupPartitionId groupPartitionId) {
+ return metas.get(groupPartitionId);
+ }
+
+ /**
+ * Adds partition meta information.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param partitionMeta Partition meta information.
+ */
+ public void addMeta(GroupPartitionId groupPartitionId, PartitionMeta partitionMeta) {
+ metas.put(groupPartitionId, partitionMeta);
+ }
+
+ /**
+ * Reads the partition {@link PartitionMeta meta} from the partition file or creates a new one.
+ *
+ * <p>If it creates a new one, it writes the meta to the file.
+ *
+ * @param checkpointId Checkpoint ID.
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param filePageStore Partition file page store.
+ */
+ public PartitionMeta readOrCreateMeta(
+ @Nullable UUID checkpointId,
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ FilePageStore filePageStore
+ ) throws IgniteInternalCheckedException {
+ ByteBuffer buffer = allocateBuffer(pageSize);
+
+ long bufferAddr = bufferAddress(buffer);
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ try {
+ if (filePageStore.size() > filePageStore.headerSize()) {
+ // Reads the partition meta.
+ if (filePageStore.pages() == 0) {
+ filePageStore.pages(1);
+ }
+
+ boolean read = filePageStore.read(partitionMetaPageId, buffer, false);
+
+ assert read : filePageStore.filePath();
+
+ return createMeta(ioRegistry.resolve(bufferAddr), bufferAddr, checkpointId);
+ } else {
+ // Creates and writes a partition meta.
+ PartitionMetaIo partitionMetaIo = PartitionMetaIo.VERSIONS.latest();
+ partitionMetaIo.initNewPage(bufferAddr, partitionMetaPageId, pageSize);
+
+ // Because we will now write this page.
+ partitionMetaIo.setPageCount(bufferAddr, 1);
+
+ filePageStore.pages(1);
+
+ filePageStore.write(partitionMetaPageId, buffer.rewind(), true);
+
+ filePageStore.sync();
+
+ return createMeta(partitionMetaIo, bufferAddr, checkpointId);
+ }
+ } finally {
+ freeBuffer(buffer);
+ }
+ }
+
+ /**
+ * Writes the partition meta to the buffer.
+ *
+ * @param groupPartitionId Partition of the group.
+ * @param pageMemory Page memory.
+ * @param partitionMeta Snapshot of the partition meta.
+ * @param writeToBuffer Direct byte buffer to write partition meta.
+ */
+ public void writeMetaToBuffer(
+ GroupPartitionId groupPartitionId,
+ PersistentPageMemory pageMemory,
+ PartitionMetaSnapshot partitionMeta,
+ ByteBuffer writeToBuffer
+ ) {
+ assert writeToBuffer.remaining() == pageSize : writeToBuffer.remaining();
+
+ long partitionMetaPageId = pageMemory.partitionMetaPageId(groupPartitionId.getGroupId(), groupPartitionId.getPartitionId());
+
+ long pageAddr = bufferAddress(writeToBuffer);
+
+ PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+
+ io.initNewPage(pageAddr, partitionMetaPageId, pageSize);
+
+ io.setTreeRootPageId(pageAddr, partitionMeta.treeRootPageId());
+ io.setReuseListRootPageId(pageAddr, partitionMeta.reuseListRootPageId());
+ io.setPageCount(pageAddr, partitionMeta.pageCount());
+ }
+
+ private PartitionMeta createMeta(PartitionMetaIo metaIo, long pageAddr, @Nullable UUID checkpointId) {
+ return new PartitionMeta(
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923221540
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -253,38 +251,21 @@ private void updatePartitions() throws IgniteInternalCheckedException {
while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
GroupPartitionId partitionId = queueResult.getValue();
- PartitionFilePageStore partitionFilePageStore = partitionFilePageStoreManager.getStore(
- partitionId.getGroupId(),
- partitionId.getPartitionId()
- );
-
ByteBuffer buffer = threadBuf.get();
PersistentPageMemory pageMemory = queueResult.getKey();
- long partitionMetaPageId = pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId());
-
- long pageAddr = GridUnsafe.bufferAddress(buffer);
-
- // TODO: IGNITE-16657 Merger will put the page into a partition file.
- PartitionMetaIo io = PartitionMetaIo.VERSIONS.latest();
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
Review Comment:
Well, you already have a collection of ids. Maybe there's an argument that we need to create GroupPartitionId objects all the time, but even that is not a problem I think. Can be easily optimized though. And contention - I don't understand what you mean here specifically. Like during a "put" into a set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923184850
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -229,15 +242,15 @@ public PersistentPageMemory pageMemory() {
* Returns the size of the view.
*/
public int size() {
- return toPosition - fromPosition + 1;
+ return toPosition - fromPosition;
}
private CheckpointDirtyPages owner() {
return CheckpointDirtyPages.this;
}
- private boolean isToPositionLast() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length - 1;
+ private boolean needsNextRegion() {
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
Review Comment:
A reference weighs more than a (**int**) primitive, and it's easier for the garbage collector with a primitive. Do you insist on removal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923197533
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -156,10 +138,18 @@ public void stop(boolean clean) throws IgniteInternalCheckedException {
/** {@inheritDoc} */
@Override
- public long allocatePage() throws IgniteInternalCheckedException {
+ public int allocatePage() throws IgniteInternalCheckedException {
ensure();
- return pageCount.getAndIncrement();
+ int pageIdx = pageCount.getAndIncrement();
+
+ PageAllocationListener listener = this.pageAllocationListener;
+
+ if (listener != null) {
+ listener.onAllocationPage(pageIdx);
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924399842
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointPagesWriter.java:
##########
@@ -240,35 +247,25 @@ public void writePage(FullPageId fullPageId, ByteBuffer buf, int tag) throws Ign
};
}
- /**
- * Updates the partitions meta page.
- *
- * @throws IgniteInternalCheckedException If failed.
- */
- private void updatePartitions() throws IgniteInternalCheckedException {
- Result<PersistentPageMemory, GroupPartitionId> queueResult = new Result<>();
+ private void writePartitionMeta(GroupPartitionId partitionId, ByteBuffer buffer) throws IgniteInternalCheckedException {
+ PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
- while (!shutdownNow.getAsBoolean() && updatePartitionIds.next(queueResult)) {
- GroupPartitionId partitionId = queueResult.getValue();
+ partitionMetaManager.writeMetaToBuffer(partitionId, partitionMetaSnapshot, buffer.rewind());
- ByteBuffer buffer = threadBuf.get();
+ FullPageId fullPageId = new FullPageId(partitionMetaPageId(partitionId.getPartitionId()), partitionId.getGroupId());
- PersistentPageMemory pageMemory = queueResult.getKey();
-
- PartitionMetaSnapshot partitionMetaSnapshot = partitionMetaManager.getMeta(partitionId).metaSnapshot(checkpointProgress.id());
-
- partitionMetaManager.writeMetaToBuffer(partitionId, pageMemory, partitionMetaSnapshot, buffer.rewind());
+ PageStore store = pageWriter.write(fullPageId, buffer.rewind());
- FullPageId fullPageId = new FullPageId(
- pageMemory.partitionMetaPageId(partitionId.getGroupId(), partitionId.getPartitionId()),
- partitionId.getGroupId()
- );
+ checkpointProgress.writtenPagesCounter().incrementAndGet();
- PageStore store = pageWriter.write(fullPageId, buffer.rewind());
+ updStores.computeIfAbsent(store, k -> new LongAdder()).increment();
+ }
- checkpointProgress.writtenPagesCounter().incrementAndGet();
+ private static boolean isPartitionChange(@Nullable GroupPartitionId partitionId, FullPageId pageId) {
Review Comment:
Why didn't you call it "hasPartitionChanged"?
"is partition change" looks weird
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924397630
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStore.java:
##########
@@ -237,9 +266,41 @@ private boolean read(long pageId, ByteBuffer pageBuf, boolean checkCrc, boolean
}
}
+ /**
+ * Writes a page unlike {@link #write(long, ByteBuffer, boolean)}, checks the page offset in the file not logically (pageOffset <=
+ * {@link #pages()} * {@link #pageSize}) but physically (pageOffset <= {@link #size()}), which can affect performance when used in
+ * production code.
+ *
+ * @param pageId Page ID.
+ * @param pageBuf Page buffer to write from.
+ * @param calculateCrc If {@code false} crc calculation will be forcibly skipped.
+ * @throws IgniteInternalCheckedException If page writing failed (IO error occurred).
+ */
+ public void writeByPhysicalOffset(long pageId, ByteBuffer pageBuf, boolean calculateCrc) throws IgniteInternalCheckedException {
Review Comment:
I think it's not right to reserve something with a **FilePageStore**, it should be responsible only for its functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r925317141
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointManager.java:
##########
@@ -191,6 +199,13 @@ public CheckpointProgress forceCheckpoint(String reason) {
return checkpointer.scheduleCheckpoint(0, reason);
}
+ /**
+ * Returns progress of current checkpoint, last finished one or {@code null}, if checkpoint has never started.
+ */
+ public @Nullable CheckpointProgress currentProgress() {
Review Comment:
Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923206822
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
+ Loggers.forClass(FilePageStoreManager.class),
igniteInstanceName,
storagePath,
fileIoFactory,
- ioRegistry,
pageSize
);
- partitionFilePageStoreManager.start();
+ filePageStoreManager.start();
} catch (IgniteInternalCheckedException e) {
throw new StorageException("Error starting file page store manager", e);
}
+ partitionMetaManager = new PartitionMetaManager(ioRegistry, pageSize);
Review Comment:
Understood nothing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923205582
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
Review Comment:
Tell me which fields exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923203540
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/store/FilePageStoreHeader.java:
##########
@@ -97,18 +97,19 @@ ByteBuffer toByteBuffer() {
* Reads the header of a file page store.
*
* @param fileIo File page store fileIo.
+ * @param readIntoBuffer Buffer for reading {@link FilePageStoreHeader header} from {@code fileIo}.
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923226503
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
+ Loggers.forClass(FilePageStoreManager.class),
igniteInstanceName,
storagePath,
fileIoFactory,
- ioRegistry,
pageSize
);
- partitionFilePageStoreManager.start();
+ filePageStoreManager.start();
} catch (IgniteInternalCheckedException e) {
throw new StorageException("Error starting file page store manager", e);
}
+ partitionMetaManager = new PartitionMetaManager(ioRegistry, pageSize);
Review Comment:
If page size it not known during the start, that would be a good answer for me. But even then we could just pass configuration instead of size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r923225201
##########
modules/storage-page-memory/src/main/java/org/apache/ignite/internal/storage/pagememory/PersistentPageMemoryStorageEngine.java:
##########
@@ -106,27 +110,29 @@ public void start() throws StorageException {
? new AsyncFileIoFactory()
: new RandomAccessFileIoFactory();
- partitionFilePageStoreManager = new PartitionFilePageStoreManager(
- Loggers.forClass(PartitionFilePageStoreManager.class),
+ filePageStoreManager = new FilePageStoreManager(
+ Loggers.forClass(FilePageStoreManager.class),
igniteInstanceName,
storagePath,
fileIoFactory,
- ioRegistry,
pageSize
);
- partitionFilePageStoreManager.start();
+ filePageStoreManager.start();
} catch (IgniteInternalCheckedException e) {
throw new StorageException("Error starting file page store manager", e);
}
+ partitionMetaManager = new PartitionMetaManager(ioRegistry, pageSize);
Review Comment:
Why do you have to assign values to fields in "start" and not in constructor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #920: IGNITE-17295 Preparing a PartitionMetaIo for a DeltaFilePageStore
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #920:
URL: https://github.com/apache/ignite-3/pull/920#discussion_r924416092
##########
modules/page-memory/src/main/java/org/apache/ignite/internal/pagememory/persistence/checkpoint/CheckpointDirtyPages.java:
##########
@@ -250,7 +215,7 @@ private CheckpointDirtyPages owner() {
}
private boolean needsNextRegion() {
- return toPosition == dirtyPages.get(regionIndex).dirtyPages.pageIds.length;
+ return toPosition == dirtyPages.get(regionIndex).dirtyPages.length;
Review Comment:
I would not like to add a field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org