You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "sashapolo (via GitHub)" <gi...@apache.org> on 2024/08/07 15:04:31 UTC
[PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
sashapolo opened a new pull request, #4200:
URL: https://github.com/apache/ignite-3/pull/4200
https://issues.apache.org/jira/browse/IGNITE-19395
Thank you for submitting the pull request.
To streamline the review process of the patch and ensure better code quality
we ask both an author and a reviewer to verify the following:
### The Review Checklist
- [ ] **Formal criteria:** TC status, codestyle, mandatory documentation. Also make sure to complete the following:
\- There is a single JIRA ticket related to the pull request.
\- The web-link to the pull request is attached to the JIRA ticket.
\- The JIRA ticket has the Patch Available state.
\- The description of the JIRA ticket explains WHAT was made, WHY and HOW.
\- The pull request title is treated as the final commit message. The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
- [ ] **Design:** new code conforms with the design principles of the components it is added to.
- [ ] **Patch quality:** patch cannot be split into smaller pieces, its size must be reasonable.
- [ ] **Code quality:** code is clean and readable, necessary developer documentation is added if needed.
- [ ] **Tests code quality:** test set covers positive/negative scenarios, happy/edge cases. Tests are effective in terms of execution time and resources.
### Notes
- [Apache Ignite Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Java+Code+Style+Guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719432866
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/RocksDbMvPartitionStorage.java:
##########
@@ -116,9 +136,17 @@
*/
public class RocksDbMvPartitionStorage implements MvPartitionStorage {
/** Thread-local on-heap byte buffer instance to use for key manipulations. */
- private static final ThreadLocal<ByteBuffer> HEAP_KEY_BUFFER = withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_COMMITTED_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
- private static final ThreadLocal<ByteBuffer> DIRECT_KEY_BUFFER = withInitial(() -> allocateDirect(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(ROW_PREFIX_SIZE).order(KEY_BYTE_ORDER));
+
+ private static final ThreadLocal<ByteBuffer> DIRECT_DATA_ID_KEY_BUFFER =
Review Comment:
> we have exactly the same buffers here and in GarbageCollector, is it intentional? We can't reuse them?
Sometimes it's possible, sometimes it's not, because GC methods can be called from the Partition Storage code. It makes it tricky to always remember which buffers can be used by which components (e.g. GC can overwrite a buffer that Partition Storage was using). I prefer to fully encapsulate these buffers and not to share state between two components.
> And in GC we have javadocs for buffers, let's add javadocs here?
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719434337
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/RocksDbStorageEngine.java:
##########
@@ -152,6 +152,7 @@ public String name() {
@Override
public void start() throws StorageException {
// TODO: IGNITE-17066 Add handling deleting/updating storage profiles configuration
+ storageConfiguration.profiles().value();
Review Comment:
oops
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1723313574
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -44,45 +43,46 @@
import org.jetbrains.annotations.Nullable;
import org.rocksdb.ColumnFamilyHandle;
import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
import org.rocksdb.RocksIterator;
import org.rocksdb.Slice;
import org.rocksdb.WriteBatchWithIndex;
/** Helper for the partition data. */
public final class PartitionDataHelper implements ManuallyCloseable {
+ /** Transaction id size (part of the transaction state). */
+ private static final int TX_ID_SIZE = 2 * Long.BYTES;
+
/** Position of row id inside the key. */
- static final int ROW_ID_OFFSET = TABLE_ID_SIZE + Short.BYTES;
+ static final int ROW_ID_OFFSET = TABLE_ID_SIZE + PARTITION_ID_SIZE;
- /** Size of the key without timestamp. */
Review Comment:
Intentional, but I forgot to add a new javadoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719843053
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/ColumnFamilyUtils.java:
##########
@@ -85,6 +89,8 @@ public static ColumnFamilyType fromCfName(String cfName) {
return PARTITION;
} else if (GC_QUEUE_CF_NAME.equals(cfName)) {
return GC_QUEUE;
+ } else if (DATA_CF_NAME.equals(cfName)) {
Review Comment:
Switch is prettier and faster, but I don't care to be honest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719432866
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/RocksDbMvPartitionStorage.java:
##########
@@ -116,9 +136,17 @@
*/
public class RocksDbMvPartitionStorage implements MvPartitionStorage {
/** Thread-local on-heap byte buffer instance to use for key manipulations. */
- private static final ThreadLocal<ByteBuffer> HEAP_KEY_BUFFER = withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_COMMITTED_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
- private static final ThreadLocal<ByteBuffer> DIRECT_KEY_BUFFER = withInitial(() -> allocateDirect(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(ROW_PREFIX_SIZE).order(KEY_BYTE_ORDER));
+
+ private static final ThreadLocal<ByteBuffer> DIRECT_DATA_ID_KEY_BUFFER =
Review Comment:
> we have exactly the same buffers here and in GarbageCollector, is it intentional? We can't reuse them?
Sometimes it's possible, sometimes it's not, because GC methods can be called from the Partition Storage code. It makes it tricky to always remember which buffers can be used by which components (e.g. GC can overwrite a buffer that Partition Storage was using). I prefer to fully encapsulate these buffers and not to share state between two components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719434875
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -263,6 +294,59 @@ static BinaryRow deserializeRow(ByteBuffer buffer) {
return new BinaryRowImpl(schemaVersion, binaryTupleSlice);
}
+ byte[] createPayloadKey(ByteBuffer dataId) {
+ byte[] result = PAYLOAD_KEY_BUFFER.get()
+ .clear()
+ .putInt(tableId)
+ .putShort((short) partitionId)
+ .put(dataId)
+ .array();
+
+ dataId.rewind();
+
+ // Always use 0 for the last bit (tombstone flag), because it only makes sense when data ID is stored as a value.
+ setLastBit(result, result.length - 1, false);
+
+ return result;
+ }
+
+ /**
+ * Changes the last bit of the byte identified by an index in an array.
+ *
+ * @param array Array containing the byte to change.
+ * @param index Index of the byte inside the array.
+ * @param value If {@code true} - sets the bit to 1, else to 0.
+ */
+ static void setLastBit(byte[] array, int index, boolean value) {
+ byte lastByte = array[index];
+
+ if (value) {
+ lastByte |= 0x01;
+ } else {
+ lastByte &= 0xFE;
+ }
+
+ array[index] = lastByte;
Review Comment:
yeah, this code was originally for ByteBuffers, forgot to rewrite it, thanks for noticing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "Phillippko (via GitHub)" <gi...@apache.org>.
Phillippko commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1718385001
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/instance/SharedRocksDbInstance.java:
##########
@@ -119,6 +119,8 @@ public void close() {
/** Column Family for GC queue. */
public final ColumnFamily gcQueueCf;
+ public final ColumnFamily dataCf;
Review Comment:
Let's add javadoc
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/RocksDbTableStorage.java:
##########
@@ -136,6 +136,10 @@ ColumnFamilyHandle gcQueueHandle() {
return rocksDb.gcQueueCf.handle();
}
+ ColumnFamilyHandle dataCfHandle() {
Review Comment:
javadoc
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/RocksDbMvPartitionStorage.java:
##########
@@ -116,9 +136,17 @@
*/
public class RocksDbMvPartitionStorage implements MvPartitionStorage {
/** Thread-local on-heap byte buffer instance to use for key manipulations. */
- private static final ThreadLocal<ByteBuffer> HEAP_KEY_BUFFER = withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_COMMITTED_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
- private static final ThreadLocal<ByteBuffer> DIRECT_KEY_BUFFER = withInitial(() -> allocateDirect(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(ROW_PREFIX_SIZE).order(KEY_BYTE_ORDER));
+
+ private static final ThreadLocal<ByteBuffer> DIRECT_DATA_ID_KEY_BUFFER =
Review Comment:
we have exactly the same buffers here and in GarbageCollector, is it intentional? We can't reuse them?
And in GC we have javadocs for buffers, let's add javadocs here?
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -263,6 +294,59 @@ static BinaryRow deserializeRow(ByteBuffer buffer) {
return new BinaryRowImpl(schemaVersion, binaryTupleSlice);
}
+ byte[] createPayloadKey(ByteBuffer dataId) {
+ byte[] result = PAYLOAD_KEY_BUFFER.get()
+ .clear()
+ .putInt(tableId)
+ .putShort((short) partitionId)
+ .put(dataId)
+ .array();
+
+ dataId.rewind();
+
+ // Always use 0 for the last bit (tombstone flag), because it only makes sense when data ID is stored as a value.
+ setLastBit(result, result.length - 1, false);
+
+ return result;
+ }
+
+ /**
+ * Changes the last bit of the byte identified by an index in an array.
+ *
+ * @param array Array containing the byte to change.
+ * @param index Index of the byte inside the array.
+ * @param value If {@code true} - sets the bit to 1, else to 0.
+ */
+ static void setLastBit(byte[] array, int index, boolean value) {
+ byte lastByte = array[index];
+
+ if (value) {
+ lastByte |= 0x01;
+ } else {
+ lastByte &= 0xFE;
+ }
+
+ array[index] = lastByte;
Review Comment:
```suggestion
if (value) {
array[index] |= 0x01;
} else {
array[index] &= 0xFE;
}
```
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/ColumnFamilyUtils.java:
##########
@@ -42,12 +42,15 @@ public class ColumnFamilyUtils {
/** Name of the meta column family matches default columns family, meaning that it always exist when new table is created. */
private static final String META_CF_NAME = "default";
- /** Name of the Column Family that stores partition data. */
+ /** Name of the Column Family that stores partition data with references to row data. */
private static final String PARTITION_CF_NAME = "cf-part";
/** Name of the Column Family that stores garbage collection queue. */
private static final String GC_QUEUE_CF_NAME = "cf-gc";
+ /** Name of the Column Family that stores row data. */
Review Comment:
Let's move it closer to cf-part, before cf-gc
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/RocksDbStorageEngine.java:
##########
@@ -152,6 +152,7 @@ public String name() {
@Override
public void start() throws StorageException {
// TODO: IGNITE-17066 Add handling deleting/updating storage profiles configuration
+ storageConfiguration.profiles().value();
Review Comment:
leftovers
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/GarbageCollector.java:
##########
@@ -73,8 +76,17 @@ class GarbageCollector {
/** Garbage collector's queue key's size. */
private static final int GC_KEY_SIZE = GC_KEY_ROW_ID_OFFSET + ROW_ID_SIZE;
- /** Thread-local direct buffer instance to read keys from RocksDB. */
- private static final ThreadLocal<ByteBuffer> GC_KEY_BUFFER = withInitial(() -> allocateDirect(GC_KEY_SIZE).order(KEY_BYTE_ORDER));
+ /** Thread-local direct buffer instance to read keys that contain Data ID. */
+ private static final ThreadLocal<ByteBuffer> DIRECT_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocateDirect(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+
+ /** Thread-local direct buffer for storing Data ID. */
Review Comment:
```suggestion
/** Thread-local direct buffer for Data ID manipulations. */
```
Also, let's fix descriptions for other buffers? They're not only "to read"
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/ColumnFamilyUtils.java:
##########
@@ -85,6 +89,8 @@ public static ColumnFamilyType fromCfName(String cfName) {
return PARTITION;
} else if (GC_QUEUE_CF_NAME.equals(cfName)) {
return GC_QUEUE;
+ } else if (DATA_CF_NAME.equals(cfName)) {
Review Comment:
I think it's time to use switch
if (cfName.startsWith(SORTED_INDEX_CF_PREFIX)) {
return SORTED_INDEX;
}
switch (cfName) {
case META_CF_NAME:
return META;
case PARTITION_CF_NAME:
return PARTITION;
case GC_QUEUE_CF_NAME:
return GC_QUEUE;
case DATA_CF_NAME:
return DATA;
case HASH_INDEX_CF_NAME:
return HASH_INDEX;
default:
return UNKNOWN;
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719435337
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/ColumnFamilyUtils.java:
##########
@@ -85,6 +89,8 @@ public static ColumnFamilyType fromCfName(String cfName) {
return PARTITION;
} else if (GC_QUEUE_CF_NAME.equals(cfName)) {
return GC_QUEUE;
+ } else if (DATA_CF_NAME.equals(cfName)) {
Review Comment:
What's wrong with the current code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719432866
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/RocksDbMvPartitionStorage.java:
##########
@@ -116,9 +136,17 @@
*/
public class RocksDbMvPartitionStorage implements MvPartitionStorage {
/** Thread-local on-heap byte buffer instance to use for key manipulations. */
- private static final ThreadLocal<ByteBuffer> HEAP_KEY_BUFFER = withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_COMMITTED_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
- private static final ThreadLocal<ByteBuffer> DIRECT_KEY_BUFFER = withInitial(() -> allocateDirect(MAX_KEY_SIZE).order(KEY_BYTE_ORDER));
+ private static final ThreadLocal<ByteBuffer> HEAP_DATA_ID_KEY_BUFFER =
+ withInitial(() -> allocate(ROW_PREFIX_SIZE).order(KEY_BYTE_ORDER));
+
+ private static final ThreadLocal<ByteBuffer> DIRECT_DATA_ID_KEY_BUFFER =
Review Comment:
> we have exactly the same buffers here and in GarbageCollector, is it intentional? We can't reuse them?
Sometimes it's possible, sometimes it's not, because GC methods can be called from the Partition Storage code. It makes it tricky to always remember which buffers can be used by which components (e.g. GC can overwrite a buffer that Partition Storage was using). I prefer to fully encapsulate these buffers and not share state between two components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719890694
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -263,6 +294,55 @@ static BinaryRow deserializeRow(ByteBuffer buffer) {
return new BinaryRowImpl(schemaVersion, binaryTupleSlice);
}
+ byte[] createPayloadKey(ByteBuffer dataId) {
+ byte[] result = PAYLOAD_KEY_BUFFER.get()
+ .clear()
+ .putInt(tableId)
+ .putShort((short) partitionId)
+ .put(dataId)
+ .array();
+
+ dataId.rewind();
+
+ // Always use 0 for the last bit (tombstone flag), because it only makes sense when data ID is stored as a value.
+ setLastBit(result, result.length - 1, false);
+
+ return result;
+ }
+
+ /**
+ * Changes the last bit of the byte identified by an index in an array.
Review Comment:
It's a first bit, not last bit. What exactly did you mean in this name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "Phillippko (via GitHub)" <gi...@apache.org>.
Phillippko commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1721254664
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -263,6 +294,55 @@ static BinaryRow deserializeRow(ByteBuffer buffer) {
return new BinaryRowImpl(schemaVersion, binaryTupleSlice);
}
+ byte[] createPayloadKey(ByteBuffer dataId) {
+ byte[] result = PAYLOAD_KEY_BUFFER.get()
+ .clear()
+ .putInt(tableId)
+ .putShort((short) partitionId)
+ .put(dataId)
+ .array();
+
+ dataId.rewind();
+
+ // Always use 0 for the last bit (tombstone flag), because it only makes sense when data ID is stored as a value.
+ setLastBit(result, result.length - 1, false);
+
+ return result;
+ }
+
+ /**
+ * Changes the last bit of the byte identified by an index in an array.
Review Comment:
We could use setRightMostBit, or setLsb / setLeastSignificantBit to avoid confusion
Maybe even "setTombstoneBit"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "Phillippko (via GitHub)" <gi...@apache.org>.
Phillippko commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719423807
##########
modules/storage-rocksdb/src/test/java/org/apache/ignite/internal/storage/rocksdb/benchmarks/CommitManyWritesBenchmark.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.rocksdb.benchmarks;
+
+import static org.apache.ignite.internal.util.IgniteUtils.capacity;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.IntStream;
+import org.apache.ignite.configuration.ConfigurationValue;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.BinaryRowImpl;
+import org.apache.ignite.internal.storage.MvPartitionStorage;
+import org.apache.ignite.internal.storage.RowId;
+import org.apache.ignite.internal.storage.configurations.StorageConfiguration;
+import org.apache.ignite.internal.storage.engine.StorageTableDescriptor;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbMvPartitionStorage;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbStorageEngine;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbTableStorage;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbProfileView;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbStorageEngineConfiguration;
+import org.apache.ignite.internal.tx.TransactionIds;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+/**
+ * Benchmark for measuring the performance of {@link RocksDbMvPartitionStorage} in a scenario, when many entries are added in one big
+ * transaction and are then committed.
+ */
+@State(Scope.Benchmark)
+public class CommitManyWritesBenchmark {
+ private static final String STORAGE_PROFILE_NAME = "test";
+ private static final int TABLE_ID = 1;
+ private static final int NUM_PARTITIONS = 50;
+ private static final int NUM_ROWS = 10_000;
+ private static final int ROW_LENGTH = 10_000;
+
+ private final HybridClock clock = new HybridClockImpl();
+
+ private RocksDbStorageEngine storageEngine;
+
+ private RocksDbTableStorage tableStorage;
+
+ /** Setup method. */
+ @Setup
+ public void setUp() throws IOException {
+ Path workDir = Files.createTempDirectory(CommitManyWritesBenchmark.class.getSimpleName());
+
+ storageEngine = new RocksDbStorageEngine(
+ "test",
+ engineConfiguration(),
+ storageConfiguration(),
+ workDir,
+ () -> {}
+ );
+
+ storageEngine.start();
+
+ var tableDescriptor = new StorageTableDescriptor(TABLE_ID, NUM_PARTITIONS, STORAGE_PROFILE_NAME);
+
+ tableStorage = storageEngine.createMvTable(tableDescriptor, indexId -> null);
+
+ CompletableFuture<?>[] createFutures = IntStream.range(0, NUM_PARTITIONS)
+ .mapToObj(tableStorage::createMvPartition)
+ .toArray(CompletableFuture[]::new);
+
+ CompletableFuture.allOf(createFutures).join();
+ }
+
+ /** Tear down method. */
+ @TearDown
+ public void tearDown() {
+ tableStorage.destroy();
+
+ storageEngine.stop();
+ }
+
+ /** Generated data for each thread. */
+ @State(Scope.Thread)
+ public static class GeneratedData {
+ int partitionId;
+
+ Map<RowId, BinaryRow> rows;
+
+ /** Setup method. */
+ @Setup
+ public void setUp() {
+ ThreadLocalRandom random = ThreadLocalRandom.current();
+
+ partitionId = random.nextInt(NUM_PARTITIONS);
+
+ rows = new HashMap<>(capacity(NUM_ROWS));
+
+ for (int i = 0; i < NUM_ROWS; i++) {
+ var rowId = new RowId(partitionId);
+
+ BinaryRow row = randomRow(random);
+
+ rows.put(rowId, row);
+ }
+ }
+
+ private static BinaryRow randomRow(ThreadLocalRandom random) {
+ ByteBuffer buffer = ByteBuffer.allocate(ROW_LENGTH);
+
+ random.nextBytes(buffer.array());
+
+ return new BinaryRowImpl(0, buffer);
+ }
+ }
+
+ private static RocksDbStorageEngineConfiguration engineConfiguration() {
+ RocksDbStorageEngineConfiguration config = mock(RocksDbStorageEngineConfiguration.class);
+
+ ConfigurationValue<Integer> flushDelayMillis = mock(ConfigurationValue.class);
+
+ when(flushDelayMillis.value()).thenReturn(100);
+
+ when(config.flushDelayMillis()).thenReturn(flushDelayMillis);
+
+ return config;
+ }
+
+ private static StorageConfiguration storageConfiguration() {
+ StorageConfiguration config = mock(StorageConfiguration.class);
+
+ NamedConfigurationTree profilesTree = mock(NamedConfigurationTree.class);
+ NamedListView profilesView = mock(NamedListView.class);
+ RocksDbProfileView rocksDbProfileView = mock(RocksDbProfileView.class);
+
+ when(rocksDbProfileView.name()).thenReturn(STORAGE_PROFILE_NAME);
+ when(rocksDbProfileView.size()).thenReturn(16777216L);
+ when(rocksDbProfileView.writeBufferSize()).thenReturn(16777216L);
+
+ when(config.profiles()).thenReturn(profilesTree);
+ when(profilesTree.value()).thenReturn(profilesView);
+ when(profilesView.iterator()).thenReturn(List.of(rocksDbProfileView).iterator());
+
+ return config;
+ }
+
+ /** Benchmark. */
+ @Benchmark
+ public void commitManyWrites(GeneratedData data) {
+ MvPartitionStorage partitionStorage = tableStorage.getMvPartition(data.partitionId);
Review Comment:
We could recreate storage after every iteration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719433477
##########
modules/storage-rocksdb/src/test/java/org/apache/ignite/internal/storage/rocksdb/benchmarks/CommitManyWritesBenchmark.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.rocksdb.benchmarks;
+
+import static org.apache.ignite.internal.util.IgniteUtils.capacity;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.IntStream;
+import org.apache.ignite.configuration.ConfigurationValue;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.BinaryRowImpl;
+import org.apache.ignite.internal.storage.MvPartitionStorage;
+import org.apache.ignite.internal.storage.RowId;
+import org.apache.ignite.internal.storage.configurations.StorageConfiguration;
+import org.apache.ignite.internal.storage.engine.StorageTableDescriptor;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbMvPartitionStorage;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbStorageEngine;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbTableStorage;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbProfileView;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbStorageEngineConfiguration;
+import org.apache.ignite.internal.tx.TransactionIds;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+/**
+ * Benchmark for measuring the performance of {@link RocksDbMvPartitionStorage} in a scenario, when many entries are added in one big
+ * transaction and are then committed.
+ */
+@State(Scope.Benchmark)
Review Comment:
Why? Defaults are reasonable here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719925609
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -263,6 +294,55 @@ static BinaryRow deserializeRow(ByteBuffer buffer) {
return new BinaryRowImpl(schemaVersion, binaryTupleSlice);
}
+ byte[] createPayloadKey(ByteBuffer dataId) {
+ byte[] result = PAYLOAD_KEY_BUFFER.get()
+ .clear()
+ .putInt(tableId)
+ .putShort((short) partitionId)
+ .put(dataId)
+ .array();
+
+ dataId.rewind();
+
+ // Always use 0 for the last bit (tombstone flag), because it only makes sense when data ID is stored as a value.
+ setLastBit(result, result.length - 1, false);
+
+ return result;
+ }
+
+ /**
+ * Changes the last bit of the byte identified by an index in an array.
Review Comment:
it's the last bit if you count from the opposite side) Let's rename it to `setFirstBit`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "Phillippko (via GitHub)" <gi...@apache.org>.
Phillippko commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719414088
##########
modules/storage-rocksdb/src/test/java/org/apache/ignite/internal/storage/rocksdb/benchmarks/CommitManyWritesBenchmark.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.rocksdb.benchmarks;
+
+import static org.apache.ignite.internal.util.IgniteUtils.capacity;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.IntStream;
+import org.apache.ignite.configuration.ConfigurationValue;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.BinaryRowImpl;
+import org.apache.ignite.internal.storage.MvPartitionStorage;
+import org.apache.ignite.internal.storage.RowId;
+import org.apache.ignite.internal.storage.configurations.StorageConfiguration;
+import org.apache.ignite.internal.storage.engine.StorageTableDescriptor;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbMvPartitionStorage;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbStorageEngine;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbTableStorage;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbProfileView;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbStorageEngineConfiguration;
+import org.apache.ignite.internal.tx.TransactionIds;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+/**
+ * Benchmark for measuring the performance of {@link RocksDbMvPartitionStorage} in a scenario, when many entries are added in one big
+ * transaction and are then committed.
+ */
+@State(Scope.Benchmark)
Review Comment:
I think it would be nice to use explicit configuration for Fork, Measurement, etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo merged PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "sashapolo (via GitHub)" <gi...@apache.org>.
sashapolo commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1719434171
##########
modules/storage-rocksdb/src/test/java/org/apache/ignite/internal/storage/rocksdb/benchmarks/CommitManyWritesBenchmark.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.rocksdb.benchmarks;
+
+import static org.apache.ignite.internal.util.IgniteUtils.capacity;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.IntStream;
+import org.apache.ignite.configuration.ConfigurationValue;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.BinaryRowImpl;
+import org.apache.ignite.internal.storage.MvPartitionStorage;
+import org.apache.ignite.internal.storage.RowId;
+import org.apache.ignite.internal.storage.configurations.StorageConfiguration;
+import org.apache.ignite.internal.storage.engine.StorageTableDescriptor;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbMvPartitionStorage;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbStorageEngine;
+import org.apache.ignite.internal.storage.rocksdb.RocksDbTableStorage;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbProfileView;
+import org.apache.ignite.internal.storage.rocksdb.configuration.schema.RocksDbStorageEngineConfiguration;
+import org.apache.ignite.internal.tx.TransactionIds;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+/**
+ * Benchmark for measuring the performance of {@link RocksDbMvPartitionStorage} in a scenario, when many entries are added in one big
+ * transaction and are then committed.
+ */
+@State(Scope.Benchmark)
+public class CommitManyWritesBenchmark {
+ private static final String STORAGE_PROFILE_NAME = "test";
+ private static final int TABLE_ID = 1;
+ private static final int NUM_PARTITIONS = 50;
+ private static final int NUM_ROWS = 10_000;
+ private static final int ROW_LENGTH = 10_000;
+
+ private final HybridClock clock = new HybridClockImpl();
+
+ private RocksDbStorageEngine storageEngine;
+
+ private RocksDbTableStorage tableStorage;
+
+ /** Setup method. */
+ @Setup
+ public void setUp() throws IOException {
+ Path workDir = Files.createTempDirectory(CommitManyWritesBenchmark.class.getSimpleName());
+
+ storageEngine = new RocksDbStorageEngine(
+ "test",
+ engineConfiguration(),
+ storageConfiguration(),
+ workDir,
+ () -> {}
+ );
+
+ storageEngine.start();
+
+ var tableDescriptor = new StorageTableDescriptor(TABLE_ID, NUM_PARTITIONS, STORAGE_PROFILE_NAME);
+
+ tableStorage = storageEngine.createMvTable(tableDescriptor, indexId -> null);
+
+ CompletableFuture<?>[] createFutures = IntStream.range(0, NUM_PARTITIONS)
+ .mapToObj(tableStorage::createMvPartition)
+ .toArray(CompletableFuture[]::new);
+
+ CompletableFuture.allOf(createFutures).join();
+ }
+
+ /** Tear down method. */
+ @TearDown
+ public void tearDown() {
+ tableStorage.destroy();
+
+ storageEngine.stop();
+ }
+
+ /** Generated data for each thread. */
+ @State(Scope.Thread)
+ public static class GeneratedData {
+ int partitionId;
+
+ Map<RowId, BinaryRow> rows;
+
+ /** Setup method. */
+ @Setup
+ public void setUp() {
+ ThreadLocalRandom random = ThreadLocalRandom.current();
+
+ partitionId = random.nextInt(NUM_PARTITIONS);
+
+ rows = new HashMap<>(capacity(NUM_ROWS));
+
+ for (int i = 0; i < NUM_ROWS; i++) {
+ var rowId = new RowId(partitionId);
+
+ BinaryRow row = randomRow(random);
+
+ rows.put(rowId, row);
+ }
+ }
+
+ private static BinaryRow randomRow(ThreadLocalRandom random) {
+ ByteBuffer buffer = ByteBuffer.allocate(ROW_LENGTH);
+
+ random.nextBytes(buffer.array());
+
+ return new BinaryRowImpl(0, buffer);
+ }
+ }
+
+ private static RocksDbStorageEngineConfiguration engineConfiguration() {
+ RocksDbStorageEngineConfiguration config = mock(RocksDbStorageEngineConfiguration.class);
+
+ ConfigurationValue<Integer> flushDelayMillis = mock(ConfigurationValue.class);
+
+ when(flushDelayMillis.value()).thenReturn(100);
+
+ when(config.flushDelayMillis()).thenReturn(flushDelayMillis);
+
+ return config;
+ }
+
+ private static StorageConfiguration storageConfiguration() {
+ StorageConfiguration config = mock(StorageConfiguration.class);
+
+ NamedConfigurationTree profilesTree = mock(NamedConfigurationTree.class);
+ NamedListView profilesView = mock(NamedListView.class);
+ RocksDbProfileView rocksDbProfileView = mock(RocksDbProfileView.class);
+
+ when(rocksDbProfileView.name()).thenReturn(STORAGE_PROFILE_NAME);
+ when(rocksDbProfileView.size()).thenReturn(16777216L);
+ when(rocksDbProfileView.writeBufferSize()).thenReturn(16777216L);
+
+ when(config.profiles()).thenReturn(profilesTree);
+ when(profilesTree.value()).thenReturn(profilesView);
+ when(profilesView.iterator()).thenReturn(List.of(rocksDbProfileView).iterator());
+
+ return config;
+ }
+
+ /** Benchmark. */
+ @Benchmark
+ public void commitManyWrites(GeneratedData data) {
+ MvPartitionStorage partitionStorage = tableStorage.getMvPartition(data.partitionId);
Review Comment:
What will it change? I think this way it tests a more realistic situation (when RocksDb has data in it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "Phillippko (via GitHub)" <gi...@apache.org>.
Phillippko commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1721229554
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/GarbageCollector.java:
##########
@@ -325,28 +336,25 @@ private boolean checkHasNewerRowAndRemoveTombstone(
* @param gcElementRowId Row id of the element from the GC queue/
* @return Key of the row that needs to be garbage collected, or {@code null} if such row doesn't exist.
*/
- private @Nullable ByteBuffer getRowForGcKey(RocksIterator it, RowId gcElementRowId) {
+ private @Nullable ByteBuffer getDataIdForGcKey(RocksIterator it, RowId gcElementRowId) {
Review Comment:
Let's update the javadoc, it still says "return key of the row" instead of smt like "key of the row's data ID"
And name is misleading, we get key, not data Id itself
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/GarbageCollector.java:
##########
@@ -167,12 +178,12 @@ AddResult tryAddToGcQueue(WriteBatchWithIndex writeBatch, RowId rowId, HybridTim
* @param lowWatermark Low watermark.
* @return Garbage collected element descriptor.
*/
- @Nullable GcEntry peek(HybridTimestamp lowWatermark) {
+ @Nullable GcEntry peek(WriteBatchWithIndex writeBatch, HybridTimestamp lowWatermark) {
Review Comment:
Forgot to add param to javadoc
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -263,6 +294,55 @@ static BinaryRow deserializeRow(ByteBuffer buffer) {
return new BinaryRowImpl(schemaVersion, binaryTupleSlice);
}
+ byte[] createPayloadKey(ByteBuffer dataId) {
+ byte[] result = PAYLOAD_KEY_BUFFER.get()
+ .clear()
+ .putInt(tableId)
+ .putShort((short) partitionId)
+ .put(dataId)
+ .array();
+
+ dataId.rewind();
+
+ // Always use 0 for the last bit (tombstone flag), because it only makes sense when data ID is stored as a value.
+ setLastBit(result, result.length - 1, false);
+
+ return result;
+ }
+
+ /**
+ * Changes the last bit of the byte identified by an index in an array.
Review Comment:
We could use setRightMostBit, or setLsb / setLeastSignificantBit to avoid confusion
Maybe even "setTombstoneBit"
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -44,45 +43,46 @@
import org.jetbrains.annotations.Nullable;
import org.rocksdb.ColumnFamilyHandle;
import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
import org.rocksdb.RocksIterator;
import org.rocksdb.Slice;
import org.rocksdb.WriteBatchWithIndex;
/** Helper for the partition data. */
public final class PartitionDataHelper implements ManuallyCloseable {
+ /** Transaction id size (part of the transaction state). */
+ private static final int TX_ID_SIZE = 2 * Long.BYTES;
+
/** Position of row id inside the key. */
- static final int ROW_ID_OFFSET = TABLE_ID_SIZE + Short.BYTES;
+ static final int ROW_ID_OFFSET = TABLE_ID_SIZE + PARTITION_ID_SIZE;
- /** Size of the key without timestamp. */
Review Comment:
unintentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-19395 Use indirect addressing when storing rows in RocksDb [ignite-3]
Posted by "Phillippko (via GitHub)" <gi...@apache.org>.
Phillippko commented on code in PR #4200:
URL: https://github.com/apache/ignite-3/pull/4200#discussion_r1721248038
##########
modules/storage-rocksdb/src/main/java/org/apache/ignite/internal/storage/rocksdb/PartitionDataHelper.java:
##########
@@ -263,6 +294,55 @@ static BinaryRow deserializeRow(ByteBuffer buffer) {
return new BinaryRowImpl(schemaVersion, binaryTupleSlice);
}
+ byte[] createPayloadKey(ByteBuffer dataId) {
+ byte[] result = PAYLOAD_KEY_BUFFER.get()
+ .clear()
+ .putInt(tableId)
+ .putShort((short) partitionId)
+ .put(dataId)
+ .array();
+
+ dataId.rewind();
+
+ // Always use 0 for the last bit (tombstone flag), because it only makes sense when data ID is stored as a value.
+ setLastBit(result, result.length - 1, false);
+
+ return result;
+ }
+
+ /**
+ * Changes the last bit of the byte identified by an index in an array.
Review Comment:
We could use setRightMostBit, or setLsb / setLeastSignificantBit to avoid confusion
Maybe even "setTombstoneBit"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org