You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2022/12/27 08:42:59 UTC
[GitHub] [ignite-3] Pochatkin opened a new pull request, #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Pochatkin opened a new pull request, #1479:
URL: https://github.com/apache/ignite-3/pull/1479
https://issues.apache.org/jira/browse/IGNITE-18360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073489228
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
+ public static final ByteOrder ORDER = ByteOrder.LITTLE_ENDIAN;
+
+ /** Size of schema version length field. */
+ private static final int SCHEMA_VERSION_FLD_LEN = Short.BYTES;
+
+ /** Row schema version field offset. */
+ private static final int SCHEMA_VERSION_OFFSET = 0;
+
+ /** Row binary tuple field offset. */
+ private static final int TUPLE_OFFSET = SCHEMA_VERSION_OFFSET + SCHEMA_VERSION_FLD_LEN;
+
+ /** Row buffer. */
+ private final ByteBuffer buf;
+
+ /**
+ * Constructor.
+ *
+ * @param data Array representation of the row.
+ */
+ public TableRow(byte[] data) {
+ this(ByteBuffer.wrap(data).order(ORDER));
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param buf Buffer representing the row.
+ */
+ public TableRow(ByteBuffer buf) {
+ assert buf.order() == ORDER;
+ assert buf.position() == 0;
+
+ this.buf = buf;
+ }
+
+ /**
+ * Factory method which creates an instance using a schema version and an existing {@link BinaryTuple}.
+ *
+ * @param schemaVersion Schema version.
+ * @param binaryTuple Binary tuple.
+ * @return Created TableRow instance.
+ */
+ public static TableRow createFromTuple(int schemaVersion, BinaryTuple binaryTuple) {
Review Comment:
What is the reason of encoding schema version directly into the binary payload? This leads to too much memory manipulation. Storages can do that more efficiently.
I suggest storing version it as an integer field, and only assemble a byte[] when asked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] valepakh commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
valepakh commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1069400175
##########
modules/storage-api/src/testFixtures/java/org/apache/ignite/internal/storage/BaseMvStoragesTest.java:
##########
@@ -102,23 +104,20 @@ static void afterAll() {
kBinaryConverter = null;
}
- protected static BinaryRow binaryKey(TestKey key) {
- try {
- return kvMarshaller.marshal(key);
- } catch (MarshallerException e) {
- throw new IgniteException(e);
- }
+ protected static TableRow tableRow(TestKey key, TestValue value) {
+ return TableRowConverter.fromBinaryRow(binaryRow(key, value), kvBinaryConverter);
Review Comment:
I would do it after the marshalers are converted to the TableRow interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
korlov42 commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1068293570
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/TableRowAndRowId.java:
##########
@@ -18,31 +18,32 @@
package org.apache.ignite.internal.storage;
import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.TableRow;
import org.jetbrains.annotations.Nullable;
/**
* Wrapper that holds both {@link BinaryRow} and {@link RowId}. {@link BinaryRow} is null for tombstones.
Review Comment:
please fix all mentions of BinRow in javadoc
##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -1931,19 +1944,19 @@ private HybridTimestampMessage hybridTimestamp(HybridTimestamp tmstmp) {
*
* @param tablePartId {@link TablePartitionId} object to construct {@link UpdateCommand} object with.
* @param rowUuid Row UUID.
- * @param rowBuf {@link ByteBuffer} representation of {@link BinaryRow}.
+ * @param row {@link BinaryRow}.
Review Comment:
something is missed here
##########
modules/table/src/test/java/org/apache/ignite/internal/table/distributed/raft/snapshot/incoming/IncomingSnapshotCopierTest.java:
##########
@@ -370,8 +373,9 @@ private static List<ResponseEntry> createSnapshotMvDataEntries(MvPartitionStorag
return responseEntries;
}
- private static BinaryRow createBinaryRow(String key, String value) {
- return new RowAssembler(SCHEMA_DESCRIPTOR, 1, 1).appendString(key).appendString(value).build();
+ private static TableRow createRow(String key, String value) {
+ BinaryRow binaryRow = new RowAssembler(SCHEMA_DESCRIPTOR, 1, 1).appendString(key).appendString(value).build();
Review Comment:
again, let's try to avoid unnecessary conversion to BinRow here and other places as well
##########
modules/storage-api/src/testFixtures/java/org/apache/ignite/internal/storage/BaseMvStoragesTest.java:
##########
@@ -102,23 +104,20 @@ static void afterAll() {
kBinaryConverter = null;
}
- protected static BinaryRow binaryKey(TestKey key) {
- try {
- return kvMarshaller.marshal(key);
- } catch (MarshallerException e) {
- throw new IgniteException(e);
- }
+ protected static TableRow tableRow(TestKey key, TestValue value) {
+ return TableRowConverter.fromBinaryRow(binaryRow(key, value), kvBinaryConverter);
Review Comment:
is it possible to skip conversion to BinRow?
##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -1931,19 +1944,19 @@ private HybridTimestampMessage hybridTimestamp(HybridTimestamp tmstmp) {
*
* @param tablePartId {@link TablePartitionId} object to construct {@link UpdateCommand} object with.
* @param rowUuid Row UUID.
- * @param rowBuf {@link ByteBuffer} representation of {@link BinaryRow}.
+ * @param row {@link BinaryRow}.
* @param txId Transaction ID.
* @return Constructed {@link UpdateCommand} object.
*/
- private UpdateCommand updateCommand(TablePartitionId tablePartId, UUID rowUuid, ByteBuffer rowBuf, UUID txId) {
+ private UpdateCommand updateCommand(TablePartitionId tablePartId, UUID rowUuid, TableRow row, UUID txId) {
Review Comment:
why did you change method's signature?
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
+ public static final ByteOrder ORDER = ByteOrder.LITTLE_ENDIAN;
+
+ /** Size of schema version length field. */
+ private static final int SCHEMA_VERSION_FLD_LEN = Short.BYTES;
+
+ /** Row schema version field offset. */
+ private static final int SCHEMA_VERSION_OFFSET = 0;
+
+ /** Row binary tuple field offset. */
+ private static final int TUPLE_OFFSET = SCHEMA_VERSION_OFFSET + SCHEMA_VERSION_FLD_LEN;
+
+ /** Row buffer. */
+ private final ByteBuffer buf;
+
+ /**
+ * Constructor.
+ *
+ * @param data Array representation of the row.
+ */
+ public TableRow(byte[] data) {
+ this(ByteBuffer.wrap(data).order(ORDER));
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param buf Buffer representing the row.
+ */
+ public TableRow(ByteBuffer buf) {
+ assert buf.order() == ORDER;
+ assert buf.position() == 0;
+
+ this.buf = buf;
+ }
+
+ /**
+ * Factory method which creates an instance using a schema version and an existing {@link BinaryTuple}.
+ *
+ * @param schemaVersion Schema version.
+ * @param binaryTuple Binary tuple.
+ * @return Created TableRow instance.
+ */
+ public static TableRow createFromTuple(int schemaVersion, BinaryTuple binaryTuple) {
+ ByteBuffer tupleBuffer = binaryTuple.byteBuffer();
+ ByteBuffer buffer = ByteBuffer.allocate(SCHEMA_VERSION_FLD_LEN + tupleBuffer.limit()).order(ORDER);
+ buffer.putShort((short) schemaVersion);
+ buffer.put(tupleBuffer);
+ buffer.position(0);
+ return new TableRow(buffer);
+ }
+
+ /** Get row schema version. */
+ public int schemaVersion() {
+ return Short.toUnsignedInt(buf.getShort(SCHEMA_VERSION_OFFSET));
+ }
+
+ /** Get ByteBuffer slice representing the binary tuple. */
+ public ByteBuffer tupleSlice() {
+ try {
+ return buf.position(TUPLE_OFFSET).slice().order(ORDER);
+ } finally {
+ buf.position(0); // Reset bounds.
Review Comment:
we have to restore previous position here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ademakov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ademakov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073371038
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Columnar storage does not store schema version with rows right now.
The whole effort was triggered because BinaryRow had too many assumptions that were plain wrong for columnar. Now this PR adds TableRow that basically has the same problem.
Schema version should be handled in the same fashion as rowid is handled now. That is it should be passed as an additional parameter or field or whatever. It should never be embedded into the binary data. This way we will avoid troubles for storage engines that do not need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
korlov42 commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073728273
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
+ public static final ByteOrder ORDER = ByteOrder.LITTLE_ENDIAN;
+
+ /** Size of schema version length field. */
+ private static final int SCHEMA_VERSION_FLD_LEN = Short.BYTES;
+
+ /** Row schema version field offset. */
+ private static final int SCHEMA_VERSION_OFFSET = 0;
+
+ /** Row binary tuple field offset. */
+ private static final int TUPLE_OFFSET = SCHEMA_VERSION_OFFSET + SCHEMA_VERSION_FLD_LEN;
+
+ /** Row buffer. */
+ private final ByteBuffer buf;
+
+ /**
+ * Constructor.
+ *
+ * @param data Array representation of the row.
+ */
+ public TableRow(byte[] data) {
+ this(ByteBuffer.wrap(data).order(ORDER));
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param buf Buffer representing the row.
+ */
+ public TableRow(ByteBuffer buf) {
+ assert buf.order() == ORDER;
+ assert buf.position() == 0;
+
+ this.buf = buf;
+ }
+
+ /**
+ * Factory method which creates an instance using a schema version and an existing {@link BinaryTuple}.
+ *
+ * @param schemaVersion Schema version.
+ * @param binaryTuple Binary tuple.
+ * @return Created TableRow instance.
+ */
+ public static TableRow createFromTuple(int schemaVersion, BinaryTuple binaryTuple) {
Review Comment:
> What is the reason of encoding schema version directly into the binary payload?
it is a direct replacement for ByteBufferRow to facilitate migration
> I suggest storing version as an integer field
if the storage doesn't mind to work not with abstract bytes, but with a specific row object, let's do this way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
korlov42 commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072247433
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
I'm not sure that erasing the boundary between data and its metadata is a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] valepakh commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
valepakh commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072139004
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
Review Comment:
BinaryTuple is an existing class which is an utility that uses a schema to read values. I don't like the name either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ademakov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ademakov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072993260
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Within a transaction at any given moment of time only one table schema is in effect. It does not change arbitrarily. It is redundant to specify schema for every inserted row. The storage engine does not have to store the schema version with every row. It is sufficient to remember the transaction timestamp since which a given table schema version is in effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1083750250
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
+ public static final ByteOrder ORDER = ByteOrder.LITTLE_ENDIAN;
+
+ /** Size of schema version length field. */
+ private static final int SCHEMA_VERSION_FLD_LEN = Short.BYTES;
+
+ /** Row schema version field offset. */
+ private static final int SCHEMA_VERSION_OFFSET = 0;
+
+ /** Row binary tuple field offset. */
+ private static final int TUPLE_OFFSET = SCHEMA_VERSION_OFFSET + SCHEMA_VERSION_FLD_LEN;
+
+ /** Row buffer. */
+ private final ByteBuffer buf;
+
+ /**
+ * Constructor.
+ *
+ * @param data Array representation of the row.
+ */
+ public TableRow(byte[] data) {
+ this(ByteBuffer.wrap(data).order(ORDER));
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param buf Buffer representing the row.
+ */
+ public TableRow(ByteBuffer buf) {
+ assert buf.order() == ORDER;
+ assert buf.position() == 0;
+
+ this.buf = buf;
+ }
+
+ /**
+ * Factory method which creates an instance using a schema version and an existing {@link BinaryTuple}.
+ *
+ * @param schemaVersion Schema version.
+ * @param binaryTuple Binary tuple.
+ * @return Created TableRow instance.
+ */
+ public static TableRow createFromTuple(int schemaVersion, BinaryTuple binaryTuple) {
Review Comment:
Ok, but we should fix this in the future. There must be a TODO. Current approach is not efficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072106772
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
Review Comment:
Why not use **BinaryTuple**? This change looks strange.
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Why not use **BinaryTuple**? This change looks strange.
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/TableRowAndRowId.java:
##########
@@ -17,32 +17,32 @@
package org.apache.ignite.internal.storage;
-import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.TableRow;
import org.jetbrains.annotations.Nullable;
/**
- * Wrapper that holds both {@link BinaryRow} and {@link RowId}. {@link BinaryRow} is null for tombstones.
+ * Wrapper that holds both {@link TableRow} and {@link RowId}. {@link TableRow} is null for tombstones.
*/
-public class BinaryRowAndRowId {
- /** Binary row. */
- private final @Nullable BinaryRow binaryRow;
+public class TableRowAndRowId {
+ /** Table row. */
+ private final @Nullable TableRow tableRow;
Review Comment:
Why not use **BinaryTuple**? This change looks strange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] valepakh commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
valepakh commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1069449286
##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -1931,19 +1944,19 @@ private HybridTimestampMessage hybridTimestamp(HybridTimestamp tmstmp) {
*
* @param tablePartId {@link TablePartitionId} object to construct {@link UpdateCommand} object with.
* @param rowUuid Row UUID.
- * @param rowBuf {@link ByteBuffer} representation of {@link BinaryRow}.
+ * @param row {@link BinaryRow}.
* @param txId Transaction ID.
* @return Constructed {@link UpdateCommand} object.
*/
- private UpdateCommand updateCommand(TablePartitionId tablePartId, UUID rowUuid, ByteBuffer rowBuf, UUID txId) {
+ private UpdateCommand updateCommand(TablePartitionId tablePartId, UUID rowUuid, TableRow row, UUID txId) {
Review Comment:
Mainly to reduce clutter at the call site, but probably it would be better if the update and updateAll singatures would be similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
korlov42 commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1071959373
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/TableRowAndRowId.java:
##########
@@ -18,31 +18,32 @@
package org.apache.ignite.internal.storage;
import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.TableRow;
import org.jetbrains.annotations.Nullable;
/**
* Wrapper that holds both {@link BinaryRow} and {@link RowId}. {@link BinaryRow} is null for tombstones.
Review Comment:
there are still mentions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ademakov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ademakov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073371038
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Columnar storage does not store schema version with rows right now.
The whole effort was triggered because BinaryRow had too many assumptions that were plain wrong for columnar. Now this PR adds TableRow that basically has the same problem.
Schema version should be handled in the same fashion as rowid is handled now. That is it should be passed as an additional parameter or field or whatever. It should never be embedded into the row binary data. This way we will avoid troubles for storage engines that do not need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
korlov42 commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072122879
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
table can change over time, so we must store the version of the schema that was used to build the tuple so we can read its contents later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ademakov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ademakov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072993260
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Within a transaction at any given moment of time only one table schema is in effect. It does not change arbitrarily. It is redundant to specify the schema for every inserted row. The storage engine does not have to store the schema version together with every row too. It is sufficient to remember the transaction timestamp since which a given table schema version is in effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ademakov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ademakov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072993260
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Within a transaction at any given moment of time only one table schema is in effect. It does not change arbitrarily. It is redundant to specify schema for every inserted row. The storage engine does not have to store the schema version with every row. It is sufficient to remember the transaction timestamps since which a given table schema version is in effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ademakov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ademakov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072993260
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Within a transaction at any given moment of time only one table schema is in effect. It does not change arbitrarily. It is redundant to specify schema for every inserted row. The storage engine does not have to store the schema with every row. It is sufficient to remember the transaction timestamps since which a given table schema version is in effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1072175125
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Maybe it would be more correct to always store the version in a tuple?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
ibessonov commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073489228
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
+ public static final ByteOrder ORDER = ByteOrder.LITTLE_ENDIAN;
+
+ /** Size of schema version length field. */
+ private static final int SCHEMA_VERSION_FLD_LEN = Short.BYTES;
+
+ /** Row schema version field offset. */
+ private static final int SCHEMA_VERSION_OFFSET = 0;
+
+ /** Row binary tuple field offset. */
+ private static final int TUPLE_OFFSET = SCHEMA_VERSION_OFFSET + SCHEMA_VERSION_FLD_LEN;
+
+ /** Row buffer. */
+ private final ByteBuffer buf;
+
+ /**
+ * Constructor.
+ *
+ * @param data Array representation of the row.
+ */
+ public TableRow(byte[] data) {
+ this(ByteBuffer.wrap(data).order(ORDER));
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param buf Buffer representing the row.
+ */
+ public TableRow(ByteBuffer buf) {
+ assert buf.order() == ORDER;
+ assert buf.position() == 0;
+
+ this.buf = buf;
+ }
+
+ /**
+ * Factory method which creates an instance using a schema version and an existing {@link BinaryTuple}.
+ *
+ * @param schemaVersion Schema version.
+ * @param binaryTuple Binary tuple.
+ * @return Created TableRow instance.
+ */
+ public static TableRow createFromTuple(int schemaVersion, BinaryTuple binaryTuple) {
Review Comment:
What is the reason of encoding schema version directly into the binary payload? This leads to too much memory manipulation. Storages can do that more efficiently.
I suggest storing version as an integer field, and only assemble a byte[] when asked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
korlov42 commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073718425
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
Ignite does not have columnar storage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] valepakh commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
valepakh commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073735758
##########
modules/schema/src/main/java/org/apache/ignite/internal/schema/TableRow.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.schema;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/** Heap byte buffer-based row. */
+public class TableRow {
+ public static final ByteOrder ORDER = ByteOrder.LITTLE_ENDIAN;
+
+ /** Size of schema version length field. */
+ private static final int SCHEMA_VERSION_FLD_LEN = Short.BYTES;
+
+ /** Row schema version field offset. */
+ private static final int SCHEMA_VERSION_OFFSET = 0;
+
+ /** Row binary tuple field offset. */
+ private static final int TUPLE_OFFSET = SCHEMA_VERSION_OFFSET + SCHEMA_VERSION_FLD_LEN;
+
+ /** Row buffer. */
+ private final ByteBuffer buf;
+
+ /**
+ * Constructor.
+ *
+ * @param data Array representation of the row.
+ */
+ public TableRow(byte[] data) {
+ this(ByteBuffer.wrap(data).order(ORDER));
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param buf Buffer representing the row.
+ */
+ public TableRow(ByteBuffer buf) {
+ assert buf.order() == ORDER;
+ assert buf.position() == 0;
+
+ this.buf = buf;
+ }
+
+ /**
+ * Factory method which creates an instance using a schema version and an existing {@link BinaryTuple}.
+ *
+ * @param schemaVersion Schema version.
+ * @param binaryTuple Binary tuple.
+ * @return Created TableRow instance.
+ */
+ public static TableRow createFromTuple(int schemaVersion, BinaryTuple binaryTuple) {
Review Comment:
I think we should make these migrations as simple as possible and avoid making changes not relevant to the binary format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 merged pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by "korlov42 (via GitHub)" <gi...@apache.org>.
korlov42 merged PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] korlov42 commented on a diff in pull request #1479: IGNITE-18360 Migrate storage to new Binary Tuple format
Posted by GitBox <gi...@apache.org>.
korlov42 commented on code in PR #1479:
URL: https://github.com/apache/ignite-3/pull/1479#discussion_r1073145119
##########
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java:
##########
@@ -151,7 +151,7 @@ public interface MvPartitionStorage extends ManuallyCloseable {
* @throws TxIdMismatchException If there's another pending update associated with different transaction id.
* @throws StorageException If failed to write data to the storage.
*/
- @Nullable BinaryRow addWrite(RowId rowId, @Nullable BinaryRow row, UUID txId, UUID commitTableId, int commitPartitionId)
+ @Nullable TableRow addWrite(RowId rowId, @Nullable TableRow row, UUID txId, UUID commitTableId, int commitPartitionId)
Review Comment:
> The storage engine does not have to store the schema version together with every row too. It is sufficient to remember the transaction timestamp since which a given table schema version is in effect.
That is not how it's working right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org