You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "virajjasani (via GitHub)" <gi...@apache.org> on 2024/07/12 06:34:34 UTC

[PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

virajjasani opened a new pull request, #1937:
URL: https://github.com/apache/phoenix/pull/1937

   Jira: PHOENIX-7357
   
   Design Doc: https://docs.google.com/document/d/1-eZ4cbLVEPC4z7Iy1U_VJgNaKC9odkRVzcVPV3ZqOcI/edit?usp=sharing
   
   As of today, Phoenix provides several variable length as well as fixed length data types. One of the variable length data types is VARBINARY. It is variable length binary blob. Using VARBINARY as only primary key can be considered as if using HBase row key.
   
   HBase provides a single row key. Any client application that requires using more than one column for primary keys, using HBase requires special handling of storing both column values as a single binary row key. Phoenix provides the ability to use more than one primary key by providing composite primary keys. Composite primary key can contain any number of primary key columns. Phoenix also provides the ability to add new nullable primary key columns to the existing composite primary keys. Phoenix uses HBase as its backing store. In order to provide the ability for users to define multiple primary keys, Phoenix internally concatenates binary encoded values of each primary key column value and uses concatenated binary value as HBase row key. In order to efficiently concatenate as well as retrieve individual primary key values, Phoenix implements two ways:
   
   For fixed length columns: The length of the given column is determined by the maximum length of the column. As part of the read flow, while iterating through the row key, fixed length numbers of bytes are retrieved while reading. While writing, if the original encoded value of the given column has less number of bytes, additional null bytes (\x00) are padded until the fixed length is filled up. Hence, for smaller values, we end up wasting some space.
   For variable length columns: Since we cannot know the length of the value of variable length data type in advance, a separator or terminator byte is used. Phoenix uses null byte as separator (\x00) byte. As of today, VARCHAR is the most commonly used variable length data type and since VARCHAR represents String, null byte is not part of valid String characters. Hence, it can be effectively used to determine when to terminate the given VARCHAR value.
    
   
   The null byte (\x00) works fine as a separator for VARCHAR. However, it cannot be used as a separator byte for VARBINARY because VARBINARY can contain any binary blob values. Due to this, Phoenix has restrictions for VARBINARY type: 
   
    
   
   It can only be used as the last part of the composite primary key.
   It cannot be used as a DESC order primary key column.
    
   
   Using VARBINARY data type as an earlier portion of the composite primary key is a valid use case. One can also use multiple VARBINARY primary key columns. After all, Phoenix provides the ability to use multiple primary key columns for users.
   
   Besides, using secondary index on data table means that the composite primary key of secondary index table includes: 
   
   ```
   <secondary-index-col1> <secondary-index-col2> … <secondary-index-colN> <primary-key-col1> <primary-key-col2> … <primary-key-colN>
   ```
   
   As primary key columns are appended to the secondary indexes columns, one cannot create a secondary index on any VARBINARY column.
   
   The proposal of this Jira is to introduce new data type VARBINARY_ENCODED, which has no restriction of being considered as composite primary key prefix or using it as DESC ordered column.
   
   This means, we need to effectively distinguish where the variable length binary data terminates in the absence of fixed length information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#issuecomment-2231515131

   Could we introduce an API which always returns a byte array and we simply iterate over the array. That should work regardless whether there is one separator byte or multiple and reduce the amount of code duplication.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1676506866


##########
phoenix-core-client/src/main/java/org/apache/phoenix/expression/RowValueConstructorExpression.java:
##########
@@ -259,7 +276,23 @@ public boolean evaluate(Tuple tuple, ImmutableBytesWritable ptr) {
             throw new RuntimeException(e); //Impossible.
         }
     }
-    
+
+    private boolean hasSeparatorBytes(byte[] outputBytes, int outputSize, int k) {
+        if (getChildren().get(k).getDataType() != PVarbinaryEncoded.INSTANCE) {
+            return outputBytes[outputSize - 1] == SchemaUtil.getSeparatorByte(true, false,
+                getChildren().get(k));
+        } else {
+            byte[] sepBytes = SchemaUtil.getSeparatorBytesForVarBinaryEncoded(true, false,
+                getChildren().get(k).getSortOrder());
+            return outputSize >= 2 && outputBytes[outputSize - 1] == sepBytes[1]
+                && outputBytes[outputSize - 2] == sepBytes[0];
+        }
+    }
+
+    private static int getSeparatorBytesLen(Expression expression) {

Review Comment:
   Sure, will do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1757955825


##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/RowKeySchema.java:
##########
@@ -232,11 +262,24 @@ public Boolean previous(ImmutableBytesWritable ptr, int position, int minOffset)
             byte[] buf = ptr.get();
             int offset = ptr.getOffset()-1-offsetAdjustment;
             // Separator always zero byte if zero length
-            if (offset > minOffset && buf[offset] != QueryConstants.SEPARATOR_BYTE) {
-                byte sepByte = SchemaUtil.getSeparatorByte(rowKeyOrderOptimizable, false, field);
-                do {
-                    offset--;
-                } while (offset > minOffset && buf[offset] != sepByte);
+            if (field.getDataType() != PVarbinaryEncoded.INSTANCE) {
+                if (offset > minOffset && buf[offset] != QueryConstants.SEPARATOR_BYTE) {
+                    byte sepByte =
+                        SchemaUtil.getSeparatorByte(rowKeyOrderOptimizable, false, field);
+                    do {
+                        offset--;
+                    } while (offset > minOffset && buf[offset] != sepByte);
+                }
+            } else {
+                if (offset > minOffset && SchemaUtil.areSeparatorBytesForVarBinaryEncoded(buf,
+                    offset, SortOrder.ASC)) {
+                    boolean ascSort =
+                        !rowKeyOrderOptimizable || field.getSortOrder() == SortOrder.ASC;
+                    do {
+                        offset++;

Review Comment:
   is this supposed to be offset-- ?



##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/RowKeySchema.java:
##########
@@ -159,30 +160,59 @@ private Boolean next(ImmutableBytesWritable ptr, int position, int maxOffset, bo
         // If positioned at SEPARATOR_BYTE, skip it.
         // Don't look back at previous fields if this is our first next call, as
         // we may have a partial key for RVCs that doesn't include the leading field.
-        if (position > 0 && !isFirst && !getField(position-1).getDataType().isFixedWidth()) {
-            ptr.set(ptr.get(), ptr.getOffset()+ptr.getLength()+1, 0);
+        if (position > 0 && !isFirst && !getField(position - 1).getDataType().isFixedWidth()) {
+            if (getField(position - 1).getDataType() != PVarbinaryEncoded.INSTANCE) {

Review Comment:
   Can we replace hard coded values 1 and 2 with properly named constants?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1759134665


##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/RowKeySchema.java:
##########
@@ -232,11 +262,24 @@ public Boolean previous(ImmutableBytesWritable ptr, int position, int minOffset)
             byte[] buf = ptr.get();
             int offset = ptr.getOffset()-1-offsetAdjustment;
             // Separator always zero byte if zero length
-            if (offset > minOffset && buf[offset] != QueryConstants.SEPARATOR_BYTE) {
-                byte sepByte = SchemaUtil.getSeparatorByte(rowKeyOrderOptimizable, false, field);
-                do {
-                    offset--;
-                } while (offset > minOffset && buf[offset] != sepByte);
+            if (field.getDataType() != PVarbinaryEncoded.INSTANCE) {
+                if (offset > minOffset && buf[offset] != QueryConstants.SEPARATOR_BYTE) {
+                    byte sepByte =
+                        SchemaUtil.getSeparatorByte(rowKeyOrderOptimizable, false, field);
+                    do {
+                        offset--;
+                    } while (offset > minOffset && buf[offset] != sepByte);
+                }
+            } else {
+                if (offset > minOffset && SchemaUtil.areSeparatorBytesForVarBinaryEncoded(buf,
+                    offset, SortOrder.ASC)) {
+                    boolean ascSort =
+                        !rowKeyOrderOptimizable || field.getSortOrder() == SortOrder.ASC;
+                    do {
+                        offset++;

Review Comment:
   Excellent catch, previous() is not being tested by the tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1679839171


##########
phoenix-core-client/src/main/java/org/apache/phoenix/expression/RowValueConstructorExpression.java:
##########
@@ -220,14 +223,24 @@ public boolean evaluate(Tuple tuple, ImmutableBytesWritable ptr) {
                             // we'd have an ambiguity if this value happened to be the
                             // min possible value.
                             previousCarryOver = childType == null || childType.isFixedWidth();
-                            int bytesToWrite = getExpressionByteCount(child);
-                            for (int m = 0; m < bytesToWrite; m++) {
-                                output.write(QueryConstants.SEPARATOR_BYTE);
+                            if (childType == PVarbinaryEncoded.INSTANCE) {
+                                output.write(QueryConstants.VARBINARY_ENCODED_SEPARATOR_BYTES);
+                            } else {
+                                int bytesToWrite = getExpressionByteCount(child);
+                                for (int m = 0; m < bytesToWrite; m++) {
+                                    output.write(QueryConstants.SEPARATOR_BYTE);
+                                }
                             }
                         } else {
                             output.write(tempPtr.get(), tempPtr.getOffset(), tempPtr.getLength());
                             if (!childType.isFixedWidth()) {
-                                output.write(SchemaUtil.getSeparatorByte(true, false, child));
+                                if (childType != PVarbinaryEncoded.INSTANCE) {
+                                    output.write(SchemaUtil.getSeparatorByte(true, false, child));
+                                } else {
+                                    output.write(
+                                        SchemaUtil.getSeparatorBytesForVarBinaryEncoded(true, false,
+                                            child.getSortOrder()));
+                                }

Review Comment:
   Could we introduce an API which always returns a byte array and we simply iterate over the array. That should work regardless whether there is one separator byte or multiple and reduce the amount of code duplication.



##########
phoenix-core-client/src/main/java/org/apache/phoenix/filter/SkipScanFilter.java:
##########
@@ -576,13 +577,21 @@ private void setStartKey() {
 
     private int setStartKey(ImmutableBytesWritable ptr, int offset, int i, int nSlots, boolean atEndOfKey) {
         int length = ptr.getOffset() - offset;
-        startKey = copyKey(startKey, length + this.maxKeyLength, ptr.get(), offset, length);
+        startKey = copyKey(startKey, length + this.maxKeyLength + 1, ptr.get(), offset, length);
         startKeyLength = length;
         // Add separator byte if we're at end of the key, since trailing separator bytes are stripped
-        if (atEndOfKey && i > 0 && i-1 < nSlots) {
-            Field field = schema.getField(i-1);
+        if (atEndOfKey && i > 0 && i - 1 < nSlots) {
+            Field field = schema.getField(i - 1);
             if (!field.getDataType().isFixedWidth()) {
-                startKey[startKeyLength++] = SchemaUtil.getSeparatorByte(schema.rowKeyOrderOptimizable(), true, field);
+                if (field.getDataType() != PVarbinaryEncoded.INSTANCE) {
+                    startKey[startKeyLength++] =
+                        SchemaUtil.getSeparatorByte(schema.rowKeyOrderOptimizable(), true, field);
+                } else {
+                    byte[] sepBytes = SchemaUtil.getSeparatorBytesForVarBinaryEncoded(
+                        schema.rowKeyOrderOptimizable(), true, field.getSortOrder());
+                    startKey[startKeyLength++] = sepBytes[0];
+                    startKey[startKeyLength++] = sepBytes[1];
+                }

Review Comment:
   Could we introduce an API which always returns a byte array and we simply iterate over the array. That should work regardless whether there is one separator byte or multiple and reduce the amount of code duplication.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #1937:
URL: https://github.com/apache/phoenix/pull/1937


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "ranganathg (via GitHub)" <gi...@apache.org>.
ranganathg commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1675718435


##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/VarBinary1IT.java:
##########
@@ -0,0 +1,2205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Properties;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.schema.PTableImpl;
+import org.apache.phoenix.util.PropertiesUtil;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+
+@Category(ParallelStatsDisabledTest.class)
+@RunWith(Parameterized.class)
+public class VarBinary1IT extends ParallelStatsDisabledIT {

Review Comment:
   VarBinaryEncodedIT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1759199610


##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/RowKeySchema.java:
##########
@@ -159,30 +160,59 @@ private Boolean next(ImmutableBytesWritable ptr, int position, int maxOffset, bo
         // If positioned at SEPARATOR_BYTE, skip it.
         // Don't look back at previous fields if this is our first next call, as
         // we may have a partial key for RVCs that doesn't include the leading field.
-        if (position > 0 && !isFirst && !getField(position-1).getDataType().isFixedWidth()) {
-            ptr.set(ptr.get(), ptr.getOffset()+ptr.getLength()+1, 0);
+        if (position > 0 && !isFirst && !getField(position - 1).getDataType().isFixedWidth()) {
+            if (getField(position - 1).getDataType() != PVarbinaryEncoded.INSTANCE) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#issuecomment-2358743673

   > Can you add tests for both varbinary and varbinary encoded to WhereCompilerTest# testWhereInclusion and PartialIndexIT?
   
   Thanks @kadirozde! Added the tests in the latest revision, and addressed previous review comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1676506594


##########
phoenix-core-client/src/main/java/org/apache/phoenix/expression/CoerceExpression.java:
##########
@@ -145,11 +147,30 @@ public void write(DataOutput output) throws IOException {
 
     @Override
     public boolean evaluate(Tuple tuple, ImmutableBytesWritable ptr) {
-        if (getChild().evaluate(tuple, ptr)) {
-            getDataType().coerceBytes(ptr, null, getChild().getDataType(),
-                    getChild().getMaxLength(), null, getChild().getSortOrder(), 
-                    maxLength, null, getSortOrder(), rowKeyOrderOptimizable);
-            return true;
+        if (getChild() instanceof LiteralExpression
+            && getChild().getDataType() == PVarbinary.INSTANCE

Review Comment:
   I will write more comments, filling up design doc section on this as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1762185174


##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/RowKeyValueAccessor.java:
##########
@@ -37,14 +47,58 @@
  * 
  * @since 0.1
  */
-public class RowKeyValueAccessor implements Writable   {
+public class RowKeyValueAccessor implements Writable {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(RowKeyValueAccessor.class);
+
     /**
      * Constructor solely for use during deserialization. Should not
      * otherwise be used.
      */
     public RowKeyValueAccessor() {
     }
-    
+
+    /**
+     * The class used to keep list of booleans representing whether the data type of the given
+     * field is of type VARBINARY_ENCODED. For example, let's say we have variable length
+     * data types in the order (VARCHAR, VARBINARY_ENCODED, VARCHAR). The corresponding object
+     * of this class will have binaryEncodedDataTypes list with values (false, true, false).
+     * The array of this class is serialized and deserialized together with other fields of
+     * RowKeyValueAccessor. The value of the list determines which separator bytes need to be
+     * used while iterating through the rowkey bytes.
+     */
+    static class BinaryEncodedTypesLists {

Review Comment:
   BinaryEncodedTypesLists does not tell what this is about.  It gets confusing when you declare BinaryEncodedTypesLists[] binaryEncodedTypesLists later.  I think ListOfBinaryEncodedTypeFlags or something similar would be more readable. Also did you consider using BitSet? If you use it, you do not need these two classes. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#issuecomment-2356622243

   Can you add tests for both varbinary and varbinary encoded to WhereCompilerTest# testWhereInclusion and PartialIndexIT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "ranganathg (via GitHub)" <gi...@apache.org>.
ranganathg commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1675704572


##########
phoenix-core-client/src/main/java/org/apache/phoenix/expression/CoerceExpression.java:
##########
@@ -145,11 +147,30 @@ public void write(DataOutput output) throws IOException {
 
     @Override
     public boolean evaluate(Tuple tuple, ImmutableBytesWritable ptr) {
-        if (getChild().evaluate(tuple, ptr)) {
-            getDataType().coerceBytes(ptr, null, getChild().getDataType(),
-                    getChild().getMaxLength(), null, getChild().getSortOrder(), 
-                    maxLength, null, getSortOrder(), rowKeyOrderOptimizable);
-            return true;
+        if (getChild() instanceof LiteralExpression
+            && getChild().getDataType() == PVarbinary.INSTANCE

Review Comment:
   what is this check fo?



##########
phoenix-core-client/src/main/java/org/apache/phoenix/expression/CoerceExpression.java:
##########
@@ -145,11 +147,30 @@ public void write(DataOutput output) throws IOException {
 
     @Override
     public boolean evaluate(Tuple tuple, ImmutableBytesWritable ptr) {
-        if (getChild().evaluate(tuple, ptr)) {
-            getDataType().coerceBytes(ptr, null, getChild().getDataType(),
-                    getChild().getMaxLength(), null, getChild().getSortOrder(), 
-                    maxLength, null, getSortOrder(), rowKeyOrderOptimizable);
-            return true;
+        if (getChild() instanceof LiteralExpression
+            && getChild().getDataType() == PVarbinary.INSTANCE

Review Comment:
   what is this check for?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#issuecomment-2285156517

   > Could we introduce an API which always returns a byte array and we simply iterate over the array. That should work regardless whether there is one separator byte or multiple and reduce the amount of code duplication.
   
   This is done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1676507834


##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/VarBinary1IT.java:
##########
@@ -0,0 +1,2205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Properties;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.schema.PTableImpl;
+import org.apache.phoenix.util.PropertiesUtil;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+
+@Category(ParallelStatsDisabledTest.class)
+@RunWith(Parameterized.class)
+public class VarBinary1IT extends ParallelStatsDisabledIT {

Review Comment:
   We don't even have much coverage for VARBINARY (obviously because it is not used as tenant id, it is not used as any but last part of composite pk), so maybe that's why i kept it VarBinary but yes let me change this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1762181131


##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/RowKeyValueAccessor.java:
##########
@@ -37,14 +47,58 @@
  * 
  * @since 0.1
  */
-public class RowKeyValueAccessor implements Writable   {
+public class RowKeyValueAccessor implements Writable {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(RowKeyValueAccessor.class);
+
     /**
      * Constructor solely for use during deserialization. Should not
      * otherwise be used.
      */
     public RowKeyValueAccessor() {
     }
-    
+
+    /**
+     * The class used to keep list of booleans representing whether the data type of the given
+     * field is of type VARBINARY_ENCODED. For example, let's say we have variable length
+     * data types in the order (VARCHAR, VARBINARY_ENCODED, VARCHAR). The corresponding object
+     * of this class will have binaryEncodedDataTypes list with values (false, true, false).
+     * The array of this class is serialized and deserialized together with other fields of
+     * RowKeyValueAccessor. The value of the list determines which separator bytes need to be
+     * used while iterating through the rowkey bytes.
+     */
+    static class BinaryEncodedTypesLists {
+        private final List<Boolean> binaryEncodedDataTypes = new ArrayList<>();
+
+        public void addBinaryEncodedDataTypes(boolean val) {
+            this.binaryEncodedDataTypes.add(val);
+        }
+
+        public List<Boolean> getBinaryEncodedDataTypes() {
+            return binaryEncodedDataTypes;
+        }
+    }
+
+    /**
+     * The class used to keep list of booleans representing whether the order of the given
+     * field is Ascending. For example, let's say we have rowkey with pk columns
+     * (A ASC, B DESC, C ASC). The corresponding object of this class will have sortOrderAsc list
+     * with values (true, false, true). The array of this class is serialized and deserialized
+     * together with other fields of RowKeyValueAccessor. The value of the list determines which
+     * separator bytes need to be used while iterating through the rowkey bytes.
+     */
+    static class SortOrderLists {

Review Comment:
   This should read SortOrderList.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "ranganathg (via GitHub)" <gi...@apache.org>.
ranganathg commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1675708368


##########
phoenix-core-client/src/main/java/org/apache/phoenix/expression/RowValueConstructorExpression.java:
##########
@@ -241,12 +254,16 @@ public boolean evaluate(Tuple tuple, ImmutableBytesWritable ptr) {
                     // Additionally for b/w compat with clients older than 4.14.1 -
                     // If SortOorder.ASC then always strip trailing separator byte (as before)
                     // else only strip for >= 4.14 client (when STRIP_TRAILING_SEPARATOR_BYTE bit is set)
-                    for (int k = expressionCount -1 ; 
-                            k >=0 &&  getChildren().get(k).getDataType() != null 
-                                  && !getChildren().get(k).getDataType().isFixedWidth()
-                                  && outputBytes[outputSize-1] == SchemaUtil.getSeparatorByte(true, false, getChildren().get(k))
-                                  &&  (getChildren().get(k).getSortOrder() == SortOrder.ASC ? true : isStripTrailingSepByte()) ; k--) {
+                    for (int k = expressionCount - 1; k >= 0
+                        && getChildren().get(k).getDataType() != null
+                        && !getChildren().get(k).getDataType().isFixedWidth()
+                        && hasSeparatorBytes(outputBytes, outputSize, k)
+                        && (getChildren().get(k).getSortOrder() == SortOrder.ASC
+                        || isStripTrailingSepByte()); k--) {

Review Comment:
   large condition in for block, can this be extracted to a method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "ranganathg (via GitHub)" <gi...@apache.org>.
ranganathg commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1675709859


##########
phoenix-core-client/src/main/java/org/apache/phoenix/expression/RowValueConstructorExpression.java:
##########
@@ -259,7 +276,23 @@ public boolean evaluate(Tuple tuple, ImmutableBytesWritable ptr) {
             throw new RuntimeException(e); //Impossible.
         }
     }
-    
+
+    private boolean hasSeparatorBytes(byte[] outputBytes, int outputSize, int k) {
+        if (getChildren().get(k).getDataType() != PVarbinaryEncoded.INSTANCE) {
+            return outputBytes[outputSize - 1] == SchemaUtil.getSeparatorByte(true, false,
+                getChildren().get(k));
+        } else {
+            byte[] sepBytes = SchemaUtil.getSeparatorBytesForVarBinaryEncoded(true, false,
+                getChildren().get(k).getSortOrder());
+            return outputSize >= 2 && outputBytes[outputSize - 1] == sepBytes[1]
+                && outputBytes[outputSize - 2] == sepBytes[0];
+        }
+    }
+
+    private static int getSeparatorBytesLen(Expression expression) {

Review Comment:
   you can keep it getSeparatorBytesLength?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PHOENIX-7357 New variable length binary data type: VARBINARY_ENCODED [phoenix]

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1937:
URL: https://github.com/apache/phoenix/pull/1937#discussion_r1759134074


##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/RowKeySchema.java:
##########
@@ -159,30 +160,59 @@ private Boolean next(ImmutableBytesWritable ptr, int position, int maxOffset, bo
         // If positioned at SEPARATOR_BYTE, skip it.
         // Don't look back at previous fields if this is our first next call, as
         // we may have a partial key for RVCs that doesn't include the leading field.
-        if (position > 0 && !isFirst && !getField(position-1).getDataType().isFixedWidth()) {
-            ptr.set(ptr.get(), ptr.getOffset()+ptr.getLength()+1, 0);
+        if (position > 0 && !isFirst && !getField(position - 1).getDataType().isFixedWidth()) {
+            if (getField(position - 1).getDataType() != PVarbinaryEncoded.INSTANCE) {

Review Comment:
   Sure, let me do this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org