You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/03/04 07:50:00 UTC

[GitHub] [hive] veghlaci05 opened a new pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

veghlaci05 opened a new pull request #3079:
URL: https://github.com/apache/hive/pull/3079


   What changes were proposed in this pull request?
   
   This PR fixes the issue, when compaction fails on tables with boolean partition column.
   
   @zabetak , @kgyrtkirk This PR affects some classes which are really not my table, and I'm unsure if my changes are fine. Could you please also review it?
   Why are the changes needed?
   
   Compaction must work properly on tables with boolean partition column as well.
   Does this PR introduce any user-facing change?
   
   No
   How was this patch tested?
   
   Using automated tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pvary commented on a change in pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3079:
URL: https://github.com/apache/hive/pull/3079#discussion_r819619418



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
##########
@@ -498,7 +498,7 @@ private static PrunedPartitionList getPartitionsFromServer(Table tab, final Stri
    * @return true iff the partition pruning expression contains non-partition columns.
    */
   static private boolean pruneBySequentialScan(Table tab, List<Partition> partitions,
-      ExprNodeGenericFuncDesc prunerExpr, HiveConf conf) throws HiveException, MetaException {
+                                               ExprNodeDesc prunerExpr, HiveConf conf) throws HiveException, MetaException {

Review comment:
       nit: line continuation is 4 spaces




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] klcopp merged pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

Posted by GitBox <gi...@apache.org>.
klcopp merged pull request #3079:
URL: https://github.com/apache/hive/pull/3079


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pvary commented on a change in pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3079:
URL: https://github.com/apache/hive/pull/3079#discussion_r819622269



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilder.java
##########
@@ -347,8 +347,16 @@ private void buildWhereClauseForInsert(StringBuilder query) {
 
       query.append(" where ");
       for (int i = 0; i < keys.size(); ++i) {
-        query.append(i == 0 ? "`" : " and `").append(keys.get(i).getName()).append("`='")
-            .append(vals.get(i)).append("'");
+        FieldSchema keySchema = keys.get(i);
+        boolean isBooleanKey = keySchema.getType().equalsIgnoreCase("boolean");
+        query.append(i == 0 ? "`" : " and `").append(keySchema.getName()).append("`=");
+        if (!isBooleanKey) {

Review comment:
       nit: Maybe easier to read:
   ```
   if (isBooleanKey) {
     query.append("'").append(vals.get(i)).append("'");
   } else {
     query.append(vals.get(i));
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pvary commented on a change in pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3079:
URL: https://github.com/apache/hive/pull/3079#discussion_r819620506



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilder.java
##########
@@ -347,8 +347,16 @@ private void buildWhereClauseForInsert(StringBuilder query) {
 
       query.append(" where ");
       for (int i = 0; i < keys.size(); ++i) {
-        query.append(i == 0 ? "`" : " and `").append(keys.get(i).getName()).append("`='")
-            .append(vals.get(i)).append("'");
+        FieldSchema keySchema = keys.get(i);
+        boolean isBooleanKey = keySchema.getType().equalsIgnoreCase("boolean");

Review comment:
       ColumnType.BOOLEAN_TYPE_NAME




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pvary commented on a change in pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3079:
URL: https://github.com/apache/hive/pull/3079#discussion_r819618344



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java
##########
@@ -810,31 +811,44 @@ private static void serializeObjectByKryo(Kryo kryo, Object plan, OutputStream o
   }
 
   /**
-   * Serializes expression via Kryo.
-   * @param expr Expression.
+   * Serializes any object via Kryo. Type information will be serialized as well, allowing dynamic deserialization
+   * without the need to pass the class.
+   * @param object The object to serialize.
    * @return Bytes.
    */
-  public static byte[] serializeExpressionToKryo(ExprNodeGenericFuncDesc expr) {
-    return serializeObjectToKryo(expr);
+  public static byte[] serializeObjectWithTypeInformation(Serializable object) {
+    ByteArrayOutputStream baos = new ByteArrayOutputStream();
+    Kryo kryo = borrowKryo();
+    try (Output output = new Output(baos)) {
+      kryo.writeClassAndObject(output, object);
+    } finally {
+      releaseKryo(kryo);
+    }
+    return baos.toByteArray();
   }
 
   /**
    * Deserializes expression from Kryo.
    * @param bytes Bytes containing the expression.
    * @return Expression; null if deserialization succeeded, but the result type is incorrect.
    */
-  public static ExprNodeGenericFuncDesc deserializeExpressionFromKryo(byte[] bytes) {
-    return deserializeObjectFromKryo(bytes, ExprNodeGenericFuncDesc.class);
+  public static <T> T deserializeObjectWithTypeInformation(byte[] bytes) {
+    Kryo kryo = borrowKryo();
+    try (Input inp = new Input(new ByteArrayInputStream(bytes))) {
+      return (T) kryo.readClassAndObject(inp);
+    } finally {
+      releaseKryo(kryo);
+    }
   }
 
   public static String serializeExpression(ExprNodeGenericFuncDesc expr) {
-    return new String(Base64.encodeBase64(serializeExpressionToKryo(expr)),
-        StandardCharsets.UTF_8);
+    return new String(Base64.encodeBase64(serializeObjectToKryo(expr)),
+            StandardCharsets.UTF_8);

Review comment:
       nit: keep 4 spaces 😄 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pvary commented on a change in pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3079:
URL: https://github.com/apache/hive/pull/3079#discussion_r819618843



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java
##########
@@ -865,13 +879,13 @@ public static ExprNodeGenericFuncDesc deserializeExpression(String s) {
 
   public static String serializeObject(Serializable expr) {
     return new String(Base64.encodeBase64(serializeObjectToKryo(expr)),
-        StandardCharsets.UTF_8);
+            StandardCharsets.UTF_8);

Review comment:
       nit: remove formatting only changes. They kill you during backports




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pvary commented on a change in pull request #3079: HIVE-25645: Query-based compaction doesn't work when partition column type is boolean

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3079:
URL: https://github.com/apache/hive/pull/3079#discussion_r819622903



##########
File path: ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java
##########
@@ -183,12 +183,12 @@ public void checkExpr(int numParts,
       String dbName, String tblName, ExprNodeGenericFuncDesc expr, Table t) throws Exception {
     List<Partition> parts = new ArrayList<Partition>();
     client.listPartitionsByExpr(dbName, tblName,
-        SerializationUtilities.serializeExpressionToKryo(expr), null, (short)-1, parts);
+        SerializationUtilities.serializeObjectWithTypeInformation(expr), null, (short)-1, parts);
     assertEquals("Partition check failed: " + expr.getExprString(), numParts, parts.size());
 
     // check with partition spec as well
     PartitionsByExprRequest req = new PartitionsByExprRequest(dbName, tblName,
-            ByteBuffer.wrap(SerializationUtilities.serializeExpressionToKryo(expr)));
+            ByteBuffer.wrap(SerializationUtilities.serializeObjectWithTypeInformation(expr)));

Review comment:
       nit: funny that the indentation is different than the above one.
   If we touch the line fix the indentation please




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org