You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/11/10 04:32:40 UTC

[GitHub] [hive] amansinha100 commented on a diff in pull request #3742: HIVE-21599: Remove predicate on partition columns from Table Scan operator

amansinha100 commented on code in PR #3742:
URL: https://github.com/apache/hive/pull/3742#discussion_r1018639310


##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java:
##########
@@ -209,6 +214,22 @@ public FilterCompat.Filter setFilter(final JobConf conf, MessageType schema) {
     }
   }
 
+  private MessageType getSchemaWithoutPartitionColumns(JobConf conf, MessageType schema) {
+    String partCols = conf.get(IOConstants.PARTITION_COLUMNS);
+    if (partCols != null && partCols.length() > 0) {
+      Set<String> partitionColumns = new HashSet<>(Arrays.asList(partCols.split(",")));
+      List<Type> newFields = new ArrayList<>();
+
+      for (Type field: schema.getFields()) {
+        if(!partitionColumns.contains(field.getName())) {

Review Comment:
   Could you confirm if this comparison works in a case-insensitive way ? i.e if the partition columns is defined as 'part_col' and the query has predicate on 'PART_COL'. 



##########
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##########
@@ -4272,6 +4273,21 @@ public static void addTableSchemaToConf(Configuration conf,
       LOG.info("schema.evolution.columns and schema.evolution.columns.types not available");
     }
   }
+  public static void setPartitionColumnsToConf(Configuration conf, TableScanOperator tableScanOp) {
+    TableScanDesc scanDesc = tableScanOp.getConf();
+    if (scanDesc != null && scanDesc.getTableMetadata() != null) {
+      List<String> partitionColsList = scanDesc.getTableMetadata().getPartColNames();
+      if (!partitionColsList.isEmpty()) {
+        conf.set(IOConstants.PARTITION_COLUMNS, String.join(",", partitionColsList));
+      }
+    } else {
+      LOG.info(IOConstants.PARTITION_COLUMNS + " not available");

Review Comment:
   Is this else condition placed correctly ?  Seems like the INFO message  for partition columns not available  should be logged when partitionColsList.isEmpty() returns true. 



##########
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##########
@@ -4272,6 +4273,21 @@ public static void addTableSchemaToConf(Configuration conf,
       LOG.info("schema.evolution.columns and schema.evolution.columns.types not available");
     }
   }
+  public static void setPartitionColumnsToConf(Configuration conf, TableScanOperator tableScanOp) {

Review Comment:
   nit: pls add a comment for this method.  Also, the unset method is named 'InConf' whereas this one has 'ToConf'.  Can both set and unset method names use the same suffix ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org