Re: [I] Support Partition stats, column stats, bloom filters creation using SQL [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #17352:
URL: https://github.com/apache/hudi/issues/17352#issuecomment-5351237219

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8812).
   
   **Findings: confirmed, still not supported on `master`.**
   
   In `hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala`, `CreateIndexCommand.run` handles exactly four cases, and the three index types in this ticket are not among the supported ones:
   
   ```scala
   if (indexType.equals(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS)
     || indexType.equals(HoodieTableMetadataUtil.PARTITION_NAME_BLOOM_FILTERS)) {
     if (indexType.equals(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS) &&
       options.asJava.getOrDefault(EXPRESSION_OPTION, ExpressionIndexSparkFunctions.IDENTITY_FUNCTION).equals(ExpressionIndexSparkFunctions.IDENTITY_FUNCTION)) {
       throw new HoodieIndexException("Column stats index without expression on any column can be created using datasource configs. " +
         "Please refer https://hudi.apache.org/docs/metadata for more info")
     }
     ...
   } else if (indexName.equals(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)) { ...
   } else if (StringUtils.isNullOrEmpty(indexType)) { ...
   } else {
     throw new HoodieIndexException(String.format("%s is not supported", indexType))
   }
   ```
   
   Concretely:
   
   - `column_stats` without an expression - explicitly rejected with a message pointing users back at datasource configs.
   - `bloom_filters` - only reachable via the same expression-index branch, not as a plain index.
   - `partition_stats` - not handled at all; falls through to the final `else` and fails with "`partition_stats` is not supported".
   
   Only `record_index` and `secondary_index` have real SQL paths today.
   
   Worth noting the workaround your description mentions is itself blocked by the same code: creating an expression index with the identity function over `column_stats` is what that first `throw` rejects. That is tracked as #17353 (HUDI-8813), and the two are pinned by the same `if` condition - fixing them together is probably one change rather than two.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org