Re: [I] Hive "Count" queries don't work with bootstrap tables w/Hive3 [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #15676:
URL: https://github.com/apache/hudi/issues/15676#issuecomment-5178522324

   I traced this on current master. The root cause is readable from the code and matches the reported paths exactly, so recording it here — including a likely workaround you can try today.
   
   ### Root cause
   
   `BootstrapBaseFileSplit` carries **two** paths, and the constructor makes the distinction easy to miss:
   
   ```java
   public BootstrapBaseFileSplit(FileSplit baseSplit, FileSplit bootstrapFileSplit) {
     super(baseSplit.getPath(), baseSplit.getStart(), baseSplit.getLength(), baseSplit.getLocations());
     this.bootstrapFileSplit = bootstrapFileSplit;
   }
   ```
   
   So **the split itself is the Hudi skeleton file**, inside the table root, while **`getBootstrapFileSplit()` is the external source file**, outside it.
   
   `HoodieParquetInputFormat.createBootstrappingRecordReader` then picks between them by projection:
   
   ```java
   if (hoodieColsProjected.isEmpty()) {
     return getRecordReaderInternal(eSplit.getBootstrapFileSplit(), job, reporter);  // external, OUTSIDE table root
   } else if (externalColsProjected.isEmpty()) {
     return getRecordReaderInternal(split, job, reporter);                          // skeleton, inside table root
   } else {
     // stitch both
   }
   ```
   
   For `SELECT COUNT(*)` no columns are projected at all, so **both** lists are empty and the first branch wins. Hudi therefore hands Hive a split whose path lies outside the table.
   
   On Hive 3 that path is then vectorized:
   
   ```
   VectorizedParquetRecordReader.<init> -> initPartitionValues
     -> VectorizedRowBatchCtx.getPartitionValues
       -> HiveFileFormatUtils.getFromPathRecursively
   ```
   
   `getFromPathRecursively` derives the partition values by looking the split path up in `pathToPartitionInfo`, which only contains the **Hudi** table's partition directories. The external path is not in it, which is precisely your error:
   
   ```
   cannot find dir = [s3://my-bucket/test-data/hudi/parquet-source-tables/hive_style_partitioned_tb/event_type=two/part-00000-...parquet]
    in pathToPartitionInfo: [[s3://my-bucket/hudi-table/test_bootstrap_hive_partitionedrt/event_type=one],
                             [s3://my-bucket/hudi-table/test_bootstrap_hive_partitionedrt/event_type=two]]
   ```
   
   That also explains "the same query works with Hive2": Hive 2 did not take the vectorized parquet path, so `initPartitionValues` was never called and the out-of-table path was never looked up.
   
   ### Likely workaround
   
   ```sql
   set hive.vectorized.execution.enabled=false;
   SELECT COUNT(*) FROM HUDI_BOOTSTRAP_TABLE;
   ```
   
   That avoids `VectorizedParquetRecordReader` entirely, so the partition-value lookup that fails never runs. I have not been able to confirm it on Hive 3 + Tez myself, so please treat it as a strong hypothesis rather than a verified fix — but it follows directly from the stack trace, and it costs nothing to try.
   
   ### This is not actually specific to COUNT
   
   Worth correcting the title's implication: the branch condition is "no Hudi meta columns projected", not "count". So a plain projection of data columns — `SELECT <some_data_col> FROM HUDI_BOOTSTRAP_TABLE` — should take the same branch and fail the same way on Hive 3. If you have a moment, confirming that would be useful, because it changes the shape of the fix.
   
   ### On fixing it
   
   Only the `COUNT(*)` case has a cheap fix. When **both** projection lists are empty nothing needs the external file, and bootstrap guarantees a 1:1 row correspondence between skeleton and external file — so reading the skeleton would give the same count from a path that *is* inside the table root, and Hive's lookup would succeed. That is a precedence change in the `if`/`else if` above.
   
   Queries that genuinely project data columns cannot be fixed that way, since the data only exists in the external file. Those need either vectorization disabled for bootstrap splits, or the partition values supplied rather than derived from the path.
   
   Two further notes for whoever picks this up:
   
   - This legacy path is **still live on 1.x**. `HoodieInputFormatUtils.shouldUseFilegroupReader` ends with `&& !(split instanceof BootstrapBaseFileSplit)`, so bootstrap tables never reach the new file-group reader and always land in `createBootstrappingRecordReader`.
   - It has **no test coverage at all**: nothing in the test tree references `BootstrapBaseFileSplit`, `createBootstrappingRecordReader` or `BootstrapColumnStichingRecordReader`. The branch selection above is unit-testable without Hive, so that would be the place to start before changing the precedence.
   
   I have not raised a PR, because the end-to-end behaviour needs a Hive 3 + Tez bootstrap table to verify and I do not have one. Happy to do the precedence change plus first unit coverage of the branch selection if a maintainer thinks that is the right direction.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org