[GitHub] [hudi] aditiwari01 opened a new issue #2801: Issues with Hive querying on MOR tables with no partitions

[GitBox <gi...@apache.org>]

aditiwari01 opened a new issue #2801:
URL: https://github.com/apache/hudi/issues/2801


   Unable to read data via Hive from both _ro & _rt tables if my data is not partitioned.
   Reading from spark api works fine.
   
   Related Write Confs used:
   
   ```
   PARTITIONPATH_FIELD_OPT_KEY->"",
   "hoodie.datasource.hive_sync.enable" -> "true"
   "hoodie.datasource.hive_sync.partition_fields"->"",
   HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY->classOf[NonPartitionedExtractor].getCanonicalName
   ```
   
   Issue faced: NullPointerException in `getTableMetaClientForBasePath` of class `HoodieInputFormatUtils`.
   
   My take:
   
   ```
     public static HoodieTableMetaClient getTableMetaClientForBasePath(FileSystem fs, Path dataPath) throws IOException {
       LOG.info("Getting Table Meta Client from path: " + dataPath.toString());
       int levels = HoodieHiveUtils.DEFAULT_LEVELS_TO_BASEPATH;
       if (HoodiePartitionMetadata.hasPartitionMetadata(fs, dataPath)) {
         HoodiePartitionMetadata metadata = new HoodiePartitionMetadata(fs, dataPath);
         metadata.readFromFS();
         levels = metadata.getPartitionDepth();
       }
       Path baseDir = HoodieHiveUtils.getNthParent(dataPath, levels);
       LOG.info("Reading hoodie metadata from path " + baseDir.toString());
       return HoodieTableMetaClient.builder().setConf(fs.getConf()).setBasePath(baseDir.toString()).build();
     }
   ```
   
   Herein if partition meta is not available (as in case of no partition), we set levels to default of 3, in which case the base path fetched is wrong.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] aditiwari01 commented on issue #2801: Issues with Hive querying on MOR tables with no partitions

[GitBox <gi...@apache.org>]

aditiwari01 commented on issue #2801:
URL: https://github.com/apache/hudi/issues/2801#issuecomment-817120596


   Update:
   
   After setting default to 0 in case of no partition, I am able to avoid above error and able to get Table meta correctly. However, now select * query on hive return empty dataset. 
   From logs I can see that `.HoodieInputFormatUtils: Total paths to process after hoodie filter 0`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] aditiwari01 commented on issue #2801: Issues with Hive querying on MOR tables with no partitions

[GitBox <gi...@apache.org>]

aditiwari01 commented on issue #2801:
URL: https://github.com/apache/hudi/issues/2801#issuecomment-817139358


   Please ignore the above comment. The empty result from hive was due to me missinng one of the configs.
   
   Everything is working as expected after changing the default value of `HoodieHiveUtils.DEFAULT_LEVELS_TO_BASEPATH` to 0 (currennt value 3), as explained in first comment.
   
   I think we should permanently change this to 0. 3 looks like a random number. Also I have confirmed that this default is not used anywhere other than the code snippet I've mentioned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] aditiwari01 closed issue #2801: Issues with Hive querying on MOR tables with no partitions

[GitBox <gi...@apache.org>]

aditiwari01 closed issue #2801:
URL: https://github.com/apache/hudi/issues/2801


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] aditiwari01 commented on issue #2801: Issues with Hive querying on MOR tables with no partitions

[GitBox <gi...@apache.org>]

aditiwari01 commented on issue #2801:
URL: https://github.com/apache/hudi/issues/2801#issuecomment-817537973


   Issue resolved after using correct key generator class. Closing the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org