[jira] [Created] (HUDI-2443) KVComparator in HFile for metadata table is tied to HBase version

["Ryan Pifer (Jira)" <ji...@apache.org>]

Ryan Pifer created HUDI-2443:
--------------------------------

             Summary: KVComparator in HFile for metadata table is tied to HBase version
                 Key: HUDI-2443
                 URL: https://issues.apache.org/jira/browse/HUDI-2443
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: Ryan Pifer


There is no guarantee of compatibility between HFile reading/writing from different versions of ** HBase. For instance, HBase version 2.2.6 modified the comparator class stored in HFile metadata to {{org.apache.hadoop.hbase.KeyValue.KVComparator}} and {{org.apache.hadoop.hbase.KeyValue$KeyComparator}} no longer exists which may create issues when reading HFiles created by earlier versions of Hudi. And this comparator class is already marked as deprecated for {{CellComparatorImpl.}}

{{This also creates issues when shading HBase dependencies under Hudi namespace}}

{{Can we refactor HoodieKVComparator which is used in bootstrap index and have it applied for all use of HFile in Hudi. }}

{{[https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBootstrapIndex.java#L580-L586]}}

 

There will need to be some logic as well for backward compatibility when reading metadata which was written from earlier version of Hudi as well. We can maybe enforce that user must perform an upgrade step beforehand which re-writes metadata base file

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)