[jira] [Created] (HUDI-4382) Add logger for HoodieCopyOnWriteTableInputFormat

["Wenning Ding (Jira)" <ji...@apache.org>]

Wenning Ding created HUDI-4382:
----------------------------------

             Summary: Add logger for HoodieCopyOnWriteTableInputFormat
                 Key: HUDI-4382
                 URL: https://issues.apache.org/jira/browse/HUDI-4382
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Wenning Ding


When querying the ro and rt bootstrap mor tables using presto I observed both are failed with the following excecption:

{{java.lang.NoSuchFieldError: LOG
    at org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat.makeExternalFileSplit(HoodieCopyOnWriteTableInputFormat.java:199)
    at org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat.makeSplit(HoodieCopyOnWriteTableInputFormat.java:100)
    at org.apache.hudi.hadoop.realtime.HoodieMergeOnReadTableInputFormat.doMakeSplitForRealtimePath(HoodieMergeOnReadTableInputFormat.java:266)
    at org.apache.hudi.hadoop.realtime.HoodieMergeOnReadTableInputFormat.makeSplit(HoodieMergeOnReadTableInputFormat.java:211)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:345)
    at org.apache.hudi.hadoop.realtime.HoodieMergeOnReadTableInputFormat.getSplits(HoodieMergeOnReadTableInputFormat.java:79)
    at org.apache.hudi.hadoop.HoodieParquetInputFormatBase.getSplits(HoodieParquetInputFormatBase.java:68)
    at com.facebook.presto.hive.StoragePartitionLoader.loadPartition(StoragePartitionLoader.java:278)
    at com.facebook.presto.hive.DelegatingPartitionLoader.loadPartition(DelegatingPartitionLoader.java:81)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:224)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$700(BackgroundHiveSplitLoader.java:50)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:153)
    at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
    at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
    at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
    at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)}}

 

The reason we saw {{java.lang.NoSuchFieldError: LOG}} during the presto query is because in this {{HoodieCopyOnWriteTableInputFormat}} class, it inherits field {{LOG}} from its parent class {{FileInputFormat}} which is a class from Hadoop.

So in the compile time, it would reference this field from {{{}FileInputFormat.class{}}}. However, in the runtime, the presto doesn't have all the Hadoop classes in its classpath, what Presto uses is its own Hadoop dependency e.g. {{{}hadoop-apache2:jar{}}}. I checked that {{hadoop-apache2}} does not have class {{FileInputFormat}} shaded which causes this runtime error.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)