You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/10 22:49:02 UTC

[GitHub] [hudi] umehrot2 opened a new pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

umehrot2 opened a new pull request #1944:
URL: https://github.com/apache/hudi/pull/1944


   ## What is the purpose of the pull request
   
   The purpose of this pull request is to implement changes required on Hudi side to get Bootstrapped tables integrated with Presto. The testing was done against **presto 0.232** and following changes were identified to make it work:
   
   - Annotation **UseRecordReaderFromInputFormat** is required on **HoodieParquetInputFormat** as well, because the reading for bootstrapped tables needs to happen through record reader to be able to perform the merge. On presto side, this annotation is already handled.
   
   - We need to internally maintain `VIRTUAL_COLUMN_NAMES` because presto's internal hive version **hive-apache-1.2.2** has `VirutalColumn` as a class, versus the one we depend on in hudi which is an **enum**. This results in following error in presto:
   ```
   2020-08-10T21:59:58.957Z	ERROR	remote-task-callback-2	com.facebook.presto.execution.StageExecutionStateMachine	Stage execution 20200810_215953_00006_34kqg.1.0 failed
   java.lang.NoSuchFieldError: VIRTUAL_COLUMN_NAMES
   	at org.apache.hudi.hadoop.HoodieParquetInputFormat.lambda$getRecordReader$2(HoodieParquetInputFormat.java:201)
   	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
   	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
   	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
   	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
   	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
   	at org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:203)
   	at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:253)
   	at com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$0(GenericHiveRecordCursorProvider.java:74)
   	at com.facebook.presto.hive.authentication.UserGroupInformationUtils.lambda$executeActionInDoAs$0(UserGroupInformationUtils.java:29)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at javax.security.auth.Subject.doAs(Subject.java:360)
   	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1824)
   	at com.facebook.presto.hive.authentication.UserGroupInformationUtils.executeActionInDoAs(UserGroupInformationUtils.java:27)
   	at com.facebook.presto.hive.authentication.ImpersonatingHdfsAuthentication.doAs(ImpersonatingHdfsAuthentication.java:39)
   	at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:82)
   	at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:73)
   	at com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:374)
   	at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:137)
   	at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:113)
   	at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:52)
   ```
   - Dependency changes in `hudi-presto-bundle` to avoid runtime exceptions.
   
   ## Brief change log
   
   ## Verify this pull request
   
   The changes have been tested on **emr-5.30.1** against **presto 0.232**.
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673102081


   > > I am kind of torn. On one hand, this completes the bootstrap story and lets us stabilize things. Given the later versions of presto anyway take hudi as a compile depedency, I think it may be ok?
   > > cc @bhasudha does prestoDB depend on the bundle or the hoodie-hadoop-mr jars?
   > 
   > @vinothchandar @umehrot2 Starting from version 0.233, Presto depends on hudi-hadoop-mr during compile time. Prior to that we needed to drop bundle jars into the plugin directory.
   > @umehrot2 If this fix is for 0.232 would this go away when picking up 0.233 ?
   
   Yeah this is based on my testing for presto 0.232. I did not know the `presto-bundle` would be deprecated going forward. But I am wondering if this was the right decision to depend on `hudi-hadoop-mr` in `presto`. How would we avoid version conflicts like the ones that came up with hbase here ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-672288068


   in some sense, due to the bundling changes, this feel very last minute to validate more. We have to rely on 1-2 rounds of RC testing to weed things out.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-671642813


   cc @vinothchandar @bvaradar @bhasudha 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468799403



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>
+                  <include>commons-lang:commons-lang</include>

Review comment:
       I think shading protobuf causes other issues. Also these dependencies were not part of presto runtime that is why I did not feel the need to shade them.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468797799



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -159,5 +172,24 @@
       <artifactId>avro</artifactId>
       <scope>compile</scope>
     </dependency>
+
+    <dependency>

Review comment:
       I remember its due a conflict between the version `hbase` needs vs what `presto` has. Most of the issues are coming in because of trying to package `hbase` and make it work through presto.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468693275



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -159,5 +172,24 @@
       <artifactId>avro</artifactId>
       <scope>compile</scope>
     </dependency>
+
+    <dependency>

Review comment:
       why are we bundling guava?  we can just use presto's guava classes right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468819625



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>

Review comment:
       Stack trace why we need guava shading:
   ```
   java.lang.NoSuchMethodError: com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
   	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.LruBlockCache.toString(LruBlockCache.java:704)
   	at java.lang.String.valueOf(String.java:2994)
   	at java.lang.StringBuilder.append(StringBuilder.java:131)
   	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.CacheConfig.toString(CacheConfig.java:502)
   	at java.lang.String.valueOf(String.java:2994)
   	at java.lang.StringBuilder.append(StringBuilder.java:131)
   	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:260)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.createReader(HFileBootstrapIndex.java:181)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.access$400(HFileBootstrapIndex.java:76)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.partitionIndexReader(HFileBootstrapIndex.java:272)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.fetchBootstrapIndexInfo(HFileBootstrapIndex.java:262)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.initIndexInfo(HFileBootstrapIndex.java:252)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.<init>(HFileBootstrapIndex.java:243)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.createReader(HFileBootstrapIndex.java:191)
   	at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$addFilesToView$2(AbstractTableFileSystemView.java:128)
   	at java.util.HashMap.forEach(HashMap.java:1289)
   	at org.apache.hudi.common.table.view.AbstractTableFileSystemView.addFilesToView(AbstractTableFileSystemView.java:125)
   	at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:131)
   	at org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.filterFileStatusForSnapshotMode(HoodieInputFormatUtils.java:376)
   	at org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:119)
   	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:288)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:370)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:263)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:95)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:192)
   	at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
   	at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
   	at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
   	at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   ```
   
   This happens because `hbase 1.2.3` needs `guava 12.0.1` whereas `presto` is on `guava 26.0`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673100458


   > I am kind of torn. On one hand, this completes the bootstrap story and lets us stabilize things. Given the later versions of presto anyway take hudi as a compile depedency, I think it may be ok?
   > 
   > cc @bhasudha does prestoDB depend on the bundle or the hoodie-hadoop-mr jars?
   
   Yes this is a good question. Should we really depend on `hud-hadoop-mr` in presto or the `presto-bundle` ? The bundle gives us the advantages of shading the dependencies to avoid such conflicts. Without bundle we would have to do the shading in `hudi-hadoop-mr` which does not seem right.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468783171



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java
##########
@@ -38,6 +40,9 @@
   public static final String HOODIE_CONSUME_MODE_PATTERN = "hoodie.%s.consume.mode";
   public static final String HOODIE_START_COMMIT_PATTERN = "hoodie.%s.consume.start.timestamp";
   public static final String HOODIE_MAX_COMMIT_PATTERN = "hoodie.%s.consume.max.commits";
+  public static final Set<String> VIRTUAL_COLUMN_NAMES = CollectionUtils.createImmutableSet(

Review comment:
       This is a concept of Hive metadata, and not specific to parquet.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bhasudha commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bhasudha commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673103896


   > @bhasudha @bschell the MOR support in presto as well, works using the compile time dependency?
   > 
   > seems, then to make presto generally work with bootstrap, we need to rethink that for higher presto versions at-least. How's athena working now, with the bundle or the compile-time dependency?
   
   Athena is using very old version of Presto I believe. And definitely using the bundle jar.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468782242



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -63,6 +62,7 @@
  * that does not correspond to a hoodie table then they are passed in as is (as what FileInputFormat.listStatus()
  * would do). The JobConf could have paths from multipe Hoodie/Non-Hoodie tables
  */
+@UseRecordReaderFromInputFormat

Review comment:
       Yes he main change required is this to integrate with presto. Rest are just fixes for the various runtime issues I ran into trying to get this working.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468866012



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>
+                  <include>commons-lang:commons-lang</include>

Review comment:
       I am able to shade as well as relocate `commons-lang` and `protobuf`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673019518


   > in some sense, due to the bundling changes, this feel very last minute to validate more. We have to rely on 1-2 rounds of RC testing to weed things out.
   
   I understand Vinoth. So are we planning to include this ? It will unblock the contribution on presto side for the bootstrapped table work.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673103082


   @umehrot2 I think we can control the shading by a flag for now. and move on for now? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bvaradar commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468702061



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>

Review comment:
       @umehrot2  Are these jars not provided by Presto runtime itself ? Version mismatch with guava had been a regular issue earlier but I see that you have shaded them. 
   
   Please also check the licensing of the new dependencies. Are they Apache ?
   

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java
##########
@@ -38,6 +40,9 @@
   public static final String HOODIE_CONSUME_MODE_PATTERN = "hoodie.%s.consume.mode";
   public static final String HOODIE_START_COMMIT_PATTERN = "hoodie.%s.consume.start.timestamp";
   public static final String HOODIE_MAX_COMMIT_PATTERN = "hoodie.%s.consume.max.commits";
+  public static final Set<String> VIRTUAL_COLUMN_NAMES = CollectionUtils.createImmutableSet(

Review comment:
       Is this specific to Parquet or Hive in general. If this is specific to Parquet, can you move it to ParquetUtils ?

##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>
+                  <include>commons-lang:commons-lang</include>

Review comment:
       Can you also shade commons-lang and protobuf ?

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -63,6 +62,7 @@
  * that does not correspond to a hoodie table then they are passed in as is (as what FileInputFormat.listStatus()
  * would do). The JobConf could have paths from multipe Hoodie/Non-Hoodie tables
  */
+@UseRecordReaderFromInputFormat

Review comment:
       Is this the main change ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673096742


   cc @bvaradar wdyt? if there is little impact to presto integration, we could just do it and iterate.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bhasudha commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bhasudha commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673115616


   > > > So if we could instead take compile time dependency on the presto-bundle, that could work for all situations here.
   > > 
   > > 
   > > let's raise a JIRA for this. the thing is sometimes its weird to depend on compile time on a bundle, because you cannot selectively exclude dependencies. but if we rework prestoDB to depend on presto-bundle and shade it, it would fix all issues.
   > > I ll let the aws folks chime in here. so we can make a call.
   > 
   > I agree that its weird to have compile time dependency on bundle jars, but that seems like the way forward here to me.
   
   https://issues.apache.org/jira/browse/HUDI-1183 I created a jira here to track this. @umehrot2 feel free to add more details to that. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468873425



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -159,5 +172,24 @@
       <artifactId>avro</artifactId>
       <scope>compile</scope>
     </dependency>
+
+    <dependency>

Review comment:
       ah. did not realize that we live in this world now. yeah makes sense. is hbase 2.x not happening? anyways, we need to shade guava. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673102805


   @bhasudha @bschell the MOR support in presto as well, works using the compile time dependency? 
   
   seems, then to make presto generally work with bootstrap, we need to rethink that for higher presto versions at-least. How's athena working now, with the bundle or the compile-time dependency? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468782242



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -63,6 +62,7 @@
  * that does not correspond to a hoodie table then they are passed in as is (as what FileInputFormat.listStatus()
  * would do). The JobConf could have paths from multipe Hoodie/Non-Hoodie tables
  */
+@UseRecordReaderFromInputFormat

Review comment:
       Yes this is the main change required to integrate with presto. Rest are just fixes for the various runtime issues I ran into trying to get this working.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar merged pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #1944:
URL: https://github.com/apache/hudi/pull/1944


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468799779



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>
+                  <include>commons-lang:commons-lang</include>

Review comment:
       Let me quickly try shading them again, and remember what issues I was running into.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673169459


   > A flag to control the inclusion of dependent packages makes sense as we will have a way to tell users what to do if we find issues post launch.
   
   Updated the PR by adding a flag that can control if we want to not shade these.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468818565



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -159,5 +172,24 @@
       <artifactId>avro</artifactId>
       <scope>compile</scope>
     </dependency>
+
+    <dependency>

Review comment:
       Here is the stack trace why we need to shade guava:
   ```
   java.lang.NoSuchMethodError: com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
   	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.LruBlockCache.toString(LruBlockCache.java:704)
   	at java.lang.String.valueOf(String.java:2994)
   	at java.lang.StringBuilder.append(StringBuilder.java:131)
   	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.CacheConfig.toString(CacheConfig.java:502)
   	at java.lang.String.valueOf(String.java:2994)
   	at java.lang.StringBuilder.append(StringBuilder.java:131)
   	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:260)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.createReader(HFileBootstrapIndex.java:181)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.access$400(HFileBootstrapIndex.java:76)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.partitionIndexReader(HFileBootstrapIndex.java:272)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.fetchBootstrapIndexInfo(HFileBootstrapIndex.java:262)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.initIndexInfo(HFileBootstrapIndex.java:252)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex$HFileBootstrapIndexReader.<init>(HFileBootstrapIndex.java:243)
   	at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.createReader(HFileBootstrapIndex.java:191)
   	at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$addFilesToView$2(AbstractTableFileSystemView.java:128)
   	at java.util.HashMap.forEach(HashMap.java:1289)
   	at org.apache.hudi.common.table.view.AbstractTableFileSystemView.addFilesToView(AbstractTableFileSystemView.java:125)
   	at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:131)
   	at org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.filterFileStatusForSnapshotMode(HoodieInputFormatUtils.java:376)
   	at org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:119)
   	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:288)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:370)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:263)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:95)
   	at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:192)
   	at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
   	at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
   	at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
   	at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   ```
   
   This happens because `hbase 1.2.3` needs `guava 12.0.1` whereas `presto` is on `guava 26.0`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673119515


   > @umehrot2 I think we can control the shading by a flag for now. and move on for now?
   
   Let me do this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bvaradar commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468723660



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>

Review comment:
       Unless, there is a strong reason, we should avoid bundling it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on a change in pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#discussion_r468797076



##########
File path: packaging/hudi-presto-bundle/pom.xml
##########
@@ -76,7 +76,12 @@
                   <include>org.apache.hbase:hbase-common</include>
                   <include>org.apache.hbase:hbase-protocol</include>
                   <include>org.apache.hbase:hbase-server</include>
+                  <include>org.apache.hbase:hbase-annotations</include>
                   <include>org.apache.htrace:htrace-core</include>
+                  <include>com.yammer.metrics:metrics-core</include>
+                  <include>com.google.guava:guava</include>

Review comment:
       These dependencies are fixes for runtime issues. `guava` is there is the class-path but there is a runtime mismatch. The main problem is coming in due to `hbase` dependencies, and they conflict with the `presto` versions. Let me try to provide the stack traces if that helps.
   
   About the licensing:
   - https://mvnrepository.com/artifact/com.yammer.metrics/metrics-core
   - https://mvnrepository.com/artifact/com.google.guava/guava
   - https://mvnrepository.com/artifact/commons-lang/commons-lang
   - https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java => this seems to be **BSD License**. So is this going to be an issue ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673112695


   >So if we could instead take compile time dependency on the presto-bundle, that could work for all situations here.
   
   let's raise a JIRA for this. the thing is sometimes its weird to depend on compile time on a bundle, because you cannot selectively exclude dependencies. but if we rework prestoDB to depend on presto-bundle and shade it, it would fix all issues. 
   
   I ll let the aws folks chime in here. so we can make a call. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bhasudha commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bhasudha commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673107178


   > @bhasudha @bschell the MOR support in presto as well, works using the compile time dependency?
   > 
   > seems, then to make presto generally work with bootstrap, we need to rethink that for higher presto versions at-least. How's athena working now, with the bundle or the compile-time dependency?
   
   
   
   > > I am kind of torn. On one hand, this completes the bootstrap story and lets us stabilize things. Given the later versions of presto anyway take hudi as a compile depedency, I think it may be ok?
   > > cc @bhasudha does prestoDB depend on the bundle or the hoodie-hadoop-mr jars?
   > 
   > Yes this is a good question. Should we really depend on `hud-hadoop-mr` in presto or the `presto-bundle` ? The bundle gives us the advantages of shading the dependencies to avoid such conflicts. Without bundle we would have to do the shading in `hudi-hadoop-mr` which does not seem right.   
   
   There is no strong reason for why we would need to depend on `hudi-hadoop-mr` from Presto.  I guess at that time, we were not yet thinking about how other use case would be supported in Presto and the dependencies that come along. We could certainly depend on the bundle instead. From my interactions, the presto community does not recommend dropping jars into plugin. So if we could instead take compile time dependency on the `presto-bundle`, that could work for all situations here. 
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] umehrot2 commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673114587


   > > So if we could instead take compile time dependency on the presto-bundle, that could work for all situations here.
   > 
   > let's raise a JIRA for this. the thing is sometimes its weird to depend on compile time on a bundle, because you cannot selectively exclude dependencies. but if we rework prestoDB to depend on presto-bundle and shade it, it would fix all issues.
   > 
   > I ll let the aws folks chime in here. so we can make a call.
   
   I agree that its weird to have compile time dependency on bundle jars, but that seems like the way forward here to me.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673096439


   I am kind of torn. On one hand, this completes the bootstrap story and lets us stabilize things. Given the later versions of presto anyway take hudi as a compile depedency, I think it may be ok?
   
   cc @bhasudha does prestoDB depend on the bundle or the hoodie-hadoop-mr jars? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bhasudha commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bhasudha commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673100564


   > I am kind of torn. On one hand, this completes the bootstrap story and lets us stabilize things. Given the later versions of presto anyway take hudi as a compile depedency, I think it may be ok?
   > 
   > cc @bhasudha does prestoDB depend on the bundle or the hoodie-hadoop-mr jars?
   
   @vinothchandar @umehrot2  Starting from version 0.233, Presto depends on hudi-hadoop-mr during compile time. Prior to that we needed to drop bundle jars into the plugin directory.  
   @umehrot2  If this fix is for 0.232 would this go away when picking up 0.233 ? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bhasudha commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bhasudha commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673103276


   > > I am kind of torn. On one hand, this completes the bootstrap story and lets us stabilize things. Given the later versions of presto anyway take hudi as a compile depedency, I think it may be ok?
   > > cc @bhasudha does prestoDB depend on the bundle or the hoodie-hadoop-mr jars?
   > 
   > Yes this is a good question. Should we really depend on `hud-hadoop-mr` in presto or the `presto-bundle` ? The bundle gives us the advantages of shading the dependencies to avoid such conflicts. Without bundle we would have to do the shading in `hudi-hadoop-mr` which does not seem right.
   
   I agree that doing shading in the `hudi-hadoop-mr` is not right. As far depending on bundle itself, we would need to test the shading in both s3 and HDFS environments to include all right shading if we went that route. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bvaradar commented on pull request #1944: [HUDI-1174] Changes for bootstrapped tables to work with presto

Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1944:
URL: https://github.com/apache/hudi/pull/1944#issuecomment-673120694


   > cc @bvaradar wdyt? if there is little impact to presto integration, we could just do it and iterate.
   
   @vinothchandar : Since this is essential for Presto support for bootstrapped tables and we are shading, I think we can get this in and iterate. A flag to control the inclusion of dependent  packages makes sense as we will have a way to tell users what to do if we find issues post launch.  
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org