You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Bhavani Sudha (Jira)" <ji...@apache.org> on 2020/08/14 18:17:00 UTC
[jira] [Resolved] (HUDI-1174) Hudi changes for bootstrapped tables
integration with Presto
[ https://issues.apache.org/jira/browse/HUDI-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bhavani Sudha resolved HUDI-1174.
---------------------------------
Resolution: Fixed
Resolving since the corresponding PR is merged. Please feel free to re-open if needed.
> Hudi changes for bootstrapped tables integration with Presto
> ------------------------------------------------------------
>
> Key: HUDI-1174
> URL: https://issues.apache.org/jira/browse/HUDI-1174
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: bootstrap
> Reporter: Udit Mehrotra
> Assignee: Udit Mehrotra
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> Hudi changes for bootstrapped tables integration with Presto.
> * Annotation *UseRecordReaderFromInputFormat* is required on *HoodieParquetInputFormat* as well, because the reading for bootstrapped tables needs to happen through record reader to be able to perform the merge. On presto side, this annotation is already handled.
> * We need to internally maintain *VIRTUAL_COLUMN_NAMES* because presto's internal hive version *hive-apache-1.2.2* has *VirutalColumn* as a *class*, versus the one we depend on in hudi which is an *enum*. This results in following error in presto:
>
> {noformat}
> 2020-08-10T21:59:58.957Z ERROR remote-task-callback-2 com.facebook.presto.execution.StageExecutionStateMachine Stage execution 20200810_215953_00006_34kqg.1.0 failed
> java.lang.NoSuchFieldError: VIRTUAL_COLUMN_NAMES
> at org.apache.hudi.hadoop.HoodieParquetInputFormat.lambda$getRecordReader$2(HoodieParquetInputFormat.java:201)
> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:203)
> at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:253)
> at com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$0(GenericHiveRecordCursorProvider.java:74)
> at com.facebook.presto.hive.authentication.UserGroupInformationUtils.lambda$executeActionInDoAs$0(UserGroupInformationUtils.java:29)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1824)
> at com.facebook.presto.hive.authentication.UserGroupInformationUtils.executeActionInDoAs(UserGroupInformationUtils.java:27)
> at com.facebook.presto.hive.authentication.ImpersonatingHdfsAuthentication.doAs(ImpersonatingHdfsAuthentication.java:39)
> at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:82)
> at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:73)
> at com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:374)
> at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:137)
> at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:113)
> at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:52)
> {noformat}
>
> * Dependency changes in *hudi-presto-bundle* to avoid runtime exceptions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)