You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Yanjia Gary Li (Jira)" <ji...@apache.org> on 2020/04/14 22:12:00 UTC
[jira] [Comment Edited] (HUDI-69) Support realtime view in Spark
datasource #136
[ https://issues.apache.org/jira/browse/HUDI-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082773#comment-17082773 ]
Yanjia Gary Li edited comment on HUDI-69 at 4/14/20, 10:11 PM:
---------------------------------------------------------------
After a closer look, I think Spark datasource support for realtime table needs:
* Support hadoop.mapreduce.xxx apis. We use hadoop.mapred.RecordReader, but Spark sql use hadoop.mapreduce.RecordReader. We need to figure how to support both apis, or upgrade to mapreduce.
* Implement the extension of ParquetInputFormat from Spark or a custom data source reader to handle merge. [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala]
* Use Datasource V2 to be the default data source.
Please let me know what you guys think.
was (Author: garyli1019):
After a closer look, I think Spark datasource support for realtime table needs:
* Refactoring HoodieRealtimeFormat and (file split, record reader). Decouple Hudi logic from the MapredParquetInputFormat. I think we can maintain the Hudi file split and path filtering in a central place, and able to be adopted by different query engines. With bootstrap support, the file format maintenance could be more complicated. I think this is very essential.
* Implement the extension of ParquetInputFormat from Spark or a custom data source reader to handle merge. [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala]
* Use Datasource V2 to be the default data source.
Please let me know what you guys think.
> Support realtime view in Spark datasource #136
> ----------------------------------------------
>
> Key: HUDI-69
> URL: https://issues.apache.org/jira/browse/HUDI-69
> Project: Apache Hudi (incubating)
> Issue Type: New Feature
> Components: Spark Integration
> Reporter: Vinoth Chandar
> Assignee: Yanjia Gary Li
> Priority: Major
> Fix For: 0.6.0
>
>
> https://github.com/uber/hudi/issues/136
--
This message was sent by Atlassian Jira
(v8.3.4#803005)