You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2022/01/02 22:52:00 UTC
[jira] [Updated] (HUDI-3081) [UMBRELLA] Revisiting Read Path Infra across Query Engines
[ https://issues.apache.org/jira/browse/HUDI-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu updated HUDI-3081:
-----------------------------
Issue Type: Epic (was: Bug)
> [UMBRELLA] Revisiting Read Path Infra across Query Engines
> ----------------------------------------------------------
>
> Key: HUDI-3081
> URL: https://issues.apache.org/jira/browse/HUDI-3081
> Project: Apache Hudi
> Issue Type: Epic
> Reporter: Alexey Kudinkin
> Assignee: Alexey Kudinkin
> Priority: Blocker
>
> Currently, our Read-path infrastructure is mostly disparate for each individual Query Engine having the same flow replicated multiple times:
> * Hive leverages hierarchy based off `InputFormat` class
> * Spark leverages hierarchy based off `SnapshotRelation`
> This leads to substantial duplication of virtually the same flows being replicated multiple times and unfortunately now diverging due to out of sync lifecycle (bug-fixes, etc).
> h3. Proposal
>
> *Phase 1: Abstracting Common Functionality*
>
> {_}T-shirt{_}: 1-1.5 weeks
> {_}Goal{_}: Abstract following common items to avoid duplication of the complex sequences across Engines
> * Unify Hive’s RecordReaders (`RealtimeCompactedRecordReader`, {{{}RealtimeUnmergedRecordReader{}}})
> ** _These Readers should only differ in the way they handle the payload, everything else should remain constant_
> * Abstract w/in common component (name TBD)
> ** Listing current file-slices at the requested instant (handling the timeline)
> ** Creating Record Iterator for the provided file-slice
>
> *Phase 2: Revisiting Record Handling*
>
> {_}T-shirt{_}: 1-1.5 weeks
> {_}Goal{_}: Avoid tight coupling with particular record representation on the Read Path (currently Avro) and enable
> * Common record handling API for combining records (Merge API)
> * Avoiding unnecessary serde by abstracting away standardized Record access routines (getting key, merging, etc)
> ** Behind the interface we'd rely on engine-specific representation to carry the payload (`InternalRow` for Spark, `ArrayWritable` for Hive, etc)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)