[jira] [Commented] (HUDI-920) Incremental view on MOR table using Spark Datasource

["Yanjia Gary Li (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173594#comment-17173594 ] 

Yanjia Gary Li commented on HUDI-920:
-------------------------------------

The most challenging thing of the incremental query for MOR was how to handle the filter based on commit time.

The current implementation simply use the Spark dataframe API [https://github.com/apache/hudi/blob/master/hudi-spark/src/main/scala/org/apache/hudi/IncrementalRelation.scala#L122]

But for RDD, we couldn't use this approach anymore.

Based on my investigation, the approach I can come up with:
 * Use the PrunedFilteredScan and push the extra filter down. For this approach, we will have to

 ** sqlContext.sparkSession.sessionState.conf.setConfString("spark.sql.parquet.recordLevelFilter.enabled", "true")
sqlContext.sparkSession.sessionState.conf.setConfString("spark.sql.parquet.enableVectorizedReader", "false")
 ** Filter Avro records ourselves on the record reading level.
 ** Trick Spark to scan the whole file when the user input won't trigger the file scan like *df.count()*. If using the default ParquetReaderFunction, it will just read the metadata for the count and won't apply the filter.
 * Apply filter before the buildScan() return.
 ** We will have to ser/deser the UnsafeRow to get the _hoodie_commit_time._
 * Go to Spark Planing to see if it's possible to force the filter somewhere. 
 * Explore Data source V2 to see if there is better support.

All we need is to force a filter before return the result to the user, but I don't see this was support in data source V1. At this point, the first approach is the most reasonable one for me and I can make a PR soon.

[~vinoth] [~bhasudha] [~uditme] WDYT?

> Incremental view on MOR table using Spark Datasource
> ----------------------------------------------------
>
>                 Key: HUDI-920
>                 URL: https://issues.apache.org/jira/browse/HUDI-920
>             Project: Apache Hudi
>          Issue Type: New Feature
>          Components: Spark Integration
>            Reporter: Yanjia Gary Li
>            Assignee: Yanjia Gary Li
>            Priority: Blocker
>             Fix For: 0.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)