You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "YannByron (via GitHub)" <gi...@apache.org> on 2023/12/06 05:15:30 UTC

[I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

YannByron opened a new issue, #10252:
URL: https://github.com/apache/hudi/issues/10252

   Currently, The integration between spark and gluten/velox has made a good performance on parquet or lake format. So I think Hudi should take part in.
   
   Here is a design I proposed in gluten before and some discussion: https://github.com/oap-project/gluten/issues/3378
   
   Now, all the `scan` types that gluten has supported are file based, like `BatchScan` or `FileSourceScanExec`. Datasource provides the list of files during planning, then gluten pass them to the native library and the native reader (parquet/orc/...) loads them.
   
   For hudi cow table without `hoodie.schema.on.read.enable`,  it can return `HadoopFSRelation` (that's file based) when call `createRelation`. So maybe we can make this integration easily if the native reader can load the hudi files correctly.
   
   But for other hudi tables, they return `HoodieBaseRelation` (with `BaseRelation`, `FileRelation`, `PrunedFilteredScan`) that will be transformed to `RowDataSourceScanExec` that's not supported in gluten. To solve this, maybe there are two ways:
   1. to make gluten support it. IMO, it's not easy, and not a high-priority thing in gluten.
   2. to make hudi be file-based scan. But mor table needs to merge data, and hudi use Spark DatasourceV1 interface that doesn't have the ability to merge data, I guess we have to migrate to DSV2 to use `BatchScan` which can use hudi-defined reader to load data. As well as, a native C++ Hudi Reader is required in velox. With these two, hudi mor tables can be queried in native env.
   
   Gluten: https://github.com/oap-project/gluten
   Velox: https://github.com/facebookincubator/velox
   
   @vinothchandar @xushiyan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1850759410

   >Honestly, item 1 (a native reader in velox for mor table) is beyond my ability.
   @linliu-code or @rmahindra123 can help here. being the c++ nerds here. 
   
   >I can implement a gluten-hudi module in gluten and verify the Cow snapshot queries can work with them.
   we are seeing great results with the new read path that the team is implementing. So sth like this will help us make some decisions. I am also working on NVIDIA rapids along similar lines, the snapshot queries are already accelerated there


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1846456893

   > After we support HadoopFsRelation for all queries types, what else has been left for Gluten/Velox integration?
   
   A native reader is required for cases where the existing parquet reader cannot directly load the data correctly, like iceberg v2(positional deletes) reader https://github.com/facebookincubator/velox/pull/5897.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1849234421

   @vinothchandar, i'm also glad to work with you guys.
   
   Honestly, item 1 (a native reader in velox for mor table) is beyond my ability. 
   I can implement a gluten-hudi module in gluten and verify the Cow snapshot queries can work with them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1845878187

   Hey @YannByron great that you brought this up.
   
   @jonvex @linliu-code and I are actively working on improving Spark read and write performance and one aspect is to return `HadoopFSRelation` for all query types (including MOR snapshot queries with log merging).  As of now, on the latest master, for snapshot, RO, and CDC queries on both COW and MOR tables in Spark the `DefaultSource` return `HadoopFSRelation` already.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1845880242

   We need to check if our read and merging logic is compatible with Velox cc @jonvex @linliu-code 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1968878263

   @YannByron Pinging on this again. Is there a WIP integration for CoW that we could build as a quick prototype? how hard is that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1846269966

   After we support HadoopFsRelation for all queries types, what else has been left for Gluten/Velox integration? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1847820275

   @YannByron Expected. 
   
   To confirm, CoW snapshot queries should work, after we support HadoopFsRelation for all queries right. We will be happy to work with you on 1 & 2 items, if you have time/interest. let @linliu-code & team know


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1845856076

   @YannByron Great to hear from you.  @rmahindra123 is actively exploring this as well. 
   
    but a lot of work is going on to build a new vectorized read path for all queries. cc @jonvex @yihua @linliu-code . Can you check out some of their recent work? 
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-1845872621

   We are actually switching to use HadoopFSRelation for all query types. So it sounds like this will make the integration easier


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #10252:
URL: https://github.com/apache/hudi/issues/10252#issuecomment-4071182250

   Closing the loop here. https://github.com/apache/gluten/blob/main/docs/get-started/Velox.md#hudi-support COW reading is there. We need to plan work on hudi-rs to enable MoR reads and writes. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org