You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Alexey Kudinkin (Jira)" <ji...@apache.org> on 2022/04/16 20:44:00 UTC

[jira] [Created] (HUDI-3896) Support Spark optimizations for `HadoopFsRelation`

Alexey Kudinkin created HUDI-3896:
-------------------------------------

             Summary: Support Spark optimizations for `HadoopFsRelation`
                 Key: HUDI-3896
                 URL: https://issues.apache.org/jira/browse/HUDI-3896
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Alexey Kudinkin
            Assignee: Alexey Kudinkin
             Fix For: 0.12.0


After migrating to Hudi's own Relation impls, we unfortunately broke off some of the optimizations that Spark apply exclusively for `HadoopFsRelation`.

 

While these optimizations could be perfectly implemented for any `FileRelation`, Spark is unfortunately predicating them on usage of HadoopFsRelation, therefore making them non-applicable to any of the Hudi's relations.

Proper longterm solutions would be fixing this in Spark and could be either of:
 # Generalizing such optimizations to any `FileRelation`
 # Making `HadoopFsRelation` extensible (making it non-case class)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)