[jira] [Created] (SPARK-22790) add a configurable factor to describe HadoopFsRelation's size

["Nan Zhu (JIRA)" <ji...@apache.org>]

Nan Zhu created SPARK-22790:
-------------------------------

             Summary: add a configurable factor to describe HadoopFsRelation's size
                 Key: SPARK-22790
                 URL: https://issues.apache.org/jira/browse/SPARK-22790
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Nan Zhu


as per discussion in https://github.com/apache/spark/pull/19864#discussion_r156847927

the current HadoopFsRelation is purely based on the underlying file size which is not accurate and makes the execution vulnerable to errors like OOM

Users can enable CBO with the functionalities in https://github.com/apache/spark/pull/19864 to avoid this issue

This JIRA proposes to add a configurable factor to sizeInBytes method in HadoopFsRelation class so that users can mitigate this problem without CBO



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org