You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/04/23 07:52:12 UTC

[jira] [Resolved] (SPARK-14551) Reduce number of NameNode calls in OrcRelation with FileSourceStrategy mode

     [ https://issues.apache.org/jira/browse/SPARK-14551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Reynold Xin resolved SPARK-14551.
---------------------------------
       Resolution: Fixed
         Assignee: Rajesh Balamohan
    Fix Version/s: 2.0.0

> Reduce number of NameNode calls in OrcRelation with FileSourceStrategy mode
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-14551
>                 URL: https://issues.apache.org/jira/browse/SPARK-14551
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> When FileSourceStrategy is used, record reader is created which incurs a NN call internally. Later in OrcRelation.unwrapOrcStructs, it ends ups reading the file information to get the ObjectInspector. This incurs additional NN call. It would be good to avoid this additional NN call (specifically for partitioned datasets)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org