You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2016/04/12 06:10:25 UTC

[jira] [Updated] (SPARK-14551) Reduce number of NameNode calls in OrcRelation with FileSourceStrategy mode

     [ https://issues.apache.org/jira/browse/SPARK-14551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajesh Balamohan updated SPARK-14551:
-------------------------------------
    Summary: Reduce number of NameNode calls in OrcRelation with FileSourceStrategy mode  (was: Reduce number of NN calls in OrcRelation with FileSourceStrategy mode)

> Reduce number of NameNode calls in OrcRelation with FileSourceStrategy mode
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-14551
>                 URL: https://issues.apache.org/jira/browse/SPARK-14551
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>
> When FileSourceStrategy is used, record reader is created which incurs a NN call internally. Later in OrcRelation.unwrapOrcStructs, it ends ups reading the file information to get the ObjectInspector. This incurs additional NN call. It would be good to avoid this additional NN call (specifically for partitioned datasets)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org