You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/04/12 04:37:26 UTC

[jira] [Commented] (SPARK-14551) Reduce number of NN calls in OrcRelation with FileSourceStrategy mode

    [ https://issues.apache.org/jira/browse/SPARK-14551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236478#comment-15236478 ] 

Apache Spark commented on SPARK-14551:
--------------------------------------

User 'rajeshbalamohan' has created a pull request for this issue:
https://github.com/apache/spark/pull/12319

> Reduce number of NN calls in OrcRelation with FileSourceStrategy mode
> ---------------------------------------------------------------------
>
>                 Key: SPARK-14551
>                 URL: https://issues.apache.org/jira/browse/SPARK-14551
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>
> When FileSourceStrategy is used, record reader is created which incurs a NN call internally. Later in OrcRelation.unwrapOrcStructs, it ends ups reading the file information to get the ObjectInspector. This incurs additional NN call. It would be good to avoid this additional NN call (specifically for partitioned datasets)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org