You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/05/14 18:57:00 UTC
[jira] [Assigned] (SPARK-7591) FSBasedRelation interface tweaks
[ https://issues.apache.org/jira/browse/SPARK-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-7591:
-----------------------------------
Assignee: Apache Spark (was: Cheng Lian)
> FSBasedRelation interface tweaks
> --------------------------------
>
> Key: SPARK-7591
> URL: https://issues.apache.org/jira/browse/SPARK-7591
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Reynold Xin
> Assignee: Apache Spark
> Priority: Blocker
>
> # Renaming {{FSBasedRelation}} to {{HadoopFsRelation}}
> Since itss all coupled with Hadoop {{FileSystem}} and job API.
> # {{HadoopFsRelation}} should have a no-arg constructor
> {{paths}} and {{partitionColumns}} should just be methods to be overridden, rather than constructor arguments. This makes data source developers life easier by having a no-arg constructor and being serialization friendly.
> # Renaming {{HadoopFsRelation.prepareForWrite}} to {{HadoopFsRelation.prepareJobForWrite}}
> The new name explicitly suggests developers should only touch the {{Job}} instance for preparation work (which is also documented in Scaladoc).
> # Allowing serialization while creating {{OutputWriter}}s
> To be more precise, {{OutputWriter}}s are never created on driver side and serialized to executor side. But the factory that creates {{OutputWriter}}s should be created on driver side and serialized.
> The reason behind this is that, passing all needed materials to {{OutputWriter}} instances via Hadoop Configuration is doable but sometimes neither intuitive nor convenient. Resorting to serialization makes data source developers' life easier. Actually this happens when I was migrating the Parquet data source, and wanted to pass the final output path (instead of temporary work path) to the output writer (see [here|https://github.com/liancheng/spark/commit/ec9950c591e5b981ce20fab96562db28488e0035#diff-53521d336f7259e859fea4d3ca4dc888R74]). There I have to put a property into the Configuration object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org