You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Cheng Lian (JIRA)" <ji...@apache.org> on 2014/06/11 21:14:02 UTC
[jira] [Comment Edited] (SPARK-2081) Undefine output() from the
abstract class Command and implement it in concrete subclasses
[ https://issues.apache.org/jira/browse/SPARK-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028245#comment-14028245 ]
Cheng Lian edited comment on SPARK-2081 at 6/11/14 7:12 PM:
------------------------------------------------------------
To [~marmbrus]: Just found out that I happened to solve this issue while working on [SPARK-2094 (exactly once semantics for DDL and commands)|https://issues.apache.org/jira/browse/SPARK-2094], please assign this issue to me.
A tricky part here is how to deal with {{NativeCommand}}. Currently, we split each output text line around {{'\t'}}, and make each of the component a field of the resulting rows. This makes the schema of a {{NativeCommand}} unpredictable (we don't know how many field are there) unless we create concrete logical/physical plan classes for each kind of command. To simplify things for now, I propose that the schema of a {{NativeCommand}} only owns 1 "result" field whose data type is {{StringType}}, and stop splitting the output text lines.
was (Author: lian cheng):
[~marmbrus] Just found out that I happened to solve this issue while working on [SPARK-2094 (exactly once semantics for DDL and commands)|https://issues.apache.org/jira/browse/SPARK-2094], please assign this issue to me.
A tricky part here is how to deal with {{NativeCommand}}. Currently, we split each output text line around {{'\t'}}, and make each of the component a field of the resulting rows. This makes the schema of a {{NativeCommand}} unpredictable (we don't know how many field are there) unless we create concrete logical/physical plan classes for each kind of command. To simplify things for now, I propose that the schema of a {{NativeCommand}} only owns 1 "result" field whose data type is {{StringType}}, and stop splitting the output text lines.
> Undefine output() from the abstract class Command and implement it in concrete subclasses
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-2081
> URL: https://issues.apache.org/jira/browse/SPARK-2081
> Project: Spark
> Issue Type: Improvement
> Reporter: Zongheng Yang
> Assignee: Zongheng Yang
> Priority: Minor
>
> It doesn't make too much sense to have that method in the abstract class.
> Relevant discussions / cases where this issue comes up:
> https://github.com/apache/spark/pull/956
> https://github.com/apache/spark/pull/1003
--
This message was sent by Atlassian JIRA
(v6.2#6252)