You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2014/12/31 00:04:13 UTC
[jira] [Commented] (SPARK-2247) Data frame (or Pandas) like API for
structured data
[ https://issues.apache.org/jira/browse/SPARK-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261613#comment-14261613 ]
Reynold Xin commented on SPARK-2247:
------------------------------------
I took another look at various data frame implementations out there and SchemaRDD. As it is right now, SchemaRDD along with its DSLs already delivers most of the basic functionalities of a data frame (e.g. you can read data into create new SchemaRDDs, you can project a SchemaRDD, you can run aggregations, and you can even join different SchemaRDDs).
The problem with the DSL is that it was originally designed for writing test cases so its various functions are often more verbose than needed. There also lacks a Python variant of the DSL. We should probably just update the DSL API to make it more useable and call it a day. Then for people programming for structured data, they can use the SchemaRDD DSL instead of directly using the RDD API.
As for R, it should be part of SparkR.
> Data frame (or Pandas) like API for structured data
> ---------------------------------------------------
>
> Key: SPARK-2247
> URL: https://issues.apache.org/jira/browse/SPARK-2247
> Project: Spark
> Issue Type: New Feature
> Components: PySpark, Spark Core, SQL
> Affects Versions: 1.0.0
> Reporter: venu k tangirala
> Labels: features
>
> I would be nice to have R or python pandas like data frames on spark.
> 1) To be able to access the RDD data frame from python with pandas
> 2) To be able to access the RDD data frame from R
> 3) To be able to access the RDD data frame from scala's saddle
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org