You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jdkorigan <su...@korigan.com> on 2016/02/09 13:22:18 UTC
createDataFrame question
Hi,
I would like to transform my rdd to a sql.dataframe.Dataframe, is there a
possible conversion to do the job? or what would be the easiest way to do
it?
def ConvertVal(iter):
# some code
return sqlContext.createDataFrame(Row("val1", "val2", "val3", "val4"))
rdd = sc.textFile("").mapPartitions(ConvertVal)
print(type(rdd)) #<class 'pyspark.rdd.PipelinedRDD'>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/createDataFrame-question-tp26178.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: createDataFrame question
Posted by satish chandra j <js...@gmail.com>.
HI,
Hope you are aware of "toDF()" which is used to convert your RDD to
DataFrame
Regards,
Satish Chandra
On Tue, Feb 9, 2016 at 5:52 PM, jdkorigan <su...@korigan.com> wrote:
> Hi,
>
> I would like to transform my rdd to a sql.dataframe.Dataframe, is there a
> possible conversion to do the job? or what would be the easiest way to do
> it?
>
> def ConvertVal(iter):
> # some code
> return sqlContext.createDataFrame(Row("val1", "val2", "val3", "val4"))
>
>
> rdd = sc.textFile("").mapPartitions(ConvertVal)
>
> print(type(rdd)) #<class 'pyspark.rdd.PipelinedRDD'>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/createDataFrame-question-tp26178.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: createDataFrame question
Posted by jdkorigan <su...@korigan.com>.
The correct way, is just to remove "sqlContext.createDataFrame", and
everything works correctly
def ConvertVal(iter):
# some code
return Row("val1", "val2", "val3", "val4")
rdd = sc.textFile("").mapPartitions(ConvertVal).toDF()
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/createDataFrame-question-tp26178p26182.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: createDataFrame question
Posted by jdkorigan <su...@korigan.com>.
When using this function: rdd =
sc.textFile("").mapPartitions(ConvertVal).toDF()
I get an exception and the last line is:
TypeError: 'JavaPackage' object is not callable
Since my function return value is already DataFrame, maybe there is a way to
access this type from my rdd?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/createDataFrame-question-tp26178p26180.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org