You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Akhil Das <ak...@hacked.work> on 2016/07/01 10:42:04 UTC
Re: RDD to DataFrame question with JsValue in the mix
Something like this?
import sqlContext.implicits._
case class Holder(str: String, js:JsValue)
yourRDD.map(x => Holder(x._1, x._2)).toDF()
On Fri, Jul 1, 2016 at 3:36 AM, Dood@ODDO <od...@gmail.com> wrote:
> Hello,
>
> I have an RDD[(String,JsValue)] that I want to convert into a DataFrame
> and then run SQL on. What is the easiest way to get the JSON (in form of
> JsValue) "understood" by the process?
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
--
Cheers!
Re: RDD to DataFrame question with JsValue in the mix
Posted by Do...@ODDO,
od...@gmail.com.
On 7/1/2016 6:42 AM, Akhil Das wrote:
> case class Holder(str: String, js:JsValue)
Hello,
Thanks!
I tried that before posting the question to the list but I keep getting
an error such as this even after the map() operation to convert
(String,JsValue) -> Holder and then toDF().
I am simply invoking the following:
val rddDF:DataFrame = rdd.map(x => Holder(x._1,x._2)).toDF
rddDF.registerTempTable("rddf")
rddDF.schema.mkString(",")
And getting the following:
[2016-07-01 11:57:02,720] WARN .jobserver.JobManagerActor []
[akka://JobServer/user/context-supervisor/test] - Exception from job
d4c9d145-92bf-4c64-8904-91c917bd61d3:
java.lang.UnsupportedOperationException: Schema for type
play.api.libs.json.JsValue is not supported
at
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:718)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:30)
at
org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:693)
at
org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:691)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:691)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:30)
at
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:630)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:30)
at
org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:414)
at
org.apache.spark.sql.SQLImplicits.rddToDataFrameHolder(SQLImplicits.scala:94)
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org