You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Verma, Rishi (398J)" <Ri...@jpl.nasa.gov> on 2014/08/28 19:28:54 UTC

Converting a DStream's RDDs to SchemaRDDs

Hi Folks,

I’d like to find out tips on how to convert the RDDs inside a Spark Streaming DStream to a set of SchemaRDDs.

My DStream contains JSON data pushed over from Kafka, and I’d like to use SparkSQL’s JSON import function (i.e. jsonRDD) to register the JSON dataset as a table, and perform queries on it.

Here’s a code snippet of my latest attempt (in Scala):
…
val sc = new SparkContext(conf)
val ssc = new StreamingContext("local", this.getClass.getName, Seconds(1))
ssc.checkpoint("checkpoint")

val stream = KafkaUtils.createStream(ssc, "localhost:2181", “group", Map(“topic" -> 10)).map(_._2)
val sql = new SQLContext(sc)

stream.foreachRDD(rdd => {
	if (rdd.count > 0) {
		// message received
		val sqlRDD = sql.jsonRDD(rdd)
		sqlRDD.printSchema()
	} else {
		println(“No message received")
	}
})
…

This compiles and runs when I submit it to Spark (local-mode); however, I never seem to be able to successfully see a schema printed on my console, via the “sqlRDD.printSchema()” method when Kafka is streaming my JSON messages to the “topic” topic name. I know my JSON is valid and my Kafka connection works fine, I’ve been able to print the stream messages in their raw format, just not as SchemaRDDs.  

Any tips? Suggestions?

Thanks much,
---
Rishi Verma
NASA Jet Propulsion Laboratory
California Institute of Technology





---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Converting a DStream's RDDs to SchemaRDDs

Posted by "Verma, Rishi (398J)" <Ri...@jpl.nasa.gov>.
Hi Tathagata,

Thanks very much for this tip and for the quick reply. It absolutely did the trick!

One small note for others: make sure your command-line invocation of the spark-submit script has the argument “—master” also setting “local[n]” with n > 1.

Thanks again,
Rishi

On Aug 28, 2014, at 1:09 PM, Tathagata Das <ta...@gmail.com>> wrote:

Try using "local[n]" with n > 1, instead of local. Since receivers take up 1 slot, and "local" is basically 1 slot, there is no slot left to process the data. That's why nothing gets printed.

TD


On Thu, Aug 28, 2014 at 10:28 AM, Verma, Rishi (398J) <Ri...@jpl.nasa.gov>> wrote:
Hi Folks,

I’d like to find out tips on how to convert the RDDs inside a Spark Streaming DStream to a set of SchemaRDDs.

My DStream contains JSON data pushed over from Kafka, and I’d like to use SparkSQL’s JSON import function (i.e. jsonRDD) to register the JSON dataset as a table, and perform queries on it.

Here’s a code snippet of my latest attempt (in Scala):
…
val sc = new SparkContext(conf)
val ssc = new StreamingContext("local", this.getClass.getName, Seconds(1))
ssc.checkpoint("checkpoint")

val stream = KafkaUtils.createStream(ssc, "localhost:2181", “group", Map(“topic" -> 10)).map(_._2)
val sql = new SQLContext(sc)

stream.foreachRDD(rdd => {
        if (rdd.count > 0) {
                // message received
                val sqlRDD = sql.jsonRDD(rdd)
                sqlRDD.printSchema()
        } else {
                println(“No message received")
        }
})
…

This compiles and runs when I submit it to Spark (local-mode); however, I never seem to be able to successfully see a schema printed on my console, via the “sqlRDD.printSchema()” method when Kafka is streaming my JSON messages to the “topic” topic name. I know my JSON is valid and my Kafka connection works fine, I’ve been able to print the stream messages in their raw format, just not as SchemaRDDs.

Any tips? Suggestions?

Thanks much,
---
Rishi Verma
NASA Jet Propulsion Laboratory
California Institute of Technology





---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>



---
Rishi Verma
NASA Jet Propulsion Laboratory
California Institute of Technology





Re: Converting a DStream's RDDs to SchemaRDDs

Posted by Tathagata Das <ta...@gmail.com>.
Try using "local[n]" with n > 1, instead of local. Since receivers take up
1 slot, and "local" is basically 1 slot, there is no slot left to process
the data. That's why nothing gets printed.

TD


On Thu, Aug 28, 2014 at 10:28 AM, Verma, Rishi (398J) <
Rishi.Verma@jpl.nasa.gov> wrote:

> Hi Folks,
>
> I’d like to find out tips on how to convert the RDDs inside a Spark
> Streaming DStream to a set of SchemaRDDs.
>
> My DStream contains JSON data pushed over from Kafka, and I’d like to use
> SparkSQL’s JSON import function (i.e. jsonRDD) to register the JSON dataset
> as a table, and perform queries on it.
>
> Here’s a code snippet of my latest attempt (in Scala):
> …
> val sc = new SparkContext(conf)
> val ssc = new StreamingContext("local", this.getClass.getName, Seconds(1))
> ssc.checkpoint("checkpoint")
>
> val stream = KafkaUtils.createStream(ssc, "localhost:2181", “group",
> Map(“topic" -> 10)).map(_._2)
> val sql = new SQLContext(sc)
>
> stream.foreachRDD(rdd => {
>         if (rdd.count > 0) {
>                 // message received
>                 val sqlRDD = sql.jsonRDD(rdd)
>                 sqlRDD.printSchema()
>         } else {
>                 println(“No message received")
>         }
> })
> …
>
> This compiles and runs when I submit it to Spark (local-mode); however, I
> never seem to be able to successfully see a schema printed on my console,
> via the “sqlRDD.printSchema()” method when Kafka is streaming my JSON
> messages to the “topic” topic name. I know my JSON is valid and my Kafka
> connection works fine, I’ve been able to print the stream messages in their
> raw format, just not as SchemaRDDs.
>
> Any tips? Suggestions?
>
> Thanks much,
> ---
> Rishi Verma
> NASA Jet Propulsion Laboratory
> California Institute of Technology
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>