You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by bdev <bu...@gmail.com> on 2014/10/09 21:40:31 UTC
How to save ReceiverInputDStream to Hadoop using
saveAsNewAPIHadoopFile
I'm using KafkaUtils.createStream for the input stream to pull messages from
kafka which seems to return a ReceiverInputDStream. I do not see
saveAsNewAPIHadoopFile available on ReceiverInputDStream and obviously run
into this error:
saveAsNewAPIHadoopFile is not a member of
org.apache.spark.streaming.dstream.DStream[String]
Any help on how to go about saving a DStream to Hadoop would be appreciated.
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-ReceiverInputDStream-to-Hadoop-using-saveAsNewAPIHadoopFile-tp16062.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to save ReceiverInputDStream to Hadoop using saveAsNewAPIHadoopFile
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Hi Buntu,
You could something similar to the following:
val receiver_stream = new ReceiverInputDStream(ssc) {
> override def getReceiver(): Receiver[Nothing] = ??? //Whatever
> }.map((x : String) => (null, x))
> val config = new Configuration()
> config.set("mongo.output.uri", "mongodb://akhld:27017/sigmoid.output")
> receiver_stream.foreachRDD(rdd => {
> val pair_rdd = new PairRDDFunctions[Null, String](rdd) // make sure
> your rdd contains a key, value
> *pair_rdd.saveAsNewAPIHadoopFile("/home/akhld/sigmoid/beta/",
> classOf[Any], classOf[Any],
> classOf[com.mongodb.hadoop.MongoOutputFormat[Any, Any]], config)*
> })
Thanks
Best Regards
On Tue, Oct 21, 2014 at 11:59 PM, Buntu Dev <bu...@gmail.com> wrote:
> Thanks Akhil,
>
> I tried this but running into similar error:
>
> ~~~~~~
> val stream = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicpMap).map(_._2)
> stream.map(message => {
> (null, message)
>
> }).saveAsNewAPIHadoopFile (destination, classOf[Void],
> classOf[Group], classOf[ExampleOutputFormat], conf)
> ~~~~
>
> Error:
> value saveAsNewAPIHadoopFile is not a member of
> org.apache.spark.rdd.RDD[(Null, String)]
>
>
> How do I go about converting to PairRDDFunctions?
>
>
> On Fri, Oct 10, 2014 at 12:01 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> You can convert this ReceiverInputDStream
>> <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.ReceiverInputDStream>
>> into PairRDDFuctions
>> <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions>
>> and call the saveAsNewAPIHadoopFile.
>>
>> Thanks
>> Best Regards
>>
>> On Fri, Oct 10, 2014 at 11:28 AM, Buntu Dev <bu...@gmail.com> wrote:
>>
>>> Basically I'm attempting to convert a JSON stream to Parquet and I get
>>> this error without the .values or .map(_._2) :
>>>
>>> value saveAsNewAPIHadoopFile is not a member of
>>> org.apache.spark.streaming.dstream.ReceiverInputDStream[(String, String)]
>>>
>>>
>>>
>>>
>>> On Thu, Oct 9, 2014 at 10:15 PM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>>> Your RDD does not contain pairs, since you ".map(_._2)" (BTW that can
>>>> just be ".values"). "Hadoop files" means "SequenceFiles" and those
>>>> store key-value pairs. That's why the method only appears for
>>>> RDD[(K,V)].
>>>>
>>>> On Fri, Oct 10, 2014 at 3:50 AM, Buntu Dev <bu...@gmail.com> wrote:
>>>> > Thanks Sean, but I'm importing
>>>> org.apache.spark.streaming.StreamingContext._
>>>> >
>>>> > Here are the spark imports:
>>>> >
>>>> > import org.apache.spark.streaming._
>>>> >
>>>> > import org.apache.spark.streaming.StreamingContext._
>>>> >
>>>> > import org.apache.spark.streaming.kafka._
>>>> >
>>>> > import org.apache.spark.SparkConf
>>>> >
>>>> > ....
>>>> >
>>>> > val stream = KafkaUtils.createStream(ssc, zkQuorum, group,
>>>> > topicpMap).map(_._2) stream.saveAsNewAPIHadoopFile
>>>> (destination,
>>>> > classOf[Void], classOf[Group], classOf[ExampleOutputFormat], conf)
>>>> >
>>>> > ....
>>>> >
>>>> > Anything else I might be missing?
>>>>
>>>
>>>
>>
>
Re: How to save ReceiverInputDStream to Hadoop using saveAsNewAPIHadoopFile
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can convert this ReceiverInputDStream
<http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.ReceiverInputDStream>
into PairRDDFuctions
<http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions>
and call the saveAsNewAPIHadoopFile.
Thanks
Best Regards
On Fri, Oct 10, 2014 at 11:28 AM, Buntu Dev <bu...@gmail.com> wrote:
> Basically I'm attempting to convert a JSON stream to Parquet and I get
> this error without the .values or .map(_._2) :
>
> value saveAsNewAPIHadoopFile is not a member of
> org.apache.spark.streaming.dstream.ReceiverInputDStream[(String, String)]
>
>
>
>
> On Thu, Oct 9, 2014 at 10:15 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> Your RDD does not contain pairs, since you ".map(_._2)" (BTW that can
>> just be ".values"). "Hadoop files" means "SequenceFiles" and those
>> store key-value pairs. That's why the method only appears for
>> RDD[(K,V)].
>>
>> On Fri, Oct 10, 2014 at 3:50 AM, Buntu Dev <bu...@gmail.com> wrote:
>> > Thanks Sean, but I'm importing
>> org.apache.spark.streaming.StreamingContext._
>> >
>> > Here are the spark imports:
>> >
>> > import org.apache.spark.streaming._
>> >
>> > import org.apache.spark.streaming.StreamingContext._
>> >
>> > import org.apache.spark.streaming.kafka._
>> >
>> > import org.apache.spark.SparkConf
>> >
>> > ....
>> >
>> > val stream = KafkaUtils.createStream(ssc, zkQuorum, group,
>> > topicpMap).map(_._2) stream.saveAsNewAPIHadoopFile
>> (destination,
>> > classOf[Void], classOf[Group], classOf[ExampleOutputFormat], conf)
>> >
>> > ....
>> >
>> > Anything else I might be missing?
>>
>
>
Re: How to save ReceiverInputDStream to Hadoop using saveAsNewAPIHadoopFile
Posted by Buntu Dev <bu...@gmail.com>.
Basically I'm attempting to convert a JSON stream to Parquet and I get this
error without the .values or .map(_._2) :
value saveAsNewAPIHadoopFile is not a member of
org.apache.spark.streaming.dstream.ReceiverInputDStream[(String, String)]
On Thu, Oct 9, 2014 at 10:15 PM, Sean Owen <so...@cloudera.com> wrote:
> Your RDD does not contain pairs, since you ".map(_._2)" (BTW that can
> just be ".values"). "Hadoop files" means "SequenceFiles" and those
> store key-value pairs. That's why the method only appears for
> RDD[(K,V)].
>
> On Fri, Oct 10, 2014 at 3:50 AM, Buntu Dev <bu...@gmail.com> wrote:
> > Thanks Sean, but I'm importing
> org.apache.spark.streaming.StreamingContext._
> >
> > Here are the spark imports:
> >
> > import org.apache.spark.streaming._
> >
> > import org.apache.spark.streaming.StreamingContext._
> >
> > import org.apache.spark.streaming.kafka._
> >
> > import org.apache.spark.SparkConf
> >
> > ....
> >
> > val stream = KafkaUtils.createStream(ssc, zkQuorum, group,
> > topicpMap).map(_._2) stream.saveAsNewAPIHadoopFile
> (destination,
> > classOf[Void], classOf[Group], classOf[ExampleOutputFormat], conf)
> >
> > ....
> >
> > Anything else I might be missing?
>
Re: How to save ReceiverInputDStream to Hadoop using saveAsNewAPIHadoopFile
Posted by Sean Owen <so...@cloudera.com>.
Your RDD does not contain pairs, since you ".map(_._2)" (BTW that can
just be ".values"). "Hadoop files" means "SequenceFiles" and those
store key-value pairs. That's why the method only appears for
RDD[(K,V)].
On Fri, Oct 10, 2014 at 3:50 AM, Buntu Dev <bu...@gmail.com> wrote:
> Thanks Sean, but I'm importing org.apache.spark.streaming.StreamingContext._
>
> Here are the spark imports:
>
> import org.apache.spark.streaming._
>
> import org.apache.spark.streaming.StreamingContext._
>
> import org.apache.spark.streaming.kafka._
>
> import org.apache.spark.SparkConf
>
> ....
>
> val stream = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicpMap).map(_._2) stream.saveAsNewAPIHadoopFile (destination,
> classOf[Void], classOf[Group], classOf[ExampleOutputFormat], conf)
>
> ....
>
> Anything else I might be missing?
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to save ReceiverInputDStream to Hadoop using saveAsNewAPIHadoopFile
Posted by Buntu Dev <bu...@gmail.com>.
Thanks Sean, but I'm importing org.apache.spark.streaming.
StreamingContext._
Here are the spark imports:
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._
import org.apache.spark.SparkConf
....
val stream = KafkaUtils.createStream(ssc, zkQuorum, group,
topicpMap).map(_._2) stream.saveAsNewAPIHadoopFile
(destination, classOf[Void], classOf[Group], classOf[ExampleOutputFormat],
conf)
....
Anything else I might be missing?
On Thu, Oct 9, 2014 at 1:14 PM, Sean Owen <so...@cloudera.com> wrote:
> I think you have not imported
> org.apache.spark.streaming.StreamingContext._ ? This gets you the
> implicits that provide these methods.
>
> On Thu, Oct 9, 2014 at 8:40 PM, bdev <bu...@gmail.com> wrote:
> > I'm using KafkaUtils.createStream for the input stream to pull messages
> from
> > kafka which seems to return a ReceiverInputDStream. I do not see
> > saveAsNewAPIHadoopFile available on ReceiverInputDStream and obviously
> run
> > into this error:
> >
> > saveAsNewAPIHadoopFile is not a member of
> > org.apache.spark.streaming.dstream.DStream[String]
> >
> > Any help on how to go about saving a DStream to Hadoop would be
> appreciated.
> >
> > Thanks!
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-ReceiverInputDStream-to-Hadoop-using-saveAsNewAPIHadoopFile-tp16062.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
Re: How to save ReceiverInputDStream to Hadoop using saveAsNewAPIHadoopFile
Posted by Sean Owen <so...@cloudera.com>.
I think you have not imported
org.apache.spark.streaming.StreamingContext._ ? This gets you the
implicits that provide these methods.
On Thu, Oct 9, 2014 at 8:40 PM, bdev <bu...@gmail.com> wrote:
> I'm using KafkaUtils.createStream for the input stream to pull messages from
> kafka which seems to return a ReceiverInputDStream. I do not see
> saveAsNewAPIHadoopFile available on ReceiverInputDStream and obviously run
> into this error:
>
> saveAsNewAPIHadoopFile is not a member of
> org.apache.spark.streaming.dstream.DStream[String]
>
> Any help on how to go about saving a DStream to Hadoop would be appreciated.
>
> Thanks!
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-ReceiverInputDStream-to-Hadoop-using-saveAsNewAPIHadoopFile-tp16062.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org