You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rafeeq s <ra...@gmail.com> on 2014/08/05 09:22:15 UTC
Spark stream data from kafka topics and output as parquet file on HDFS
Hi,
I am new to Apache Spark and Trying to Develop spark streaming program
to *stream
data from kafka topics and output as parquet file on HDFS*.
Please share the *sample reference* program to stream data from kafka
topics and output as parquet file on HDFS.
Thanks in Advance.
Regards,
Rafeeq S
*(“What you do is what matters, not what you think or say or plan.” )*
RE: Spark stream data from kafka topics and output as parquet file
on HDFS
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Rafeeq,
I think current Spark Streaming api can offer you the ability to fetch data from Kafka and store to another external store, if you do not care about management of consumer offset manually, there’s no need to use low level api as SimpleConsumer.
For Kafka 0.8.1 compatibility, you can try to modify the pom file and rebuild Spark to try it, mostly I think it can work.
For parquet file, I think if parquet offers its own OutputFormat that is extended from Hadoop’s OutputFormat, Spark can write data into parquet file, like sequence file or text file, you can do this as:
DStream.foreach { rdd => rdd.saveAsHadoopFile(…) } to specify the OutputFormat you want.
Thanks
Jerry
From: rafeeq s [mailto:rafeeq.ece71@gmail.com]
Sent: Tuesday, August 05, 2014 5:37 PM
To: Dibyendu Bhattacharya
Cc: user@spark.incubator.apache.org
Subject: Re: Spark stream data from kafka topics and output as parquet file on HDFS
Thanks Dibyendu.
1. Spark itself have api jar for kafka, still we require manual offset management (using simple consumer concept) and manual consumer ?
2.Kafka Spark Consumer which is implemented in kafka 0.8.0 ,Can we use it for kafka 0.8.1 ?
3.How to use Kafka Spark Consumer to produce output as parquet file on HDFS ?
Please give your suggestion.
Regards,
Rafeeq S
(“What you do is what matters, not what you think or say or plan.” )
On Tue, Aug 5, 2014 at 11:55 AM, Dibyendu Bhattacharya <di...@gmail.com>> wrote:
You can try this Kafka Spark Consumer which I recently wrote. This uses the Low Level Kafka Consumer
https://github.com/dibbhatt/kafka-spark-consumer
Dibyendu
On Tue, Aug 5, 2014 at 12:52 PM, rafeeq s <ra...@gmail.com>> wrote:
Hi,
I am new to Apache Spark and Trying to Develop spark streaming program to stream data from kafka topics and output as parquet file on HDFS.
Please share the sample reference program to stream data from kafka topics and output as parquet file on HDFS.
Thanks in Advance.
Regards,
Rafeeq S
(“What you do is what matters, not what you think or say or plan.” )
Re: Spark stream data from kafka topics and output as parquet file on HDFS
Posted by rafeeq s <ra...@gmail.com>.
Thanks Dibyendu.
1. Spark itself have api jar for kafka, still we require manual offset
management (using simple consumer concept) and manual consumer ?
2.Kafka Spark Consumer which is implemented in kafka 0.8.0 ,Can we use it
for kafka 0.8.1 ?
3.How to use Kafka Spark Consumer to produce output
*as parquet file on HDFS ?*
*Please give your suggestion.*
Regards,
Rafeeq S
*(“What you do is what matters, not what you think or say or plan.” )*
On Tue, Aug 5, 2014 at 11:55 AM, Dibyendu Bhattacharya <
dibyendu.bhattachary@gmail.com> wrote:
> You can try this Kafka Spark Consumer which I recently wrote. This uses
> the Low Level Kafka Consumer
>
> https://github.com/dibbhatt/kafka-spark-consumer
>
> Dibyendu
>
>
>
>
> On Tue, Aug 5, 2014 at 12:52 PM, rafeeq s <ra...@gmail.com> wrote:
>
>> Hi,
>>
>> I am new to Apache Spark and Trying to Develop spark streaming program
>> to *stream data from kafka topics and output as parquet file on HDFS*.
>>
>> Please share the *sample reference* program to stream data from kafka
>> topics and output as parquet file on HDFS.
>>
>> Thanks in Advance.
>>
>> Regards,
>>
>> Rafeeq S
>> *(“What you do is what matters, not what you think or say or plan.” )*
>>
>>
>
Re: Spark stream data from kafka topics and output as parquet file on HDFS
Posted by Sameer Sayyed <sa...@gmail.com>.
hello,
Code:
ZkState zkState = new ZkState(kafkaConfig);
DynamicBrokersReader kafkaBrokerReader = new
DynamicBrokersReader(kafkaConfig, zkState);
int partionCount = kafkaBrokerReader.getNumPartitions();
SparkConf _sparkConf = new SparkConf().setAppName("KafkaReceiver");
final JavaStreamingContext ssc = new JavaStreamingContext(_sparkConf, new
Duration(1000));
final JavaReceiverInputDStream<String> inputStream = ssc.receiverStream(new
KafkaReceiver(_props, partionCount));
final JavaSQLContext sqlContext = new
org.apache.spark.sql.api.java.JavaSQLContext(ssc.sparkContext());
inputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() {
@Override
public Void call(JavaRDD<String> rdd, Time time) throws Exception {
if(rdd != null) {
JavaSchemaRDD schemaObject = sqlContext.jsonRDD(rdd);
schemaObject.saveAsParquetFile("tweet" + time.toString() + ".parquet");
System.out.println("File Saved Successfully");
}else {
System.out.println("rdd is empty");
}
Error:
java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:806)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:806)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:806)
at org.apache.spark.sql.json.JsonRDD$.inferSchema(JsonRDD.scala:40)
at
org.apache.spark.sql.api.java.JavaSQLContext.jsonRDD(JavaSQLContext.scala:123)
at consumer.kafka.client.Consumer$1.call(Consumer.java:115)
at consumer.kafka.client.Consumer$1.call(Consumer.java:109)
at
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:282)
at
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:282)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Please suggest me solution.
Thanks in Advance.
On Thu, Aug 7, 2014 at 1:06 AM, Tathagata Das <ta...@gmail.com>
wrote:
> You can use SparkSQL for that very easily. You can convert the rdds you
> get from kafka input stream, convert them to a RDDs of case classes and
> save as parquet files.
> More information here.
>
> https://spark.apache.org/docs/latest/sql-programming-guide.html#parquet-files
>
>
> On Wed, Aug 6, 2014 at 5:23 AM, Mahebub Sayyed <ma...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I have referred link "https://github.com/dibbhatt/kafka-spark-consumer"
>> and I have successfully consumed tuples from kafka.
>> Tuples are JSON objects and I want to store that objects in HDFS as
>> parque format.
>>
>> Please suggest me any sample example for that.
>> Thanks in advance.
>>
>>
>>
>>
>>
>> On Tue, Aug 5, 2014 at 11:55 AM, Dibyendu Bhattacharya <
>> dibyendu.bhattachary@gmail.com> wrote:
>>
>>> You can try this Kafka Spark Consumer which I recently wrote. This uses
>>> the Low Level Kafka Consumer
>>>
>>> https://github.com/dibbhatt/kafka-spark-consumer
>>>
>>> Dibyendu
>>>
>>>
>>>
>>>
>>> On Tue, Aug 5, 2014 at 12:52 PM, rafeeq s <ra...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am new to Apache Spark and Trying to Develop spark streaming program
>>>> to *stream data from kafka topics and output as parquet file on HDFS*.
>>>>
>>>> Please share the *sample reference* program to stream data from kafka
>>>> topics and output as parquet file on HDFS.
>>>>
>>>> Thanks in Advance.
>>>>
>>>> Regards,
>>>>
>>>> Rafeeq S
>>>> *(“What you do is what matters, not what you think or say or plan.” )*
>>>>
>>>>
>>>
>>
>>
>> --
>>
>>
>
Re: Spark stream data from kafka topics and output as parquet file on HDFS
Posted by Tathagata Das <ta...@gmail.com>.
You can use SparkSQL for that very easily. You can convert the rdds you get
from kafka input stream, convert them to a RDDs of case classes and save as
parquet files.
More information here.
https://spark.apache.org/docs/latest/sql-programming-guide.html#parquet-files
On Wed, Aug 6, 2014 at 5:23 AM, Mahebub Sayyed <ma...@gmail.com> wrote:
> Hello,
>
> I have referred link "https://github.com/dibbhatt/kafka-spark-consumer"
> and I have successfully consumed tuples from kafka.
> Tuples are JSON objects and I want to store that objects in HDFS as parque
> format.
>
> Please suggest me any sample example for that.
> Thanks in advance.
>
>
>
>
>
> On Tue, Aug 5, 2014 at 11:55 AM, Dibyendu Bhattacharya <
> dibyendu.bhattachary@gmail.com> wrote:
>
>> You can try this Kafka Spark Consumer which I recently wrote. This uses
>> the Low Level Kafka Consumer
>>
>> https://github.com/dibbhatt/kafka-spark-consumer
>>
>> Dibyendu
>>
>>
>>
>>
>> On Tue, Aug 5, 2014 at 12:52 PM, rafeeq s <ra...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am new to Apache Spark and Trying to Develop spark streaming program
>>> to *stream data from kafka topics and output as parquet file on HDFS*.
>>>
>>> Please share the *sample reference* program to stream data from kafka
>>> topics and output as parquet file on HDFS.
>>>
>>> Thanks in Advance.
>>>
>>> Regards,
>>>
>>> Rafeeq S
>>> *(“What you do is what matters, not what you think or say or plan.” )*
>>>
>>>
>>
>
>
> --
> *Regards,*
> *Mahebub Sayyed*
>
Re: Spark stream data from kafka topics and output as parquet file on HDFS
Posted by Mahebub Sayyed <ma...@gmail.com>.
Hello,
I have referred link "https://github.com/dibbhatt/kafka-spark-consumer" and
I have successfully consumed tuples from kafka.
Tuples are JSON objects and I want to store that objects in HDFS as parque
format.
Please suggest me any sample example for that.
Thanks in advance.
On Tue, Aug 5, 2014 at 11:55 AM, Dibyendu Bhattacharya <
dibyendu.bhattachary@gmail.com> wrote:
> You can try this Kafka Spark Consumer which I recently wrote. This uses
> the Low Level Kafka Consumer
>
> https://github.com/dibbhatt/kafka-spark-consumer
>
> Dibyendu
>
>
>
>
> On Tue, Aug 5, 2014 at 12:52 PM, rafeeq s <ra...@gmail.com> wrote:
>
>> Hi,
>>
>> I am new to Apache Spark and Trying to Develop spark streaming program
>> to *stream data from kafka topics and output as parquet file on HDFS*.
>>
>> Please share the *sample reference* program to stream data from kafka
>> topics and output as parquet file on HDFS.
>>
>> Thanks in Advance.
>>
>> Regards,
>>
>> Rafeeq S
>> *(“What you do is what matters, not what you think or say or plan.” )*
>>
>>
>
--
*Regards,*
*Mahebub Sayyed*
Re: Spark stream data from kafka topics and output as parquet file on HDFS
Posted by Dibyendu Bhattacharya <di...@gmail.com>.
You can try this Kafka Spark Consumer which I recently wrote. This uses the
Low Level Kafka Consumer
https://github.com/dibbhatt/kafka-spark-consumer
Dibyendu
On Tue, Aug 5, 2014 at 12:52 PM, rafeeq s <ra...@gmail.com> wrote:
> Hi,
>
> I am new to Apache Spark and Trying to Develop spark streaming program to *stream
> data from kafka topics and output as parquet file on HDFS*.
>
> Please share the *sample reference* program to stream data from kafka
> topics and output as parquet file on HDFS.
>
> Thanks in Advance.
>
> Regards,
>
> Rafeeq S
> *(“What you do is what matters, not what you think or say or plan.” )*
>
>