You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by foobar <he...@fb.com> on 2015/06/12 04:25:53 UTC
Spark Streaming reads from stdin or output from command line
utility
Hi, I'm new to Spark Streaming, and I want to create a application where
Spark Streaming could create DStream from stdin. Basically I have a command
line utility that generates stream data, and I'd like to pipe data into
DStream. What's the best way to do that? I thought rdd.pipe() could help,
but it seems that requires an rdd in the first place, which does not apply.
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Streaming reads from stdin or output from command line utility
Posted by Gerard Maas <ge...@gmail.com>.
Would using the socketTextStream and `yourApp | nc -lk <port>` work?? Not
sure how resilient the socket receiver is though. I've been playing with it
for a little demo and I don't understand yet its reconnection behavior.
Although I would think that putting some elastic buffer in between would be
a good idea to decouple producer from consumer. Kafka would be my first
choice.
-kr, Gerard.
On Fri, Jun 12, 2015 at 8:46 AM, Heath Guo <he...@fb.com> wrote:
> Yes, it is lots of data, and the utility I'm working with prints out
> infinite real time data stream. Thanks.
>
>
> From: Tathagata Das <td...@databricks.com>
> Date: Thursday, June 11, 2015 at 11:43 PM
>
> To: Heath Guo <he...@fb.com>
> Cc: user <us...@spark.apache.org>
> Subject: Re: Spark Streaming reads from stdin or output from command line
> utility
>
> Is it a lot of data that is expected to come through stdin? I mean is
> it even worth parallelizing the computation using something like Spark
> Streaming?
>
> On Thu, Jun 11, 2015 at 9:56 PM, Heath Guo <he...@fb.com> wrote:
>
>> Thanks for your reply! In my use case, it would be stream from only
>> one stdin. Also I'm working with Scala.
>> It would be great if you could talk about multi stdin case as well!
>> Thanks.
>>
>> From: Tathagata Das <td...@databricks.com>
>> Date: Thursday, June 11, 2015 at 8:11 PM
>> To: Heath Guo <he...@fb.com>
>> Cc: user <us...@spark.apache.org>
>> Subject: Re: Spark Streaming reads from stdin or output from command
>> line utility
>>
>> Are you going to receive data from one stdin from one machine, or
>> many stdins on many machines?
>>
>>
>> On Thu, Jun 11, 2015 at 7:25 PM, foobar <he...@fb.com> wrote:
>>
>>> Hi, I'm new to Spark Streaming, and I want to create a application where
>>> Spark Streaming could create DStream from stdin. Basically I have a
>>> command
>>> line utility that generates stream data, and I'd like to pipe data into
>>> DStream. What's the best way to do that? I thought rdd.pipe() could help,
>>> but it seems that requires an rdd in the first place, which does not
>>> apply.
>>> Thanks!
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html
>>> <https://urldefense.proofpoint.com/v1/url?u=http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=4Z2U8tLm1orBgymimfryIw%3D%3D%0A&m=4O1SseOzl0OsOY1s4%2B3jfsvy21wseYOJS0gxhf1IYc8%3D%0A&s=3df5e3f1e40970c1cb5191b7e3d6c9957c86993d2ac1f2d7fb6b622c7ebeccdd>
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: Spark Streaming reads from stdin or output from command line
utility
Posted by Heath Guo <he...@fb.com>.
Yes, it is lots of data, and the utility I'm working with prints out infinite real time data stream. Thanks.
From: Tathagata Das <td...@databricks.com>>
Date: Thursday, June 11, 2015 at 11:43 PM
To: Heath Guo <he...@fb.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Spark Streaming reads from stdin or output from command line utility
Is it a lot of data that is expected to come through stdin? I mean is it even worth parallelizing the computation using something like Spark Streaming?
On Thu, Jun 11, 2015 at 9:56 PM, Heath Guo <he...@fb.com>> wrote:
Thanks for your reply! In my use case, it would be stream from only one stdin. Also I'm working with Scala.
It would be great if you could talk about multi stdin case as well! Thanks.
From: Tathagata Das <td...@databricks.com>>
Date: Thursday, June 11, 2015 at 8:11 PM
To: Heath Guo <he...@fb.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Spark Streaming reads from stdin or output from command line utility
Are you going to receive data from one stdin from one machine, or many stdins on many machines?
On Thu, Jun 11, 2015 at 7:25 PM, foobar <he...@fb.com>> wrote:
Hi, I'm new to Spark Streaming, and I want to create a application where
Spark Streaming could create DStream from stdin. Basically I have a command
line utility that generates stream data, and I'd like to pipe data into
DStream. What's the best way to do that? I thought rdd.pipe() could help,
but it seems that requires an rdd in the first place, which does not apply.
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html<https://urldefense.proofpoint.com/v1/url?u=http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=4Z2U8tLm1orBgymimfryIw%3D%3D%0A&m=4O1SseOzl0OsOY1s4%2B3jfsvy21wseYOJS0gxhf1IYc8%3D%0A&s=3df5e3f1e40970c1cb5191b7e3d6c9957c86993d2ac1f2d7fb6b622c7ebeccdd>
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
Re: Spark Streaming reads from stdin or output from command line utility
Posted by Tathagata Das <td...@databricks.com>.
Is it a lot of data that is expected to come through stdin? I mean is it
even worth parallelizing the computation using something like Spark
Streaming?
On Thu, Jun 11, 2015 at 9:56 PM, Heath Guo <he...@fb.com> wrote:
> Thanks for your reply! In my use case, it would be stream from only one
> stdin. Also I'm working with Scala.
> It would be great if you could talk about multi stdin case as well!
> Thanks.
>
> From: Tathagata Das <td...@databricks.com>
> Date: Thursday, June 11, 2015 at 8:11 PM
> To: Heath Guo <he...@fb.com>
> Cc: user <us...@spark.apache.org>
> Subject: Re: Spark Streaming reads from stdin or output from command line
> utility
>
> Are you going to receive data from one stdin from one machine, or many
> stdins on many machines?
>
>
> On Thu, Jun 11, 2015 at 7:25 PM, foobar <he...@fb.com> wrote:
>
>> Hi, I'm new to Spark Streaming, and I want to create a application where
>> Spark Streaming could create DStream from stdin. Basically I have a
>> command
>> line utility that generates stream data, and I'd like to pipe data into
>> DStream. What's the best way to do that? I thought rdd.pipe() could help,
>> but it seems that requires an rdd in the first place, which does not
>> apply.
>> Thanks!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html
>> <https://urldefense.proofpoint.com/v1/url?u=http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=4Z2U8tLm1orBgymimfryIw%3D%3D%0A&m=4O1SseOzl0OsOY1s4%2B3jfsvy21wseYOJS0gxhf1IYc8%3D%0A&s=3df5e3f1e40970c1cb5191b7e3d6c9957c86993d2ac1f2d7fb6b622c7ebeccdd>
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Spark Streaming reads from stdin or output from command line
utility
Posted by Heath Guo <he...@fb.com>.
Thanks for your reply! In my use case, it would be stream from only one stdin. Also I'm working with Scala.
It would be great if you could talk about multi stdin case as well! Thanks.
From: Tathagata Das <td...@databricks.com>>
Date: Thursday, June 11, 2015 at 8:11 PM
To: Heath Guo <he...@fb.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Spark Streaming reads from stdin or output from command line utility
Are you going to receive data from one stdin from one machine, or many stdins on many machines?
On Thu, Jun 11, 2015 at 7:25 PM, foobar <he...@fb.com>> wrote:
Hi, I'm new to Spark Streaming, and I want to create a application where
Spark Streaming could create DStream from stdin. Basically I have a command
line utility that generates stream data, and I'd like to pipe data into
DStream. What's the best way to do that? I thought rdd.pipe() could help,
but it seems that requires an rdd in the first place, which does not apply.
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html<https://urldefense.proofpoint.com/v1/url?u=http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=4Z2U8tLm1orBgymimfryIw%3D%3D%0A&m=4O1SseOzl0OsOY1s4%2B3jfsvy21wseYOJS0gxhf1IYc8%3D%0A&s=3df5e3f1e40970c1cb5191b7e3d6c9957c86993d2ac1f2d7fb6b622c7ebeccdd>
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
Re: Spark Streaming reads from stdin or output from command line utility
Posted by Tathagata Das <td...@databricks.com>.
Are you going to receive data from one stdin from one machine, or many
stdins on many machines?
On Thu, Jun 11, 2015 at 7:25 PM, foobar <he...@fb.com> wrote:
> Hi, I'm new to Spark Streaming, and I want to create a application where
> Spark Streaming could create DStream from stdin. Basically I have a command
> line utility that generates stream data, and I'd like to pipe data into
> DStream. What's the best way to do that? I thought rdd.pipe() could help,
> but it seems that requires an rdd in the first place, which does not apply.
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-reads-from-stdin-or-output-from-command-line-utility-tp23289.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>