You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Cui Lin <cu...@hds.com> on 2015/03/09 19:37:09 UTC
Spark Streaming input data source list
Dear all,
Could you send me a list for input data source that spark streaming could support?
My list is HDFS, Kafka, textfile?…
I am wondering if spark streaming could directly read data from certain port (443 e.g.) that my devices directly send to?
Best regards,
Cui Lin
Re: Spark Streaming input data source list
Posted by Tathagata Das <td...@databricks.com>.
Link to custom receiver guide
https://spark.apache.org/docs/latest/streaming-custom-receivers.html
On Mon, Mar 9, 2015 at 5:55 PM, Shao, Saisai <sa...@intel.com> wrote:
> Hi Lin,
>
>
>
> AFAIK, currently there’s no built-in receiver API for RDBMs, but you can
> customize your own receiver to get data from RDBMs, for the details you can
> refer to the docs.
>
>
>
> Thanks
>
> Jerry
>
>
>
> *From:* Cui Lin [mailto:cui.lin@hds.com]
> *Sent:* Tuesday, March 10, 2015 8:36 AM
> *To:* Tathagata Das
> *Cc:* user@spark.apache.org
> *Subject:* Re: Spark Streaming input data source list
>
>
>
> Tathagata,
>
>
>
> Thanks for your quick response. The link is helpful to me.
>
> Do you know any API for streaming data from RMDB ?
>
>
>
>
>
> Best regards,
>
>
>
> Cui Lin
>
>
>
> *From: *Tathagata Das <td...@databricks.com>
> *Date: *Monday, March 9, 2015 at 11:28 AM
> *To: *Cui Lin <Cu...@hds.com>
> *Cc: *"user@spark.apache.org" <us...@spark.apache.org>
> *Subject: *Re: Spark Streaming input data source list
>
>
>
> Spark Streaming has StreamingContext.socketStream()
>
>
> http://spark.apache.org/docs/1.2.1/api/java/org/apache/spark/streaming/StreamingContext.html#socketStream(java.lang.String,
> int, scala.Function1, org.apache.spark.storage.StorageLevel,
> scala.reflect.ClassTag)
>
>
>
> TD
>
>
>
> On Mon, Mar 9, 2015 at 11:37 AM, Cui Lin <cu...@hds.com> wrote:
>
> Dear all,
>
>
>
> Could you send me a list for input data source that spark streaming could
> support?
>
> My list is HDFS, Kafka, textfile?…
>
>
>
> I am wondering if spark streaming could directly read data from certain
> port (443 e.g.) that my devices directly send to?
>
>
>
>
>
>
>
> Best regards,
>
>
>
> Cui Lin
>
>
>
RE: Spark Streaming input data source list
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Lin,
AFAIK, currently there's no built-in receiver API for RDBMs, but you can customize your own receiver to get data from RDBMs, for the details you can refer to the docs.
Thanks
Jerry
From: Cui Lin [mailto:cui.lin@hds.com]
Sent: Tuesday, March 10, 2015 8:36 AM
To: Tathagata Das
Cc: user@spark.apache.org
Subject: Re: Spark Streaming input data source list
Tathagata,
Thanks for your quick response. The link is helpful to me.
Do you know any API for streaming data from RMDB ?
Best regards,
Cui Lin
From: Tathagata Das <td...@databricks.com>>
Date: Monday, March 9, 2015 at 11:28 AM
To: Cui Lin <Cu...@hds.com>>
Cc: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: Spark Streaming input data source list
Spark Streaming has StreamingContext.socketStream()
http://spark.apache.org/docs/1.2.1/api/java/org/apache/spark/streaming/StreamingContext.html#socketStream(java.lang.String, int, scala.Function1, org.apache.spark.storage.StorageLevel, scala.reflect.ClassTag)
TD
On Mon, Mar 9, 2015 at 11:37 AM, Cui Lin <cu...@hds.com>> wrote:
Dear all,
Could you send me a list for input data source that spark streaming could support?
My list is HDFS, Kafka, textfile?...
I am wondering if spark streaming could directly read data from certain port (443 e.g.) that my devices directly send to?
Best regards,
Cui Lin
Re: Spark Streaming input data source list
Posted by Cui Lin <cu...@hds.com>.
Tathagata,
Thanks for your quick response. The link is helpful to me.
Do you know any API for streaming data from RMDB ?
Best regards,
Cui Lin
From: Tathagata Das <td...@databricks.com>>
Date: Monday, March 9, 2015 at 11:28 AM
To: Cui Lin <Cu...@hds.com>>
Cc: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: Spark Streaming input data source list
Spark Streaming has StreamingContext.socketStream()
http://spark.apache.org/docs/1.2.1/api/java/org/apache/spark/streaming/StreamingContext.html#socketStream(java.lang.String, int, scala.Function1, org.apache.spark.storage.StorageLevel, scala.reflect.ClassTag)
TD
On Mon, Mar 9, 2015 at 11:37 AM, Cui Lin <cu...@hds.com>> wrote:
Dear all,
Could you send me a list for input data source that spark streaming could support?
My list is HDFS, Kafka, textfile?…
I am wondering if spark streaming could directly read data from certain port (443 e.g.) that my devices directly send to?
Best regards,
Cui Lin
Re: Spark Streaming input data source list
Posted by Tathagata Das <td...@databricks.com>.
Spark Streaming has StreamingContext.socketStream()
http://spark.apache.org/docs/1.2.1/api/java/org/apache/spark/streaming/StreamingContext.html#socketStream(java.lang.String,
int, scala.Function1, org.apache.spark.storage.StorageLevel,
scala.reflect.ClassTag)
TD
On Mon, Mar 9, 2015 at 11:37 AM, Cui Lin <cu...@hds.com> wrote:
> Dear all,
>
> Could you send me a list for input data source that spark streaming
> could support?
> My list is HDFS, Kafka, textfile?…
>
> I am wondering if spark streaming could directly read data from certain
> port (443 e.g.) that my devices directly send to?
>
>
>
> Best regards,
>
> Cui Lin
>