You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by saulshanabrook <s....@gmail.com> on 2017/05/06 18:31:33 UTC

Re: Join streams Apache Spark

I  wrote a server in Go
<https://gist.github.com/saulshanabrook/ce65baf5d82460b655e1232b9fe796d3>  
that allows many TCP connections for incoming data on one port, writing each
line to the client listening on another port. The  environmental variable
set's what port client's should connect to, to send data to Spark (the
sensors in your case) and the  sets the port that Spark should connect to,
to listen for data.

If anyone knows a simpler way of doing this, by using some existing
software, I would love to know about it.

If you are interested in this code, I would be happy to clean it up and
release it with some documentation.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28655.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Join streams Apache Spark

Posted by saulshanabrook <s....@gmail.com>.
I, actually, just ran it in a Docker image. But the point is, it doesn't
need to run in the JVM, because it just runs as a separate process. Then
your Java (or any other client) code sends messages to it over TCP and it
relays them to Spark.

On Mon, May 8, 2017 at 4:07 AM tencas [via Apache Spark User List] <
ml+s1001560n28662h87@n3.nabble.com> wrote:

> Yep, I mean the first script you posted. So, you can compile it to Java
> binaries for example ? Ok, I have no idea about Go.
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28662.html
> To unsubscribe from Join streams Apache Spark, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=28603&code=cy5zaGFuYWJyb29rQGdtYWlsLmNvbXwyODYwM3wtMTIyOTI1NzM1Ng==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28665.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Join streams Apache Spark

Posted by tencas <di...@gmail.com>.
Yep, I mean the first script you posted. So, you can compile it to Java
binaries for example ? Ok, I have no idea about Go.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28662.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Join streams Apache Spark

Posted by saulshanabrook <s....@gmail.com>.
The script I wrote in Go? No I do not, but it's very easy to compile it to
whatever platform you are running on! Doesn't need to be integrated in the
same language as the rest of your code.

On Sat, May 6, 2017 at 3:13 PM tencas [via Apache Spark User List] <
ml+s1001560n28658h63@n3.nabble.com> wrote:

> There exists an Spark Streaming example of the classic word count, using
> apache kafka connector:
>
>
> https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCount.java
>
> (maybe you already know)
>
> The point is, what are the benefits from using Kafka, instead of a lighter
> solution like yours. Maybe anybody could help us. Anyway, when I try it
> out, I'll give you feedback.
>
> On the other hand, have you got ,by any chance, the same script written on
> Scala, Phyton or Java ?
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28658.html
> To unsubscribe from Join streams Apache Spark, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=28603&code=cy5zaGFuYWJyb29rQGdtYWlsLmNvbXwyODYwM3wtMTIyOTI1NzM1Ng==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28659.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Join streams Apache Spark

Posted by tencas <di...@gmail.com>.
There exists an Spark Streaming example of the classic word count, using
apache kafka connector:

https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCount.java

(maybe you already know)

The point is, what are the benefits from using Kafka, instead of a lighter
solution like yours. Maybe anybody could help us. Anyway, when I try it out,
I'll give you feedback.

On the other hand, have you got ,by any chance, the same script written on
Scala, Phyton or Java ?





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28658.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Join streams Apache Spark

Posted by saulshanabrook <s....@gmail.com>.
Would love to hear if you try it out. I was also considering that. I
recently changed to using the file based streaming input. I made another Go
script
<https://github.com/saulshanabrook/ici.recorder/blob/fd8110e490691cc9e98dce1fefbddba973c29deb/server/main.go>
that let's me connect over TCP and writes each newline it receives to a new
file in a folder. Then Spark can read them from that folder.

On Sat, May 6, 2017 at 2:38 PM tencas [via Apache Spark User List] <
ml+s1001560n28656h1@n3.nabble.com> wrote:

> Thanks @saulshanabrook, I'll have a look at it.
>
> I think apache kafka could be an alternative solution, but I haven't
> checked it yet.
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28656.html
> To unsubscribe from Join streams Apache Spark, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=28603&code=cy5zaGFuYWJyb29rQGdtYWlsLmNvbXwyODYwM3wtMTIyOTI1NzM1Ng==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28657.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Join streams Apache Spark

Posted by tencas <di...@gmail.com>.
Thanks @saulshanabrook, I'll have a look at it.

I think apache kafka could be an alternative solution, but I haven't checked
it yet.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-streams-Apache-Spark-tp28603p28656.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org