streaming: code to simulate a network socket data source

[Diana Carroll <dc...@cloudera.com>]

If you are learning about Spark Streaming, as I am, you've probably use
netcat "nc" as mentioned in the spark streaming programming guide.  I
wanted something a little more useful, so I modified the
ClickStreamGenerator code to make a very simple script that simply reads a
file off disk and passes it to a socket, character by character.  You
specify the port, filename, and bytesPerSecond that you want it to send.

Thought someone else might find this helpful, so here it is.

import java.net.ServerSocket
import java.io.PrintWriter
import scala.io.Source

object StreamingDataGenerator {

  def main(args : Array[String]) {
    if (args.length != 3) {
      System.err.println("Usage: StreamingDataGenerator <port> <file>
<bytesPerSecond>")
      System.exit(1)
    }
    val port = args(0).toInt
    val file = Source.fromFile(args(1))
    val bytesPerSecond = args(2).toFloat

    val sleepDelayMs = (1000.0 / bytesPerSecond).toInt
    val listener = new ServerSocket(port)

    println("Reading from file: " + file.descr)

    while (true) {
      println("Listening on port: " + port)
      val socket = listener.accept()
      new Thread() {
        override def run = {
          println("Got client connect from: " + socket.getInetAddress)
          val out = new PrintWriter(socket.getOutputStream(), true)

          file.foreach(c =>
            {
              Thread.sleep(sleepDelayMs)
              // write the byte to the socket
              out.write(c)
              out.flush()
              // also print the byte to stdout, for debugging ease
              print(c)
            }
          )
          socket.close()
        }
      }.start()
    }
  }
}

Re: streaming: code to simulate a network socket data source

[danilopds <da...@gmail.com>]

I utilize this code in separated but the program block in this character:
val socket = listener.accept()

Do you have any suggestion?
Thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/streaming-code-to-simulate-a-network-socket-data-source-tp3431p13817.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: streaming: code to simulate a network socket data source

[danilopds <da...@gmail.com>]

Hello Diana,

How can I include this implementation in my code, in terms of start this
task together the NetworkWordCount.

In my case, I have a directory with several files.

Then,
I include this line:
  StreamingDataGenerator.streamingGenerator(NetPort, BytesSecond, DirFiles)

But the program stays in my loop of files.
And after returning to NetworkWordCount.

Can you suggest me something to start these tasks in parallel?
Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/streaming-code-to-simulate-a-network-socket-data-source-tp3431p13814.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org