[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming

["Anuj Ojha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659014#comment-14659014 ] 

Anuj Ojha commented on SPARK-1022:
----------------------------------

[~tdas] and [~jerryshao2015] I was working on writing Integration test for my application, which reads data from kafka, performs transformation on the data and then writes the Dstream back to other kafka topic. To do this I had to get KafkaTestUtils and some other dependent classes locally as I needed embedded zookeeper and kafa queue. I also had to modify sendMessage to accept my specific type of ket and value. I was wondering if it is possible to make the KafkaTestUtils generic? so that others can use it for in-memory  unit/integration testing. I have created generic sendMessage method hopefully it's of some use. 
{code}
class KafkaTestUtils [T,U]{
.......
  // Kafka producer
  private var producer: Producer[T, U] = _

..........

  def sendMessages(topic: String, messageToFreq: JMap[T, U]): Unit = {
    import scala.collection.JavaConversions._
    producer = new Producer[T, U](new ProducerConfig(producerConfiguration))
    for ((k,v) <- messageToFreq) {
      var message = new KeyedMessage(topic, k,v)
      producer.send(message)
    }
    producer.close()
    producer = null
  }

.........
}
{code}

I also had to change producer properties to provide key and value serialization class. It might be worth making possible to set these values from user test. I did not create JIRA for the request as I wanted to see if this is even a acceptable request. Please let me know.

> Add unit tests for kafka streaming
> ----------------------------------
>
>                 Key: SPARK-1022
>                 URL: https://issues.apache.org/jira/browse/SPARK-1022
>             Project: Spark
>          Issue Type: Bug
>            Reporter: Patrick Wendell
>            Assignee: Saisai Shao
>
> It would be nice if we could add unit tests to verify elements of kafka's stream. Right now we do integration tests only which makes it hard to upgrade versions of kafka. The place to start here would be to look at how kafka tests itself and see if the functionality can be exposed to third party users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org