[jira] [Comment Edited] (KAFKA-3625) Move kafka-streams test fixtures into a published package

["Scott Davis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320605#comment-16320605 ] 

Scott Davis edited comment on KAFKA-3625 at 1/10/18 4:45 PM:
-------------------------------------------------------------

FYI I just setup some unit tests using the (not documented) internal test classes in Kafka Streams, and I thought I'd share some thoughts on my experience with it.

First, it's super-helpful! The high-level DSL style of Kafka Streams applications doesn't fit well into standard unit testing frameworks, and this solves that problem. It's also helpful compared with integration testing in the sense that it isn't necessary to produce source messages on a broker, wait for timeouts, etc.

The internal test classes in 1.0.0 require the unit tests to provide serializers. This can create some extra boilerplate work to configure the serializers (especially the Confluent AVRO Serde, which requires a schema registry). However, I only need to unit test my application logic (i.e. the contents of the "filter" and "map" methods, etc). I can see how testing serializers is a requirement for testing Kafka Streams internally, but it isn't a requirement for testing the logic of Kafka Streams applications. Note: To work around this, I created a "JavaObjectSerde", which uses java.io.Object(In|Out)putStream, for use by the unit tests.

I needed about 10 lines of boilerplate code in an @Before method to setup the ProcessorTopologyTestDriver, which seemed slightly excessive but not burdensome. Most of it was to create the serializers and the StreamsConfig, which aren't part of my application logic but were required to create the test driver.

I was able to get it to work by adding {{org.apache.kafka:kafka-streams:1.0.0:test}} to my build.gradle, so it seems there already is an artifact. However, there is no documentation, which seems to me like the biggest drawback. I learned it by reading the Kafka Streams source code.


was (Author: scott.davis):
FYI I just setup some unit tests using the (not documented) internal test classes in Kafka Streams, and I thought I'd share some thoughts on my experience with it.

First, it's super-helpful! The high-level DSL style of Kafka Streams applications doesn't fit well into standard unit testing frameworks, and this solves that problem. It's also helpful compared with integration testing in the sense that it isn't necessary to produce source messages on a broker, wait for timeouts, etc.

The internal test classes in 1.0.0 require the unit tests to provide serializers. This can create some extra boilerplate work to configure the serializers (especially the Confluent AVRO Serde, which requires a schema registry). However, I only need to unit test my application logic (i.e. the contents of the "filter" and "map" methods, etc). I can see how testing serializers is a requirement for testing Kafka Streams internally, but it isn't a requirement for testing the logic of Kafka Streams applications. Note: To work around this, I created a "JavaObjectSerde", which uses java.io.Object(In|Out)putStream, for use by the unit tests.

I needed about 10 lines of boilerplate code in an @Before method to setup the ProcessorTopologyTestDriver, which seemed slightly excessive but not burdensome. Most of it was to create the serializers and the StreamsConfig, which aren't part of my application logic but were required to create the test driver.


> Move kafka-streams test fixtures into a published package
> ---------------------------------------------------------
>
>                 Key: KAFKA-3625
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3625
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Jeff Klukas
>            Assignee: Matthias J. Sax
>            Priority: Minor
>              Labels: needs-kip, user-experience
>
> The KStreamTestDriver and related fixtures defined in streams/src/test/java/org/apache/kafka/test would be useful to developers building applications on top of Kafka Streams, but they are not currently exposed in a package.
> I propose moving this directory to live under streams/fixtures/src/main and creating a new 'streams:fixtures' project in the gradle configuration to publish these as a separate package.
> KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)