MockProducer and MockConsumer

[Laurynas Katkus <la...@mambu.com>]

Hello,

First of all, I just want to express admiration on the job you did guys.
Thank you very much for providing such tool to the world.

To the topic at hand. In our company, we'd like to use Kafka. Currently
we're in testing phase. I'm a big believer in blackbox testing or real unit
testing instead of mocking everything out. That is why I was very happy to
find your provided mock implementation of Consumer and Producer -
MockConsumer and MockProducer.

However, there is lacking documentation about how to use them. I tried
resorting to books like "Kafka The Definitive Guide 2nd edition" and "Kafka
In Action", but both contain very small sections about them. Furhermore, I
haven't found any official documentation from Confluent side as well.
Despite a small blog post in Baeldung[0][1], there isn't anything else.
It's a shame that so much work went into making these implementations, but
no documentation provided. One could argue, that just use the API as it's
self-documenting. Unfortunetally I've encountered weird behavour when
trying to use them. For example, using MockProducer is pretty
straightfoward:

```java
@Test
void givenKeyValue_whenSend_thenVerifyHistory() {

MockProducer mockProducer = new MockProducer<>(true, new StringSerializer(),
new StringSerializer());

kafkaProducer = new KafkaProducer(mockProducer);
Future<RecordMetadata> recordMetadataFuture = kafkaProducer.send("soccer",
"{\"site\" : \"baeldung\"}");

assertTrue(mockProducer.history().size() == 1);
}
```

```java
@Test
void givenKeyValue_whenSend_thenReturnException() {
MockProducer<String, String> mockProducer = new MockProducer<>(false, new
StringSerializer(), new StringSerializer())

kafkaProducer = new KafkaProducer(mockProducer);
Future<RecordMetadata> record = kafkaProducer.send("site", "{\"site\" : \"
baeldung\"}");
mockProducer.errorNext(new RuntimeException());

try {
record.get();
} catch (ExecutionException | InterruptedException ex) {
assertEquals(e, ex.getCause());
}
assertTrue(record.isDone());
}
```

When it comes to MockConsumer, it's unclear how to use it:

```java
private static final MockConsumer<String, String> mockConsumer = new
MockConsumer<>(OffsetResetStrategy.EARLIEST);
...
@Test
void test() {
String hello = "hello";
mockConsumer.schedulePollTask(() -> {
mockConsumer.addRecord(new ConsumerRecord<>(hello, 0, 0, "key", "value"));
});
mockConsumer.subscribe(Collections.singleton(hello));

ConsumerRecords<String, String> poll = mockConsumer.poll(Duration.ofMillis(
100));
}
```

Throws "Cannot add records for a partition that is not assigned to the
consumer". To make it work, I need this boilerplate:

```java
private static final MockConsumer<String, String> mockConsumer = new
MockConsumer<>(OffsetResetStrategy.EARLIEST);
...
@Test
void name3() {
String hello = "hello";
mockConsumer.schedulePollTask(() -> {
mockConsumer.rebalance(Collections.singletonList(new TopicPartition(hello, 0
)));
mockConsumer.addRecord(new ConsumerRecord<>(hello, 0, 0, "key", "value"));
});
mockConsumer.subscribe(Collections.singleton(hello));

HashMap<TopicPartition, Long> startOffsets = new HashMap<>();
TopicPartition tp = new TopicPartition(hello, 0);
startOffsets.put(tp, 0L);
mockConsumer.updateBeginningOffsets(startOffsets);

ConsumerRecords<String, String> poll = mockConsumer.poll(Duration.ofMillis(
100));
}
```

There are parts like making consumer to rebalance. It feels like I need to
do "broker's work". Why it cannot rebalance itself depending on how many
consumers are subscribe to the topic? Another question, I need to manually
update offset. Otherwise it doesn't work either. This is not intuitive.

My question is:
- Will you expand on MockConsumer and MockProducer? Or is this abandoned
initiative/good enough for internal usage and others are best using tools
like Mockito?
- If you're still looking to develop it, will you add documentation? Do you
see this as requirement?
- If you're going to add documentation, maybe you have some drafts or
timelines?

Thank you for reading and reply.


Links:
[0] https://www.baeldung.com/kafka-mockconsumer
[1] https://www.baeldung.com/kafka-mockproducer
Laurynas Katkus
Senior Developer