You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Cameron Lee (Jira)" <ji...@apache.org> on 2019/08/20 22:11:00 UTC
[jira] [Created] (SAMZA-2307) Migrate tests in samza-test to use
in-memory system instead of Kafka/Zookeeper
Cameron Lee created SAMZA-2307:
----------------------------------
Summary: Migrate tests in samza-test to use in-memory system instead of Kafka/Zookeeper
Key: SAMZA-2307
URL: https://issues.apache.org/jira/browse/SAMZA-2307
Project: Samza
Issue Type: Improvement
Reporter: Cameron Lee
Assignee: Cameron Lee
Many tests in samza-test use the IntegrationTestHarness which sets up local instances of Kafka and Zookeeper. However, these systems take time to start up. Also, using Kafka as a system is slower since there are inter-process calls.
If we can replace the Kafka usage with an in-memory system, then we don't have to start up Kafka/Zookeeper, and test runs should be significantly faster.
Some tests that could possibly be updated (not a complete list). I have added some notes about the current state of some of the tests. For example, the tests which directly use Kafka would need changes to produce to the in-memory system instead.
- FaultInjectionTest: directly produces to kafka
- WatermarkIntegrationTest: need special runner to inspect processors
- EndOfStreamIntegrationTest
- TestStreamProcessor: directly depends on kafka
- TestZkLocalApplicationRunner: tied to publishing kafka events
- TestRepartitionJoinWindowApp: tied to KafkaSystemFactory for checking delete messages
- StreamApplicationIntegrationTestHarness: need to create input topics with correct partition counts
- TestLocalTable*: already uses some system that is in-memory-like; uses kafka for default systems
- update IntegrationTestHarness.createTopic to use in-memory system instead of kafka?
--
This message was sent by Atlassian Jira
(v8.3.2#803003)