You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Cameron Lee (Jira)" <ji...@apache.org> on 2019/08/20 22:11:00 UTC

[jira] [Created] (SAMZA-2307) Migrate tests in samza-test to use in-memory system instead of Kafka/Zookeeper

Cameron Lee created SAMZA-2307:
----------------------------------

             Summary: Migrate tests in samza-test to use in-memory system instead of Kafka/Zookeeper
                 Key: SAMZA-2307
                 URL: https://issues.apache.org/jira/browse/SAMZA-2307
             Project: Samza
          Issue Type: Improvement
            Reporter: Cameron Lee
            Assignee: Cameron Lee


Many tests in samza-test use the IntegrationTestHarness which sets up local instances of Kafka and Zookeeper. However, these systems take time to start up. Also, using Kafka as a system is slower since there are inter-process calls.

If we can replace the Kafka usage with an in-memory system, then we don't have to start up Kafka/Zookeeper, and test runs should be significantly faster.

Some tests that could possibly be updated (not a complete list). I have added some notes about the current state of some of the tests. For example, the tests which directly use Kafka would need changes to produce to the in-memory system instead.

- FaultInjectionTest: directly produces to kafka
 - WatermarkIntegrationTest: need special runner to inspect processors
 - EndOfStreamIntegrationTest
 - TestStreamProcessor: directly depends on kafka
 - TestZkLocalApplicationRunner: tied to publishing kafka events
 - TestRepartitionJoinWindowApp: tied to KafkaSystemFactory for checking delete messages
 - StreamApplicationIntegrationTestHarness: need to create input topics with correct partition counts
 - TestLocalTable*: already uses some system that is in-memory-like; uses kafka for default systems
 - update IntegrationTestHarness.createTopic to use in-memory system instead of kafka?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)