Re: Question about E2E tests for pipelines

[Brian Hulette <bh...@google.com>]

KafkaIOIT can also use testcontainers to create a fake Kafka service [1].
In theory we could do something similar to test PubSubIO by using the
provided emulator [2], but currently we only test against the production
pubsub service. There's a useful TestPubsub [3] rule which can be used to
create a test PubSub topic that you can read or write to and make
assertions against. Jenkins has permissions to use the apache-beam-testing
project to create these topics on GCP.

[1]
https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java#L336
[2] https://cloud.google.com/pubsub/docs/emulator
[3]
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java

On Thu, Nov 26, 2020 at 9:37 AM Artur Khanin <ar...@akvelon.com>
wrote:

> Thank you for the information and links, Alexey! We will try to follow
> this approach.
>
> On 25 Nov 2020, at 21:27, Alexey Romanenko <ar...@gmail.com>
> wrote:
>
> For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It
> leverages a real Kafka cluster that runs on k8s. So, probably you can
> follow the similar approach.
>
> In the same time, we fake Kafka consumer and its output for KafkaIO unit
> tests.
>
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
> [2]
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
>
>
> On 25 Nov 2020, at 13:05, Artur Khanin <ar...@akvelon.com> wrote:
>
> Hi Devs,
>
> We are finalizing this PR <https://github.com/apache/beam/pull/13112> with
> a pipeline that reads from Kafka and writes to Pub/Sub without any
> transformations in between. We would like to implement e2e tests where we
> create and execute a pipeline, but we haven't found much information and
> relevant examples about it.How exactly should we implement such kind of
> tests? Can we mock somehow Kafka and Pub/Sub or maybe can we set them up
> using some test environment?
>
> Any information and hints will be greatly appreciated!
>
> Thanks,
> Artur Khanin
> Akvelon, Inc
>
>
>
>