[D] Implementing Integration E2E Tests for Java Client

["Yuukadesu (via GitHub)" <gi...@apache.org>]

GitHub user Yuukadesu created a discussion: Implementing Integration E2E Tests for Java Client

# Proposal: Implementing Integration E2E Tests for Java Client

## Motivation

To ensure the stability and reliability of the StreamPipes Java client library, this implementation adopts an end-to-end testing strategy aligned with the Go Client E2E. First, a test environment closely resembling production is set up, including the Backend, multiple Extension service instances, Kafka, CouchDB, and other required dependencies, so that test results are authentic and reproducible. Second, test cases are designed to cover two main objectives: **data path correctness** and **load balancing effectiveness**. They verify not only the end-to-end data flow (Kafka topic-in → adapter/source → processor → sink → Kafka topic-out), but also assert under the load of 70 pipelines and 70 adapters that elements do not all land on a single extension instance. To improve test efficiency and consistency, automated tests are written with JUnit 5, and connection parameters are injected via custom startup scripts and system properties. Finally, E2E tests are integrated into GitHub 
 Actions CI: on every PR, images are built, the load-balanced environment is started, and Java Client E2E is executed, enabling continuous monitoring and timely feedback on the Java client and load-balancing behavior, and delivering a reliable and efficient Java client library for StreamPipes users.

## Goals

- **Verify data path**: Ensure that adapters and pipelines created via the Java Client correctly complete the end-to-end flow (Kafka input → adapter → Boolean filter processor → Kafka output), and that filter semantics are validated.
- **Verify load balancing**: At the scale of 70 resource groups (70 adapters + 70 pipelines), ensure that `selectedEndpointUrl` hits at least multiple extension instances (default ≥3), and that no single instance exceeds a configured share threshold (e.g. 40%), so that load is not concentrated on one instance.
- **Early defect detection**: Through automated, repeatable E2E flows and CI integration, find and fix defects related to the Java Client API, Compact Adapter/Pipeline APIs, and load-balancing strategy as early as possible, reducing the cost of production fixes.
- **Alignment with Go E2E style**: Execution flow is aligned with Go Client E2E (cleanup → create → start → verify → cleanup), making it easier to maintain and compare multi-language E2E tests later.

## Implementation

The implementation ensures that the Java E2E test flow is consistent with the Go E2E test flow. A Docker-based environment is used with custom startup scripts that accept host, port, and other configuration parameters. E2E tests are triggered automatically on each PR via a GitHub Actions workflow and a script chain: the workflow builds images, starts the load-balanced stack with `docker-compose-loadbalancer.yml`, then runs `start-streampipes-client-e2e.sh -t java-client-e2e.sh`, which performs login, API key retrieval, and `mvn test` with the appropriate system properties; any failure fails the workflow so that code quality and load-balancing behavior are continuously monitored.


GitHub link: https://github.com/apache/streampipes/discussions/4216

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Implementing Integration E2E Tests for Java Client

["tenthe (via GitHub)" <gi...@apache.org>]

GitHub user tenthe added a comment to the discussion: Implementing Integration E2E Tests for Java Client

Hi @Yuukadesu , 
thanks you very much for stariting for the discussion and for putting together the proposal.

I had one thought regarding the tests. If I understand it correctly, the proposal currently tries to test **both the Java client itself and the load-balancing behavior** in the same E2E setup. Would it maybe make sense to **split this into two separate tests**?

For example:

* one **simple E2E test for the Java client**, mainly verifying that creating adapters/pipelines via the client works and that the data path (including the filter semantics) behaves correctly
* a **separate test focused on load balancing**, where we specifically check the distribution of adapters and pipeline elements across the extension instances

That way the client test stays relatively simple and failures might be easier to interpret. 

Also one small question about the broker: since **NATS is the default broker since the last release**, would it make sense to **use NATS instead of Kafka in the test setup**?

These are just some thoughts I am happy to hear your opinion on this.


GitHub link: https://github.com/apache/streampipes/discussions/4216#discussioncomment-16008836

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Implementing Integration E2E Tests for Java Client

["Yuukadesu (via GitHub)" <gi...@apache.org>]

GitHub user Yuukadesu edited a discussion: Implementing Integration E2E Tests for Java Client

# Proposal: Implementing Integration E2E Tests for Java Client

## Motivation

To ensure the stability and reliability of the StreamPipes Java client library, this implementation adopts an end-to-end testing strategy aligned with the Go Client E2E. First, a test environment closely resembling production is set up (a load-balanced topology based on Docker Compose), including the Backend, multiple Extension service instances, Kafka, CouchDB, and other required dependencies, so that test results are authentic and reproducible. Second, test cases are designed to cover two main objectives: **data path correctness** and **load balancing effectiveness**. They verify not only the end-to-end data flow (Kafka topic-in → adapter/source → processor → sink → Kafka topic-out), but also assert under the load of 70 pipelines and 70 adapters that **adapters** and **pipeline elements** are each distributed across multiple extension instances, with each instance’s share roughly equal (within a configurable deviation, default 20% from the average). To improve test effici
 ency and consistency, automated tests are written with JUnit 5, and connection parameters are injected via custom startup scripts and system properties. Finally, E2E tests are integrated into GitHub Actions CI: on every PR, images are built, the load-balanced environment is started, and Java Client E2E is executed, enabling continuous monitoring and timely feedback on the Java client and load-balancing behavior, and delivering a reliable and efficient Java client library for StreamPipes users.

## Goals

- **Verify data path**: Ensure that adapters and pipelines created via the Java Client correctly complete the end-to-end flow (Kafka input → adapter → Boolean filter processor → Kafka output), and that filter semantics are validated (e.g. only messages with `sensor_fault_flags == false` are passed through).
- **Verify load balancing**: The real balancer equalizes **instance load** (CPU, memory, etc.), not adapter/pipeline counts. In the test, under **uniform workload** (same adapter and pipeline type for all 70), equal **count** per instance approximates equal **load** per instance. Verify separately for adapters and pipeline elements that: distinct instances used ≥ configured minimum (default 3); and each instance’s count is roughly equal (within default 20% of the average).
- **Early defect detection**: Through automated, repeatable E2E flows and CI integration, find and fix defects related to the Java Client API, Compact Adapter/Pipeline APIs, and load-balancing strategy as early as possible, reducing the cost of production fixes.
- **Alignment with Go E2E style**: Execution flow is aligned with Go Client E2E (cleanup → create → start → verify → cleanup), making it easier to maintain and compare multi-language E2E tests later.

## Implementation

The implementation ensures that the Java E2E test flow is consistent with the Go E2E test flow. A Docker-based environment is used with custom startup scripts that accept host, port, and other configuration parameters. E2E tests are triggered automatically on each PR via a GitHub Actions workflow and a script chain: the workflow builds images, starts the load-balanced stack with `docker-compose-loadbalancer.yml`, then runs `start-streampipes-client-e2e.sh -t java-client-e2e.sh`, which performs login, API key retrieval, and `mvn test` with the appropriate system properties; any failure fails the workflow so that code quality and load-balancing behavior are continuously monitored.

GitHub link: https://github.com/apache/streampipes/discussions/4216

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Implementing Integration E2E Tests for Java Client

["Yuukadesu (via GitHub)" <gi...@apache.org>]

GitHub user Yuukadesu added a comment to the discussion: Implementing Integration E2E Tests for Java Client

Hi! @tenthe ,
thank you very much for your suggestions ! You put forward a very good idea, which will make E2E test more easier and failures easier to interpret.
Next, I will build simple E2E tests to test the Java client and load balancing respectively. In addition, I will use NATS instead of Kafka to make the E2E test more suitable for Streampipes.

GitHub link: https://github.com/apache/streampipes/discussions/4216#discussioncomment-16017331

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org