You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "nzw921rx (via GitHub)" <gi...@apache.org> on 2026/07/15 12:18:23 UTC
[PR] [Fix][E2E] Fix flaky Kafka latest and partition tests [seatunnel]
nzw921rx opened a new pull request, #11442:
URL: https://github.com/apache/seatunnel/pull/11442
### Purpose of this pull request
Fix flaky tests in the Kafka E2E module by making startup and partition-discovery assertions wait for actual consumption readiness instead of assuming the source is ready immediately after the job reaches RUNNING.
**Root cause:**
The Kafka CI job intermittently fails with `ConditionTimeout` in tests that use `start_mode = latest`:
```
KafkaIT.testSourceKafkaRestoreWithLatestMode
KafkaIT.testDynamicPartitionDiscovery
```
For latest-mode restore, records produced immediately after `awaitJobRunning` can be skipped if the Kafka source has not finished subscribing and seeking yet.
For dynamic partition discovery, records sent to the new partition immediately after partition creation can be skipped before SeaTunnel discovers the new partition and starts consuming it.
**Changes in `KafkaIT`:**
- Add a probe record loop before the latest-mode assertions to confirm the running job is consuming newly produced records
- Count first-run and restore records by their payload instead of relying on the full sink partition offset count
- Add a probe record loop after increasing Kafka partitions to confirm SeaTunnel has discovered and consumed the new partition
- Assert the dynamic-partition test against `new_partition_message_` records only, so probe records do not affect the result
### Does this PR introduce _any_ user-facing change?
No. This change only affects E2E test stability.
### How was this patch tested?
Ran:
```
./mvnw spotless:apply
```
Per the flaky-CI fix workflow, no E2E tests were run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][E2E] Fix flaky Kafka latest and partition tests [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11442:
URL: https://github.com/apache/seatunnel/pull/11442#issuecomment-4981495188
Closing this PR to reduce the number of CI-fix pull requests. I will consolidate related fixes by module or root cause and reopen a smaller set of PRs for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][E2E] Fix flaky Kafka latest and partition tests [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx closed pull request #11442: [Fix][E2E] Fix flaky Kafka latest and partition tests
URL: https://github.com/apache/seatunnel/pull/11442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org