You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org> on 2026/08/18 13:33:44 UTC

[I] [Bug][CI] unit-test matrix omits fail-fast: false, so one flaky leg cancels the other three [seatunnel]

SEPURI-SAI-KRISHNA opened a new issue, #11854:
URL: https://github.com/apache/seatunnel/issues/11854

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   ### What happened
   
   The `unit-test` job in `.github/workflows/backend.yml` runs a 2x2 matrix (`java: ['8', '11']` x `os: ['ubuntu-latest', 'windows-latest']`) but does not set `fail-fast: false`. GitHub Actions defaults `strategy.fail-fast` to `true`, so as soon as any one of the four legs fails, the other three are cancelled mid-run.
   
   The consequence is that a failure on one platform destroys the cross-platform signal the matrix exists to produce. A reviewer sees one red leg and three `cancelled` legs, and cannot tell from the check list whether the other three would have passed — which is the single most useful fact when triaging, because it is exactly what separates a platform-specific flake from a real regression. Recovering it currently requires downloading and reading the raw job log by hand, or rerunning the whole matrix.
   
   The `benchmark-test` job, 30 lines further down the same file, does set `fail-fast: false`. Since the adjacent job makes the setting explicit, the omission on `unit-test` looks like an oversight rather than a deliberate cost decision.
   
   **`.github/workflows/backend.yml:476-484`** — no `fail-fast`:
   
   ```yaml
     unit-test:
       needs: [ changes, sanity-check ]
       if: needs.changes.outputs.api == 'true' || (needs.changes.outputs.api == 'false' && needs.changes.outputs.ut-modules != '')
       runs-on: ${{ matrix.os }}
       strategy:
         matrix:
           java: [ '8', '11' ]
           os: [ 'ubuntu-latest', 'windows-latest' ]
       timeout-minutes: 90
   ```
   
   **`.github/workflows/backend.yml:506-515`** — `benchmark-test` sets it:
   
   ```yaml
     benchmark-test:
       needs: [ changes, sanity-check ]
       if: needs.changes.outputs.benchmarks == 'true'
       runs-on: ${{ matrix.os }}
       timeout-minutes: 60
       strategy:
         fail-fast: false
         matrix:
           java: [ '8', '11' ]
           os: [ 'ubuntu-latest' ]
   ```
   
   ### Observed impact
   
   Two runs from 2026-08-18, on two unrelated PRs. Each shows 1 failure and 3 cancellations, and in both cases the one failure was environmental — so all six cancelled legs were discarded for nothing:
   
   | PR | Run | Failing leg | Actual cause | Cancelled legs |
   |---|---|---|---|---|
   | #11724 | [`32101324858`](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/32101324858/attempts/1) | [`unit-test (11, windows-latest)`](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/32101324858/job/95634437084) | `AbstractSeaTunnelServerTest.before` → `java.lang.IllegalStateException: Node failed to start!` — Hazelcast node bootstrap in `seatunnel-engine-server`, windows-only | 3 |
   | #11721 | [`32100301798`](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/32100301798/attempts/1) | [`unit-test (8, ubuntu-latest)`](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/32100301798/job/95601487349) | `Could not transfer artifact com.clickhouse:clickhouse-client:jar:0.3.2-patch11 from/to central: Connection reset` — dependency download; **zero test failures in the entire 18,544-line log** | 3 |
   
   The #11721 case is the sharpest illustration. The reactor contains exactly one `FAILURE` line:
   
   ```
   [INFO] SeaTunnel : Connectors V2 : Clickhouse ............. FAILURE [  1.351 s]
   [ERROR] Failed to execute goal on project connector-clickhouse: Could not resolve
           dependencies ... transfer failed for .../clickhouse-client-0.3.2-patch11.jar:
           Connection reset -> [Help 1]
   ```
   
   A transient TCP reset against Maven Central, 1.35 seconds into a module unrelated to the change under test. Everything before it in the reactor passed. Three legs were cancelled because of that.
   
   Both PRs then required a manual log pull to establish something the check list should have shown directly: that the failure was not in the changed code.
   
   The run links above are pinned to attempt 1, since rerunning the failed legs creates a new attempt and the default run view then shows that instead.
   
   ### Relationship to #11513
   
   This is a narrow, concrete instance of the feedback-quality problem #11513 describes, and I am filing it separately because it is a one-line fix that does not depend on that redesign landing.
   
   There is a real tension worth naming rather than glossing over. #11513 proposes to *"cancel queued/running expensive jobs when a required fast gate fails"* — i.e. it wants **more** cancellation, not less. I do not think that conflicts with this request, because the two are on different axes:
   
   - Cancelling **downstream expensive jobs** after a gate has already failed saves runner minutes and loses no information, since the gate result already tells you what you needed.
   - Cancelling **peer legs of the same matrix** destroys information, because each leg is testing a different platform and no other leg answers for it.
   
   Fail-fast on `unit-test` is the second kind. If #11513's redesign lands with a fast gate in front, this setting stays correct — the gate would cancel the expensive IT shards, while the four unit legs still finish and still report per-platform.
   
   (Minor: #11513's source link for the unit matrix points at `backend.yml#L380-L400`; the block has since moved to `L476-L484`.)
   
   ### Proposed fix
   
   One line:
   
   ```yaml
     unit-test:
       ...
       strategy:
         fail-fast: false          # <-- add
         matrix:
           java: [ '8', '11' ]
           os: [ 'ubuntu-latest', 'windows-latest' ]
   ```
   
   **On cost.** When all four legs are green — the common case — nothing changes, because fail-fast never triggers. Runner minutes increase only in the failing case, by at most three legs. Against that, recovering the same information today requires rerunning the full matrix, which costs strictly more than letting a run that is already in flight finish once. `benchmark-test` already accepts this tradeoff on a smaller matrix.
   
   If the added minutes are still judged too expensive, the alternative levers are `max-parallel` or trimming the matrix — but both change what gets tested, whereas `fail-fast: false` only stops discarding results that have already been paid for.
   
   ### SeaTunnel Version
   
   dev (`3.0.0-SNAPSHOT`)
   
   ### SeaTunnel Config
   
   ```conf
   N/A — this is a CI workflow configuration issue, not a job-runtime issue.
   ```
   
   ### Running Command
   
   ```shell
   N/A — reproduced by any PR whose unit-test matrix has one leg fail; see the two runs linked above.
   ```
   
   ### Error Exception
   
   ```log
   [INFO] SeaTunnel : Connectors V2 : Clickhouse ............. FAILURE [  1.351 s]
   [INFO] BUILD FAILURE
   [INFO] Total time:  13:42 min
   [ERROR] Failed to execute goal on project connector-clickhouse: Could not resolve dependencies
   for project org.apache.seatunnel:connector-clickhouse:jar:3.0.0-SNAPSHOT: Could not transfer
   artifact com.clickhouse:clickhouse-client:jar:0.3.2-patch11 from/to central
   (https://repo.maven.apache.org/maven2): transfer failed for
   https://repo.maven.apache.org/maven2/com/clickhouse/clickhouse-client/0.3.2-patch11/clickhouse-client-0.3.2-patch11.jar:
   Connection reset -> [Help 1]
   ```
   
   ### Zeta or Flink or Spark Version
   
   N/A
   
   ### Java or Scala Version
   
   JDK 8 and JDK 11 (both matrix legs affected)
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][CI] unit-test matrix omits fail-fast: false, so one flaky leg cancels the other three [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #11854: [Bug][CI] unit-test matrix omits fail-fast: false, so one flaky leg cancels the other three
URL: https://github.com/apache/seatunnel/issues/11854


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][CI] unit-test matrix omits fail-fast: false, so one flaky leg cancels the other three [seatunnel]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on issue #11854:
URL: https://github.com/apache/seatunnel/issues/11854#issuecomment-5329521803

   Follow-up with a correction to my own issue: I scoped this to `unit-test`, and that understates the problem considerably. I audited every matrix job in `backend.yml` and **43 of 44 omit `fail-fast: false`** — `benchmark-test` is the only one that sets it.
   
   ```
   unit-test                                  fail-fast: NO   (matrix at :482)
   benchmark-test                             fail-fast: YES  (:514)
   updated-modules-integration-test-part-1..8 fail-fast: NO   (:548 .. :762)
   engine-v2-it / edge-agent-it / engine-k8s-it        NO     (:793, :818, :843)
   transform-v2-it-part-1..2                           NO     (:880, :908)
   all-connectors-it-1..8                              NO     (:936 .. :1153)
   jdbc-connectors-it-part-1..7, -ddl                  NO     (:1182 .. :1378)
   kudu / amazonSqs / kafka / rocketmq / elasticsearch  NO     (:1403 .. :1504)
   mysql-cdc / doris / paimon / oracle-cdc              NO     (:1529 .. :1605)
   connector-file-local / -sftp / redis / sensorsdata   NO     (:1635 .. :1710)
   ```
   
   All 44 are `java: ['8', '11']` matrices. `unit-test` is the only one that also spans `windows-latest`, so it has four legs; the other 43 are JDK-8/JDK-11 pairs on Ubuntu, where one failure cancels the twin. That makes the "1 of 44 sets it" reading even harder to explain as intentional than the "the adjacent job sets it" argument I originally made.
   
   **The same run I cited above demonstrates the IT case directly.** Beyond the `unit-test` leg already in the table, run [`32100301798`](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/32100301798/attempts/1) produced three more:
   
   | Shard | JDK 8 | JDK 11 |
   |---|---|---|
   | `all-connectors-it-6` | **failure** | `cancelled` |
   | `jdbc-connectors-it-ddl` | `cancelled` | **failure** |
   | `rocketmq-connector-it` | **failure** | **success** |
   
   `rocketmq-connector-it` is the informative row, because it is the only one where fail-fast did *not* get to cancel the sibling — the two legs finished close enough together that both completed. And that pair is exactly the diagnostic that fail-fast destroys everywhere else: identical code, identical test, JDK 11 green and JDK 8 red, which localises the problem to the JDK-8 environment in one glance. For `all-connectors-it-6` and `jdbc-connectors-it-ddl` that same comparison was available and was thrown away.
   
   The asymmetry is also reproducible rather than a one-off: on an earlier run of the same branch ([`31807140823`](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/31807140823)), `rocketmq-connector-it (8, ubuntu-latest)` failed while `(11, ubuntu-latest)` passed — the same shape. And on [`31942236205`](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/31942236205) all six of these legs were green. Intermittent, JDK-8-leaning, and only visible as such because the sibling leg happened to survive.
   
   **Revised proposal.** Same one-line change, applied to all 43 jobs rather than just `unit-test`. It is mechanical and reviewable as a pure diff of repeated `fail-fast: false` insertions, with no change to what any job runs.
   
   I am happy to submit that as the PR. If the cost of letting sibling legs finish is a concern for the large IT shards specifically, a reasonable middle ground is to apply it to `unit-test` and the `transform-v2-it` / `engine-*-it` jobs first — the cheap, fast ones where the per-platform signal is most often the thing under question — and treat the connector IT shards separately under #11513's matrix rework. Happy to go either way; I would rather have the scope decision made by someone with the runner-minute budget in view.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org