You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhang-arvin (via GitHub)" <gi...@apache.org> on 2026/08/19 14:08:10 UTC
[PR] [Fix][Connector-V2] Fix MySQL sink connection leak and stale statement errors [seatunnel]
zhang-arvin opened a new pull request, #11883:
URL: https://github.com/apache/seatunnel/pull/11883
## Purpose
Fix the MySQL sink "No operations allowed after statement closed" error in long-running streaming jobs (#11769).
## Root Cause
Three interrelated issues cause stale connections and statement leaks:
1. **ConnectionPoolManager caches connections that HikariCP can evict**: When HikariCP evicts a connection (due to `idleTimeout` or `maxLifetime`), the connection is closed but the reference remains in the `ConnectionPoolManager.connectionMap`. Subsequent calls to `getConnection()` return the stale (closed) connection, causing "No operations allowed after statement closed" errors.
2. **JdbcConnectionValidationUtils ignores user-configured connectionTestQuery**: The `getConnectionValidationQuery()` method only returns a validation query for the Xugu driver. For MySQL and other drivers, it returns `Optional.empty()`, so `isConnectionValid()` uses `Connection.isValid()` which can return false positives in environments with proxies, NATs, or firewalls.
3. **HikariCP pool-level properties are silently ignored**: User-configured properties like `max-lifetime`, `keepalive-time`, `connection-test-query` are passed via `addDataSourceProperty()` which routes them to the underlying JDBC DataSource, not to HikariCP's pool configuration. These properties are silently ignored, preventing proper connection lifecycle management.
## Changes
### 1. ConnectionPoolManager.getConnection()
- Check if the cached connection has been closed by HikariCP
- If closed, evict the stale reference and obtain a fresh connection from the pool
- This prevents "No operations allowed after statement closed" errors in long-running streaming jobs
### 2. JdbcConnectionValidationUtils.getConnectionValidationQuery()
- After checking the driver-specific hook (Xugu), fall back to the user-configured `connectionTestQuery` / `connection-test-query` from the `properties` block
- This enables SQL-based liveness checks instead of `Connection.isValid()` which can return false positives
### 3. JdbcSinkWriter.applyConnectionValidation()
- Apply all HikariCP pool-level properties (maxLifetime, keepaliveTime, validationTimeout, idleTimeout, connectionTimeout, maximumPoolSize, minimumIdle) from user properties config
- Add isHikariPoolProperty() filter to prevent these properties from being redundantly passed to addDataSourceProperty()
### Files Changed
- `seatunnel-connectors-v2/connector-jdbc/src/main/java/.../sink/ConnectionPoolManager.java`
- `seatunnel-connectors-v2/connector-jdbc/src/main/java/.../internal/connection/JdbcConnectionValidationUtils.java`
- `seatunnel-connectors-v2/connector-jdbc/src/main/java/.../sink/JdbcSinkWriter.java`
### Related Issues
- Closes #11769
- Related to PR #11771 (partial fix for connection-test-query only)
### Verification
- All existing tests pass: JdbcSinkWriterTest (9 tests), JdbcConnectionValidationUtilsTest (2 tests)
- Connection validation query from user properties is now properly applied to both HikariCP pool and isConnectionValid() check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MySQL sink connection leak and stale statement errors [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11883:
URL: https://github.com/apache/seatunnel/pull/11883#issuecomment-5617176830
CI-fact update: the `Build` check on this exact head has since completed, but with conclusion `cancelled` rather than a pass or fail (it was still queued/in-progress at review time). Only the initial `changes` gating job ran before cancellation, so this run carries no real pass/fail signal and will need to be re-triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MySQL sink connection leak and stale statement errors [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11883:
URL: https://github.com/apache/seatunnel/pull/11883#issuecomment-5574008732
@DanielLeens Thanks for your patience — you're right that the previous cycles were mostly rebase-only without substantive changes. I've now made the real fixes requested:
1. **No-collector path now guarded (Issue 1)**: `write()` in the no-collector branch now captures the connection via `connectionProvider.getConnection()` and records it in `transactionConnection`, so a silent swap is detected. `close()` no longer calls `connection.commit()` directly — it reuses `commitIfNeeded()`, which throws when `transactionConnection` was swapped, instead of silently committing an empty replacement connection.
2. **Fast path validates (Issue 2)**: `ConnectionPoolManager.getConnection()` no longer returns the cached connection after only an `isClosed()` check. The fast path now runs the same `isUsable()` validation (honoring `connectionTestQuery` / `isValid` with the pool's validation timeout) before returning the cached connection.
3. Tests: `ConnectionPoolManagerTest` passes — 9 tests, 0 failures (`mvnw -pl seatunnel-connectors-v2/connector-jdbc -Dtest=ConnectionPoolManagerTest test`).
Would you mind taking another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MySQL sink connection leak and stale statement errors [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11883:
URL: https://github.com/apache/seatunnel/pull/11883#issuecomment-6051742553
@DanielLeens Thanks for holding the line on Issue 1 — you're right that the fix reached for the validating accessor. Fixed as you suggested. Pushed head `3121d7fa6`.
**Issue 1 (High, blocker) — resolved.** The no-collector `write()` path no longer calls `ConnectionPoolManager.getConnection()` (the validating/evicting accessor) per row. Instead:
- `JdbcOutputFormat` now records the connection its statement executor was prepared with (`createAndOpenStatementExecutor()` and `updateExecutor()` both set it) and exposes `getExecutorConnection()`, which does **no** validation or eviction — safe on the write hot path.
- `JdbcSinkWriter.write()`'s no-collector branch sets `transactionConnection = outputFormat.getExecutorConnection()`. So the tracking follows the connection actually bound to the executor, refreshed only when the executor reconnects, matching the pattern already used by the row-collector path.
**Issue 3 (Medium) — done.** Documented `connectionTestQuery` / the HikariCP pool-property behavior in `docs/en/connectors/sink/Jdbc.md` and `docs/zh/connectors/sink/Jdbc.md` (which keys go to the pool vs. the driver DataSource, and how `connectionTestQuery` is used for validation).
Verified locally: the module compiles and the pre-existing suites stay green — `ConnectionPoolManagerTest` `Tests run: 9, Failures: 0`, `JdbcSinkWriterTest` `Tests run: 9, Failures: 0`. `spotless:apply` clean.
Still open: Issue 2 (a new test that exercises the stale-connection scenario through the no-collector `write()` path) and Issues 4-6 (Low) — happy to take those in a follow-up push; flagging so the state is explicit.
(Also note the head moved `6e3ee70c3 -> 19bc4543a` via a rebase-only push while this was in flight; I rebased this change onto `19bc4543a`, so no prior work was dropped.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix MySQL sink connection leak and stale statement errors [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11883:
URL: https://github.com/apache/seatunnel/pull/11883#issuecomment-5373039829
Thanks @DanielLeens for the detailed review! I've addressed the connection identity tracking issue: `commitIfNeeded()` now checks whether the current connection matches `savepointConnection`, and throws `SQLException` when they diverge. The fix is force-pushed — please re-review when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org