You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by "quantranhong1999 (via GitHub)" <gi...@apache.org> on 2026/09/25 12:58:28 UTC
[PR] JAMES-2586 Fix jOOQ reactive timeout wrongly failing slow consumers of streamed Postgres results [james-project]
quantranhong1999 opened a new pull request, #3208:
URL: https://github.com/apache/james-project/pull/3208
cc @felixauringer @chibenwa
### Problem
A throttled full re-indexing on the Postgres app (`/mailboxes?task=reIndex&messagesPerSecond=N`)
always fails after exactly `jooq.reactive.timeout` (10s by default), having indexed 9 x N messages:
```
PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany'
```
Reported on server-dev and in linagora/tmail-backend#1599, where the timeout is followed by
further Postgres failures until the server "sometimes recovers".
### Root cause
Not a hanging query. The full re-indexing lists all mailboxes with a mailbox concurrency of 1 and
throttles the messages of each mailbox, so the mailbox listing query stays open while the first
mailbox is slowly re-indexed. `PostgresExecutor` guarded streamed results with `Flux.timeout`,
whose timer restarts on every emitted row regardless of whether the consumer requested more rows.
A consumer slower than the timeout is therefore failed while Postgres is only waiting for demand.
A second issue was found while reproducing: cancelling the pipeline on timeout does not stop the
query on the Postgres side. The busy connection goes back to the pool, the next query on it queues
behind the still running statement and times out too, which is the cascade seen in the field.
### Fixes
- **Demand aware timeout**: new `TimeoutOnPendingDemand` operator, used for streamed Flux results.
The timer only runs while the downstream has pending demand and restarts on each delivered row.
A genuine stall while rows are awaited still raises `TimeoutException`. Mono results are unchanged.
- **Cancel on timeout**: on `TimeoutException`, `PostgresExecutor` sends a r2dbc-postgresql
`cancelRequest()` before releasing the connection, so it is reusable within milliseconds.
- **Library upgrade**: jOOQ 3.20.5 -> 3.20.20 and r2dbc-postgresql 1.0.7 -> 1.1.3, which ship the
fixes for cancelled queries poisoning pooled connections (jOOQ#20112, r2dbc-postgresql#661).
### Tests (TDD, reproducing tests committed before each fix)
- `PostgresReIndexingIntegrationTest`: the reported webadmin scenario on a Guice Postgres server.
- `PostgresExecutorTimeoutTest`: slow consumer must not time out, a stalled query must, and a
single-connection pool is usable right after a timeout (60s `pg_sleep`).
- `TimeoutOnPendingDemandTest`: 10 virtual-time cases for the operator.
Validated on the Postgres backend, mailbox, blob, data, JMAP data, event-bus, task and vault
module suites, and the full Postgres webadmin integration module.
### Note for reviewers
As discussed on the list, paginating the mailbox listing (per-page queries) would additionally avoid
holding a pooled connection during slow consumption, and would preserve the previous accidental
"deadlock breaker" behaviour of the timeout under pool exhaustion. This PR keeps streaming and can
be combined with that follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5866165622
I would merge this in ~2 days if there is no disagreement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5885706252
> Just a quick reminder that you need to add the timeout handling also to the new function introduced by https://github.com/apache/james-project/pull/3203 when rebasing :)
I have rebased. And the latest commit should handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5885338929
> Just a quick reminder that you need to add the timeout handling also to the new function introduced by https://github.com/apache/james-project/pull/3203 when rebasing :)
Indeed, very good catch. I will definitely handle that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5864152397
> I understand that @chibenwa's fix should prevent these timeout errors, but if a timeout error would occur for some unrelated reason, this would currently still return that unusable connection to the pool, right?
Indeed. Thanks for your attention! Let me see to bring it back :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa merged PR #3208:
URL: https://github.com/apache/james-project/pull/3208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Fix jOOQ reactive timeout wrongly failing slow consumers of streamed Postgres results [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5862815281
I reworked this to make it complementary for https://github.com/apache/james-project/pull/3209:
- Add a TDD test to reproduce the issue (and yes, https://github.com/apache/james-project/pull/3209 fixed the test)
- Upgrade jOOQ and r2dbc-postgresql versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "felixauringer (via GitHub)" <gi...@apache.org>.
felixauringer commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5866926245
> **UPDATED**: The 2 latest commits should resolve your concern.
Thanks, looks good to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "felixauringer (via GitHub)" <gi...@apache.org>.
felixauringer commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5885313468
Just a quick reminder that you need to add the timeout handling also to the new function introduced by https://github.com/apache/james-project/pull/3203 when rebasing :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-2586 Add TDD test for jOOQ timeout issue when reindexing mailbox [james-project]
Posted by "felixauringer (via GitHub)" <gi...@apache.org>.
felixauringer commented on PR #3208:
URL: https://github.com/apache/james-project/pull/3208#issuecomment-5864118293
@quantranhong1999 You also identified the problem that if running into a timeout, the query is not properly terminated on the postgres side.
You included a fix in which the
> PostgresExecutor sends a r2dbc-postgresql cancelRequest() before releasing the connection, so it is reusable within milliseconds.
That sounded very reasonable to me.
I understand that @chibenwa's fix should prevent these timeout errors, but if a timeout error would occur for some unrelated reason, this would currently still return that unusable connection to the pool, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org