You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Felix Auringer <fe...@giz.berlin> on 2026/09/25 10:51:09 UTC
Postgres Timeouts
Hello,
For some time now, I have seen a timeout error in the postgres executor:
o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
This was mainly happening in TMail and was reported here: https://github.com/linagora/tmail-backend/issues/1599
I have now found a way to reproduce this reliably with the Docker image built from the current James master (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
When using opensearch as the search backend and triggering a full reindex with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1', I observe the following behavior:
2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter - WebAdmin request received
2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer - Starting a full reindex
2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter - WebAdmin response received
2026-09-25T10:29:43.178Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
2026-09-25T10:29:44.120Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
2026-09-25T10:29:45.125Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
2026-09-25T10:29:46.128Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
2026-09-25T10:29:47.131Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
2026-09-25T10:29:48.135Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
2026-09-25T10:29:49.137Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
2026-09-25T10:29:50.147Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
2026-09-25T10:29:51.143Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer - Full reindex finished
2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker - Task was partially performed. Check logs for more details. Taskid : TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
This error always happens 10 seconds after starting the reindex task.
Therefore, exactly 9 batches of the size specified with messagesPerSecond are processed.
When indexing 2 messages per second, the indexer manages to index the first 18 messages, when indexing 5 messages per second, the first 45 messages are indexed, and so on...
I don't know much about this part of the codebase and would be happy about help to debug this.
Did somebody else ever see this error or can somebody else reproduce it?
Best regards,
Felix
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
www.giz.berlin; felix.auringer@giz.berlin
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Postgres Timeouts
Posted by Quan tran hong <qu...@gmail.com>.
Hello Felix,
I did attempt a shot at this (with my agent) and likely got a fix for it.
But I will need to validate the fix first before pushing it when I have
time.
Stay tuned ^^.
Quan
On Fri, Sep 25, 2026 at 5:51 PM Felix Auringer <fe...@giz.berlin>
wrote:
> Hello,
>
> For some time now, I have seen a timeout error in the postgres executor:
> o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres
> query. May need to check either jOOQ reactive issue or Postgres DB
> performance.
> This was mainly happening in TMail and was reported here:
> https://github.com/linagora/tmail-backend/issues/1599
>
> I have now found a way to reproduce this reliably with the Docker image
> built from the current James master
> (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
> When using opensearch as the search backend and triggering a full reindex
> with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1',
> I observe the following behavior:
>
> 2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter -
> WebAdmin request received
> 2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
> Starting a full reindex
> 2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter
> - WebAdmin response received
> 2026-09-25T10:29:43.178Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
> 2026-09-25T10:29:44.120Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
> 2026-09-25T10:29:45.125Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
> 2026-09-25T10:29:46.128Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
> 2026-09-25T10:29:47.131Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
> 2026-09-25T10:29:48.135Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
> 2026-09-25T10:29:49.137Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
> 2026-09-25T10:29:50.147Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
> 2026-09-25T10:29:51.143Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
> 2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor -
> Time out executing Postgres query. May need to check either jOOQ reactive
> issue or Postgres DB performance.
> java.util.concurrent.TimeoutException: Did not observe any item or
> terminal signal within 10000ms in 'flatMapMany' (and no fallback has been
> configured)
> at
> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
> at
> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
> at
> reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
> at
> reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
> at
> reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
> at
> reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
> at
> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at
> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
> Full reindex finished
> 2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker -
> Task was partially performed. Check logs for more details. Taskid :
> TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
>
> This error always happens 10 seconds after starting the reindex task.
> Therefore, exactly 9 batches of the size specified with messagesPerSecond
> are processed.
> When indexing 2 messages per second, the indexer manages to index the
> first 18 messages, when indexing 5 messages per second, the first 45
> messages are indexed, and so on...
>
> I don't know much about this part of the codebase and would be happy about
> help to debug this.
> Did somebody else ever see this error or can somebody else reproduce it?
>
> Best regards,
> Felix
> ---
> Gesellschaft für interkulturelles
> Zusammenleben gGmbH (GIZ)
> Felix Auringer
> IT
> Reformationsplatz 2
> 13597 Berlin
>
> Tel: 030/513 0100 00; Fax: 030/513 0100 09
> www.giz.berlin; felix.auringer@giz.berlin
>
> Amtsgericht Charlottenburg HRB 200872 B
> Geschäftsführerin: Dr. Britta Marschke
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Re: Postgres Timeouts
Posted by Felix Auringer <fe...@giz.berlin>.
Hey Benoit,
thanks for the quick fix! I reviewed your MR on GitHub.
> Sorry for missing this - I thought the existing test suite covered this.
No, problem, I also thought the test was covering this.
Best regards,
Felix
On 28/09/2026 17.01, Benoit TELLIER via server-dev wrote:
> I see.
>
> So the issue is that we shall buffer the page of result from PG in James memory and replay it so that we are sure to release the underlying connection if I get it right.
>
> I think I found something which collectList() withholds upstream requests until downstream demand exists, so expand's page N+1 subscription arrives with zero demand and just idles on its connection.
> I'll open a draft PR: https://github.com/apache/james-project/pull/3215
>
> I'm happy if you can test it and see if it fixes the issue.
>
> Sorry for missing this - I thought the existing test suite covered this.
> --
>
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Twake-Mail product.
> Chairman of the Apache James project.
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
> Le sept. 28, 2026 9:53 AM, de Felix Auringer <fe...@giz.berlin>Hey Benoit,
>
> thanks for the very fast response!
> I only managed to test your changes today and they definitely improved the issue.
> However, I can still reproduce it in a similar way.
> When using opensearch as the search backend and triggering a full reindex with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=500', the reindexing fails for some folders:
>
> 2026-09-28T07:22:25.769Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2336
> 2026-09-28T07:22:25.778Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2339
> 2026-09-28T07:22:25.766Z [1;31m[ERROR][0;39m o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
> java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
> at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
> at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
> at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
> at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
> at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
> at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2026-09-28T07:22:25.780Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2346
> 2026-09-28T07:22:25.783Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2342
> 2026-09-28T07:22:25.784Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2345
> 2026-09-28T07:22:25.786Z [31m[WARN ][0;39m o.a.m.tools.indexer.ReIndexerPerformer - ReIndexing failed for #private:<user>:Sent
> java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
> at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
> at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
> at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
> at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
> at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
> at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2026-09-28T07:22:25.789Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2343
> 2026-09-28T07:22:25.792Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2341
>
> In total, the reindexing now processed a few dozen thousand emails and terminated in a short time.
> The folders the process fails for all have more than 5000 emails, so I guess James currently opens one streaming connection per mailbox it indexes and then fails after processing 500 (messages per seconds) * 10 (seconds timeout) ≈ 5000 emails.
>
> Best regards,
> Felix
>
> On 25/09/2026 15.17, Benoit TELLIER via server-dev wrote:
>> Thanks for the look @Quan but I'd rather address the root cause: the listing should not hold a permanent connection. It should use one every time it fetches one page.
>>
>> I did do an alternative proposition here: github.com/apache/james-project/pull/3209
>>
>> --
>>
>>
>> Best regards,
>>
>> Benoit TELLIER
>>
>> General manager of Linagora VIETNAM.
>> Product owner for Twake-Mail product.
>> Chairman of the Apache James project.
>>
>> Mail: btellier@linagora.com
>> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>>
>>
>>
>> On Sep 25, 2026 3:02 PM, from Quan Tran Hong <qu...@gmail.com>Hi Benoit,
>>
>> Food for thought: github.com/apache/james-project/pull/3208
>>
>> I did a quick review of it, and the fixes make sense to me (I asked the
>> agent to do TDD: reproduce the reported issue first).
>>
>> I hope it helps. I will properly double check the fixes later.
>>
>> Quan
>>
>> On Fri, Sep 25, 2026 at 7:42 PM Benoit TELLIER via server-dev <
>> server-dev@james.apache.org> wrote:
>>
>>> Hello Felix
>>>
>>> That's a classic Jooq / R2DBC issue: it badly support streaming as it hold
>>> a live connection.
>>>
>>> The reindexing read live data as it needs it (so 2 message per second for
>>> instance) and keep it opens as it iterates/
>>>
>>> The listing query then times out after 10s, thus the symptoms.
>>>
>>> We encountered that in over parts of the PG implem and used buffering (
>>> intermediate .collectList ) to pull results in memory, and return the
>>> driver thread as fast as we could.
>>>
>>> The tricky point is that EG Cassandra driver supports reading large result
>>> sets without monopolizing driver resources. It thus don't need that memory
>>> buffering. Any fix doing this would then need not to impact Cassandra
>>> implementation that could be doing the optimal list-as-it-comes operation.
>>>
>>> Another solution is to rethink "streaming" and switch it to explicit
>>> separated request per page size. This lift the time limit as we no longer
>>> hold a collection at the cost of repeated queries. A sample can be found
>>> here
>>> github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487
>>>
>>> A generic adoption of this pattern can solve the problem you mention (at
>>> the cost of snapshot isolation which I believe we do not really care.
>>> I'll try to propose a little something in that direction.
>>>
>>> --
>>>
>>>
>>> Best regards,
>>>
>>> Benoit TELLIER
>>>
>>> General manager of Linagora VIETNAM.
>>> Product owner for Twake-Mail product.
>>> Chairman of the Apache James project.
>>>
>>> Mail: btellier@linagora.com
>>> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>>>
>>>
>>>
>>> On Sep 25, 2026 12:51 PM, from Felix Auringer <felix.auringer@giz.berlin
>>>> Hello,
>>>
>>> For some time now, I have seen a timeout error in the postgres executor:
>>> o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres
>>> query. May need to check either jOOQ reactive issue or Postgres DB
>>> performance.
>>> This was mainly happening in TMail and was reported here:
>>> github.com/linagora/tmail-backend/issues/1599
>>>
>>> I have now found a way to reproduce this reliably with the Docker image
>>> built from the current James master
>>> (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
>>> When using opensearch as the search backend and triggering a full reindex
>>> with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1',
>>> I observe the following behavior:
>>>
>>> 2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter -
>>> WebAdmin request received
>>> 2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
>>> Starting a full reindex
>>> 2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter
>>> - WebAdmin response received
>>> 2026-09-25T10:29:43.178Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
>>> 2026-09-25T10:29:44.120Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
>>> 2026-09-25T10:29:45.125Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
>>> 2026-09-25T10:29:46.128Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
>>> 2026-09-25T10:29:47.131Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
>>> 2026-09-25T10:29:48.135Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
>>> 2026-09-25T10:29:49.137Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
>>> 2026-09-25T10:29:50.147Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
>>> 2026-09-25T10:29:51.143Z [INFO ]
>>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
>>> 2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor -
>>> Time out executing Postgres query. May need to check either jOOQ reactive
>>> issue or Postgres DB performance.
>>> java.util.concurrent.TimeoutException: Did not observe any item or
>>> terminal signal within 10000ms in 'flatMapMany' (and no fallback has been
>>> configured)
>>> at
>>> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
>>> at
>>> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
>>> at
>>> reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
>>> at
>>> reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
>>> at
>>> reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
>>> at
>>> reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
>>> at
>>> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
>>> at
>>> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
>>> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
>>> at
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> Source)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>> at java.base/java.lang.Thread.run(Unknown Source)
>>> 2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
>>> Full reindex finished
>>> 2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker -
>>> Task was partially performed. Check logs for more details. Taskid :
>>> TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
>>>
>>> This error always happens 10 seconds after starting the reindex task.
>>> Therefore, exactly 9 batches of the size specified with messagesPerSecond
>>> are processed.
>>> When indexing 2 messages per second, the indexer manages to index the
>>> first 18 messages, when indexing 5 messages per second, the first 45
>>> messages are indexed, and so on...
>>>
>>> I don't know much about this part of the codebase and would be happy about
>>> help to debug this.
>>> Did somebody else ever see this error or can somebody else reproduce it?
>>>
>>> Best regards,
>>> Felix
>>> ---
>>> Gesellschaft für interkulturelles
>>> Zusammenleben gGmbH (GIZ)
>>> Felix Auringer
>>> IT
>>> Reformationsplatz 2
>>> 13597 Berlin
>>>
>>> Tel: 030/513 0100 00; Fax: 030/513 0100 09
>>> giz.berlin; felix.auringer@giz.berlin
>>>
>>> Amtsgericht Charlottenburg HRB 200872 B
>>> Geschäftsführerin: Dr. Britta Marschke
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>
>
> ---
> Gesellschaft für interkulturelles
> Zusammenleben gGmbH (GIZ)
> Felix Auringer
> IT
> Reformationsplatz 2
> 13597 Berlin
>
> Tel: 030/513 0100 00; Fax: 030/513 0100 09
> giz.berlin; felix.auringer@giz.berlin
>
> Amtsgericht Charlottenburg HRB 200872 B
> Geschäftsführerin: Dr. Britta Marschke
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
www.giz.berlin; felix.auringer@giz.berlin
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Postgres Timeouts
Posted by Benoit TELLIER via server-dev <se...@james.apache.org>.
I see.
So the issue is that we shall buffer the page of result from PG in James memory and replay it so that we are sure to release the underlying connection if I get it right.
I think I found something which collectList() withholds upstream requests until downstream demand exists, so expand's page N+1 subscription arrives with zero demand and just idles on its connection.
I'll open a draft PR: https://github.com/apache/james-project/pull/3215
I'm happy if you can test it and see if it fixes the issue.
Sorry for missing this - I thought the existing test suite covered this.
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Twake-Mail product.
Chairman of the Apache James project.
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
Le sept. 28, 2026 9:53 AM, de Felix Auringer <fe...@giz.berlin>Hey Benoit,
thanks for the very fast response!
I only managed to test your changes today and they definitely improved the issue.
However, I can still reproduce it in a similar way.
When using opensearch as the search backend and triggering a full reindex with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=500', the reindexing fails for some folders:
2026-09-28T07:22:25.769Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2336
2026-09-28T07:22:25.778Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2339
2026-09-28T07:22:25.766Z [1;31m[ERROR][0;39m o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-28T07:22:25.780Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2346
2026-09-28T07:22:25.783Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2342
2026-09-28T07:22:25.784Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2345
2026-09-28T07:22:25.786Z [31m[WARN ][0;39m o.a.m.tools.indexer.ReIndexerPerformer - ReIndexing failed for #private:<user>:Sent
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-28T07:22:25.789Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2343
2026-09-28T07:22:25.792Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2341
In total, the reindexing now processed a few dozen thousand emails and terminated in a short time.
The folders the process fails for all have more than 5000 emails, so I guess James currently opens one streaming connection per mailbox it indexes and then fails after processing 500 (messages per seconds) * 10 (seconds timeout) ≈ 5000 emails.
Best regards,
Felix
On 25/09/2026 15.17, Benoit TELLIER via server-dev wrote:
> Thanks for the look @Quan but I'd rather address the root cause: the listing should not hold a permanent connection. It should use one every time it fetches one page.
>
> I did do an alternative proposition here: github.com/apache/james-project/pull/3209
>
> --
>
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Twake-Mail product.
> Chairman of the Apache James project.
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
> On Sep 25, 2026 3:02 PM, from Quan Tran Hong <qu...@gmail.com>Hi Benoit,
>
> Food for thought: github.com/apache/james-project/pull/3208
>
> I did a quick review of it, and the fixes make sense to me (I asked the
> agent to do TDD: reproduce the reported issue first).
>
> I hope it helps. I will properly double check the fixes later.
>
> Quan
>
> On Fri, Sep 25, 2026 at 7:42 PM Benoit TELLIER via server-dev <
> server-dev@james.apache.org> wrote:
>
>> Hello Felix
>>
>> That's a classic Jooq / R2DBC issue: it badly support streaming as it hold
>> a live connection.
>>
>> The reindexing read live data as it needs it (so 2 message per second for
>> instance) and keep it opens as it iterates/
>>
>> The listing query then times out after 10s, thus the symptoms.
>>
>> We encountered that in over parts of the PG implem and used buffering (
>> intermediate .collectList ) to pull results in memory, and return the
>> driver thread as fast as we could.
>>
>> The tricky point is that EG Cassandra driver supports reading large result
>> sets without monopolizing driver resources. It thus don't need that memory
>> buffering. Any fix doing this would then need not to impact Cassandra
>> implementation that could be doing the optimal list-as-it-comes operation.
>>
>> Another solution is to rethink "streaming" and switch it to explicit
>> separated request per page size. This lift the time limit as we no longer
>> hold a collection at the cost of repeated queries. A sample can be found
>> here
>> github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487
>>
>> A generic adoption of this pattern can solve the problem you mention (at
>> the cost of snapshot isolation which I believe we do not really care.
>> I'll try to propose a little something in that direction.
>>
>> --
>>
>>
>> Best regards,
>>
>> Benoit TELLIER
>>
>> General manager of Linagora VIETNAM.
>> Product owner for Twake-Mail product.
>> Chairman of the Apache James project.
>>
>> Mail: btellier@linagora.com
>> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>>
>>
>>
>> On Sep 25, 2026 12:51 PM, from Felix Auringer <felix.auringer@giz.berlin
>>> Hello,
>>
>> For some time now, I have seen a timeout error in the postgres executor:
>> o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres
>> query. May need to check either jOOQ reactive issue or Postgres DB
>> performance.
>> This was mainly happening in TMail and was reported here:
>> github.com/linagora/tmail-backend/issues/1599
>>
>> I have now found a way to reproduce this reliably with the Docker image
>> built from the current James master
>> (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
>> When using opensearch as the search backend and triggering a full reindex
>> with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1',
>> I observe the following behavior:
>>
>> 2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter -
>> WebAdmin request received
>> 2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
>> Starting a full reindex
>> 2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter
>> - WebAdmin response received
>> 2026-09-25T10:29:43.178Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
>> 2026-09-25T10:29:44.120Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
>> 2026-09-25T10:29:45.125Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
>> 2026-09-25T10:29:46.128Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
>> 2026-09-25T10:29:47.131Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
>> 2026-09-25T10:29:48.135Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
>> 2026-09-25T10:29:49.137Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
>> 2026-09-25T10:29:50.147Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
>> 2026-09-25T10:29:51.143Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
>> 2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor -
>> Time out executing Postgres query. May need to check either jOOQ reactive
>> issue or Postgres DB performance.
>> java.util.concurrent.TimeoutException: Did not observe any item or
>> terminal signal within 10000ms in 'flatMapMany' (and no fallback has been
>> configured)
>> at
>> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
>> at
>> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
>> at
>> reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
>> at
>> reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
>> at
>> reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
>> at
>> reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
>> at
>> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
>> at
>> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
>> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> at java.base/java.lang.Thread.run(Unknown Source)
>> 2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
>> Full reindex finished
>> 2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker -
>> Task was partially performed. Check logs for more details. Taskid :
>> TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
>>
>> This error always happens 10 seconds after starting the reindex task.
>> Therefore, exactly 9 batches of the size specified with messagesPerSecond
>> are processed.
>> When indexing 2 messages per second, the indexer manages to index the
>> first 18 messages, when indexing 5 messages per second, the first 45
>> messages are indexed, and so on...
>>
>> I don't know much about this part of the codebase and would be happy about
>> help to debug this.
>> Did somebody else ever see this error or can somebody else reproduce it?
>>
>> Best regards,
>> Felix
>> ---
>> Gesellschaft für interkulturelles
>> Zusammenleben gGmbH (GIZ)
>> Felix Auringer
>> IT
>> Reformationsplatz 2
>> 13597 Berlin
>>
>> Tel: 030/513 0100 00; Fax: 030/513 0100 09
>> giz.berlin; felix.auringer@giz.berlin
>>
>> Amtsgericht Charlottenburg HRB 200872 B
>> Geschäftsführerin: Dr. Britta Marschke
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
giz.berlin; felix.auringer@giz.berlin
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Postgres Timeouts
Posted by Felix Auringer <fe...@giz.berlin>.
Hey Benoit,
thanks for the very fast response!
I only managed to test your changes today and they definitely improved the issue.
However, I can still reproduce it in a similar way.
When using opensearch as the search backend and triggering a full reindex with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=500', the reindexing fails for some folders:
2026-09-28T07:22:25.769Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2336
2026-09-28T07:22:25.778Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2339
2026-09-28T07:22:25.766Z [1;31m[ERROR][0;39m o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-28T07:22:25.780Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2346
2026-09-28T07:22:25.783Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2342
2026-09-28T07:22:25.784Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2345
2026-09-28T07:22:25.786Z [31m[WARN ][0;39m o.a.m.tools.indexer.ReIndexerPerformer - ReIndexing failed for #private:<user>:Sent
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-28T07:22:25.789Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2343
2026-09-28T07:22:25.792Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2341
In total, the reindexing now processed a few dozen thousand emails and terminated in a short time.
The folders the process fails for all have more than 5000 emails, so I guess James currently opens one streaming connection per mailbox it indexes and then fails after processing 500 (messages per seconds) * 10 (seconds timeout) ≈ 5000 emails.
Best regards,
Felix
On 25/09/2026 15.17, Benoit TELLIER via server-dev wrote:
> Thanks for the look @Quan but I'd rather address the root cause: the listing should not hold a permanent connection. It should use one every time it fetches one page.
>
> I did do an alternative proposition here: https://github.com/apache/james-project/pull/3209
>
> --
>
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Twake-Mail product.
> Chairman of the Apache James project.
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
> On Sep 25, 2026 3:02 PM, from Quan Tran Hong <qu...@gmail.com>Hi Benoit,
>
> Food for thought: github.com/apache/james-project/pull/3208
>
> I did a quick review of it, and the fixes make sense to me (I asked the
> agent to do TDD: reproduce the reported issue first).
>
> I hope it helps. I will properly double check the fixes later.
>
> Quan
>
> On Fri, Sep 25, 2026 at 7:42 PM Benoit TELLIER via server-dev <
> server-dev@james.apache.org> wrote:
>
>> Hello Felix
>>
>> That's a classic Jooq / R2DBC issue: it badly support streaming as it hold
>> a live connection.
>>
>> The reindexing read live data as it needs it (so 2 message per second for
>> instance) and keep it opens as it iterates/
>>
>> The listing query then times out after 10s, thus the symptoms.
>>
>> We encountered that in over parts of the PG implem and used buffering (
>> intermediate .collectList ) to pull results in memory, and return the
>> driver thread as fast as we could.
>>
>> The tricky point is that EG Cassandra driver supports reading large result
>> sets without monopolizing driver resources. It thus don't need that memory
>> buffering. Any fix doing this would then need not to impact Cassandra
>> implementation that could be doing the optimal list-as-it-comes operation.
>>
>> Another solution is to rethink "streaming" and switch it to explicit
>> separated request per page size. This lift the time limit as we no longer
>> hold a collection at the cost of repeated queries. A sample can be found
>> here
>> github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487
>>
>> A generic adoption of this pattern can solve the problem you mention (at
>> the cost of snapshot isolation which I believe we do not really care.
>> I'll try to propose a little something in that direction.
>>
>> --
>>
>>
>> Best regards,
>>
>> Benoit TELLIER
>>
>> General manager of Linagora VIETNAM.
>> Product owner for Twake-Mail product.
>> Chairman of the Apache James project.
>>
>> Mail: btellier@linagora.com
>> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>>
>>
>>
>> On Sep 25, 2026 12:51 PM, from Felix Auringer <felix.auringer@giz.berlin
>>> Hello,
>>
>> For some time now, I have seen a timeout error in the postgres executor:
>> o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres
>> query. May need to check either jOOQ reactive issue or Postgres DB
>> performance.
>> This was mainly happening in TMail and was reported here:
>> github.com/linagora/tmail-backend/issues/1599
>>
>> I have now found a way to reproduce this reliably with the Docker image
>> built from the current James master
>> (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
>> When using opensearch as the search backend and triggering a full reindex
>> with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1',
>> I observe the following behavior:
>>
>> 2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter -
>> WebAdmin request received
>> 2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
>> Starting a full reindex
>> 2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter
>> - WebAdmin response received
>> 2026-09-25T10:29:43.178Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
>> 2026-09-25T10:29:44.120Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
>> 2026-09-25T10:29:45.125Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
>> 2026-09-25T10:29:46.128Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
>> 2026-09-25T10:29:47.131Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
>> 2026-09-25T10:29:48.135Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
>> 2026-09-25T10:29:49.137Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
>> 2026-09-25T10:29:50.147Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
>> 2026-09-25T10:29:51.143Z [INFO ]
>> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
>> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
>> 2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor -
>> Time out executing Postgres query. May need to check either jOOQ reactive
>> issue or Postgres DB performance.
>> java.util.concurrent.TimeoutException: Did not observe any item or
>> terminal signal within 10000ms in 'flatMapMany' (and no fallback has been
>> configured)
>> at
>> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
>> at
>> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
>> at
>> reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
>> at
>> reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
>> at
>> reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
>> at
>> reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
>> at
>> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
>> at
>> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
>> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> at java.base/java.lang.Thread.run(Unknown Source)
>> 2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
>> Full reindex finished
>> 2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker -
>> Task was partially performed. Check logs for more details. Taskid :
>> TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
>>
>> This error always happens 10 seconds after starting the reindex task.
>> Therefore, exactly 9 batches of the size specified with messagesPerSecond
>> are processed.
>> When indexing 2 messages per second, the indexer manages to index the
>> first 18 messages, when indexing 5 messages per second, the first 45
>> messages are indexed, and so on...
>>
>> I don't know much about this part of the codebase and would be happy about
>> help to debug this.
>> Did somebody else ever see this error or can somebody else reproduce it?
>>
>> Best regards,
>> Felix
>> ---
>> Gesellschaft für interkulturelles
>> Zusammenleben gGmbH (GIZ)
>> Felix Auringer
>> IT
>> Reformationsplatz 2
>> 13597 Berlin
>>
>> Tel: 030/513 0100 00; Fax: 030/513 0100 09
>> giz.berlin; felix.auringer@giz.berlin
>>
>> Amtsgericht Charlottenburg HRB 200872 B
>> Geschäftsführerin: Dr. Britta Marschke
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
www.giz.berlin; felix.auringer@giz.berlin
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Postgres Timeouts
Posted by Benoit TELLIER via server-dev <se...@james.apache.org>.
Thanks for the look @Quan but I'd rather address the root cause: the listing should not hold a permanent connection. It should use one every time it fetches one page.
I did do an alternative proposition here: https://github.com/apache/james-project/pull/3209
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Twake-Mail product.
Chairman of the Apache James project.
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
On Sep 25, 2026 3:02 PM, from Quan Tran Hong <qu...@gmail.com>Hi Benoit,
Food for thought: github.com/apache/james-project/pull/3208
I did a quick review of it, and the fixes make sense to me (I asked the
agent to do TDD: reproduce the reported issue first).
I hope it helps. I will properly double check the fixes later.
Quan
On Fri, Sep 25, 2026 at 7:42 PM Benoit TELLIER via server-dev <
server-dev@james.apache.org> wrote:
> Hello Felix
>
> That's a classic Jooq / R2DBC issue: it badly support streaming as it hold
> a live connection.
>
> The reindexing read live data as it needs it (so 2 message per second for
> instance) and keep it opens as it iterates/
>
> The listing query then times out after 10s, thus the symptoms.
>
> We encountered that in over parts of the PG implem and used buffering (
> intermediate .collectList ) to pull results in memory, and return the
> driver thread as fast as we could.
>
> The tricky point is that EG Cassandra driver supports reading large result
> sets without monopolizing driver resources. It thus don't need that memory
> buffering. Any fix doing this would then need not to impact Cassandra
> implementation that could be doing the optimal list-as-it-comes operation.
>
> Another solution is to rethink "streaming" and switch it to explicit
> separated request per page size. This lift the time limit as we no longer
> hold a collection at the cost of repeated queries. A sample can be found
> here
> github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487
>
> A generic adoption of this pattern can solve the problem you mention (at
> the cost of snapshot isolation which I believe we do not really care.
> I'll try to propose a little something in that direction.
>
> --
>
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Twake-Mail product.
> Chairman of the Apache James project.
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
> On Sep 25, 2026 12:51 PM, from Felix Auringer <felix.auringer@giz.berlin
> >Hello,
>
> For some time now, I have seen a timeout error in the postgres executor:
> o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres
> query. May need to check either jOOQ reactive issue or Postgres DB
> performance.
> This was mainly happening in TMail and was reported here:
> github.com/linagora/tmail-backend/issues/1599
>
> I have now found a way to reproduce this reliably with the Docker image
> built from the current James master
> (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
> When using opensearch as the search backend and triggering a full reindex
> with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1',
> I observe the following behavior:
>
> 2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter -
> WebAdmin request received
> 2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
> Starting a full reindex
> 2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter
> - WebAdmin response received
> 2026-09-25T10:29:43.178Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
> 2026-09-25T10:29:44.120Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
> 2026-09-25T10:29:45.125Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
> 2026-09-25T10:29:46.128Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
> 2026-09-25T10:29:47.131Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
> 2026-09-25T10:29:48.135Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
> 2026-09-25T10:29:49.137Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
> 2026-09-25T10:29:50.147Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
> 2026-09-25T10:29:51.143Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
> 2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor -
> Time out executing Postgres query. May need to check either jOOQ reactive
> issue or Postgres DB performance.
> java.util.concurrent.TimeoutException: Did not observe any item or
> terminal signal within 10000ms in 'flatMapMany' (and no fallback has been
> configured)
> at
> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
> at
> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
> at
> reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
> at
> reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
> at
> reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
> at
> reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
> at
> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at
> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
> Full reindex finished
> 2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker -
> Task was partially performed. Check logs for more details. Taskid :
> TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
>
> This error always happens 10 seconds after starting the reindex task.
> Therefore, exactly 9 batches of the size specified with messagesPerSecond
> are processed.
> When indexing 2 messages per second, the indexer manages to index the
> first 18 messages, when indexing 5 messages per second, the first 45
> messages are indexed, and so on...
>
> I don't know much about this part of the codebase and would be happy about
> help to debug this.
> Did somebody else ever see this error or can somebody else reproduce it?
>
> Best regards,
> Felix
> ---
> Gesellschaft für interkulturelles
> Zusammenleben gGmbH (GIZ)
> Felix Auringer
> IT
> Reformationsplatz 2
> 13597 Berlin
>
> Tel: 030/513 0100 00; Fax: 030/513 0100 09
> giz.berlin; felix.auringer@giz.berlin
>
> Amtsgericht Charlottenburg HRB 200872 B
> Geschäftsführerin: Dr. Britta Marschke
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Re: Postgres Timeouts
Posted by Quan tran hong <qu...@gmail.com>.
Hi Benoit,
Food for thought: https://github.com/apache/james-project/pull/3208
I did a quick review of it, and the fixes make sense to me (I asked the
agent to do TDD: reproduce the reported issue first).
I hope it helps. I will properly double check the fixes later.
Quan
On Fri, Sep 25, 2026 at 7:42 PM Benoit TELLIER via server-dev <
server-dev@james.apache.org> wrote:
> Hello Felix
>
> That's a classic Jooq / R2DBC issue: it badly support streaming as it hold
> a live connection.
>
> The reindexing read live data as it needs it (so 2 message per second for
> instance) and keep it opens as it iterates/
>
> The listing query then times out after 10s, thus the symptoms.
>
> We encountered that in over parts of the PG implem and used buffering (
> intermediate .collectList ) to pull results in memory, and return the
> driver thread as fast as we could.
>
> The tricky point is that EG Cassandra driver supports reading large result
> sets without monopolizing driver resources. It thus don't need that memory
> buffering. Any fix doing this would then need not to impact Cassandra
> implementation that could be doing the optimal list-as-it-comes operation.
>
> Another solution is to rethink "streaming" and switch it to explicit
> separated request per page size. This lift the time limit as we no longer
> hold a collection at the cost of repeated queries. A sample can be found
> here
> https://github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487
>
> A generic adoption of this pattern can solve the problem you mention (at
> the cost of snapshot isolation which I believe we do not really care.
> I'll try to propose a little something in that direction.
>
> --
>
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Twake-Mail product.
> Chairman of the Apache James project.
>
> Mail: btellier@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
> On Sep 25, 2026 12:51 PM, from Felix Auringer <felix.auringer@giz.berlin
> >Hello,
>
> For some time now, I have seen a timeout error in the postgres executor:
> o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres
> query. May need to check either jOOQ reactive issue or Postgres DB
> performance.
> This was mainly happening in TMail and was reported here:
> github.com/linagora/tmail-backend/issues/1599
>
> I have now found a way to reproduce this reliably with the Docker image
> built from the current James master
> (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
> When using opensearch as the search backend and triggering a full reindex
> with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1',
> I observe the following behavior:
>
> 2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter -
> WebAdmin request received
> 2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
> Starting a full reindex
> 2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter
> - WebAdmin response received
> 2026-09-25T10:29:43.178Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
> 2026-09-25T10:29:44.120Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
> 2026-09-25T10:29:45.125Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
> 2026-09-25T10:29:46.128Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
> 2026-09-25T10:29:47.131Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
> 2026-09-25T10:29:48.135Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
> 2026-09-25T10:29:49.137Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
> 2026-09-25T10:29:50.147Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
> 2026-09-25T10:29:51.143Z [INFO ]
> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
> 2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor -
> Time out executing Postgres query. May need to check either jOOQ reactive
> issue or Postgres DB performance.
> java.util.concurrent.TimeoutException: Did not observe any item or
> terminal signal within 10000ms in 'flatMapMany' (and no fallback has been
> configured)
> at
> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
> at
> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
> at
> reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
> at
> reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
> at
> reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
> at
> reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
> at
> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at
> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
> Full reindex finished
> 2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker -
> Task was partially performed. Check logs for more details. Taskid :
> TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
>
> This error always happens 10 seconds after starting the reindex task.
> Therefore, exactly 9 batches of the size specified with messagesPerSecond
> are processed.
> When indexing 2 messages per second, the indexer manages to index the
> first 18 messages, when indexing 5 messages per second, the first 45
> messages are indexed, and so on...
>
> I don't know much about this part of the codebase and would be happy about
> help to debug this.
> Did somebody else ever see this error or can somebody else reproduce it?
>
> Best regards,
> Felix
> ---
> Gesellschaft für interkulturelles
> Zusammenleben gGmbH (GIZ)
> Felix Auringer
> IT
> Reformationsplatz 2
> 13597 Berlin
>
> Tel: 030/513 0100 00; Fax: 030/513 0100 09
> giz.berlin; felix.auringer@giz.berlin
>
> Amtsgericht Charlottenburg HRB 200872 B
> Geschäftsführerin: Dr. Britta Marschke
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Re: Postgres Timeouts
Posted by Benoit TELLIER via server-dev <se...@james.apache.org>.
Hello Felix
That's a classic Jooq / R2DBC issue: it badly support streaming as it hold a live connection.
The reindexing read live data as it needs it (so 2 message per second for instance) and keep it opens as it iterates/
The listing query then times out after 10s, thus the symptoms.
We encountered that in over parts of the PG implem and used buffering ( intermediate .collectList ) to pull results in memory, and return the driver thread as fast as we could.
The tricky point is that EG Cassandra driver supports reading large result sets without monopolizing driver resources. It thus don't need that memory buffering. Any fix doing this would then need not to impact Cassandra implementation that could be doing the optimal list-as-it-comes operation.
Another solution is to rethink "streaming" and switch it to explicit separated request per page size. This lift the time limit as we no longer hold a collection at the cost of repeated queries. A sample can be found here https://github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487
A generic adoption of this pattern can solve the problem you mention (at the cost of snapshot isolation which I believe we do not really care.
I'll try to propose a little something in that direction.
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Twake-Mail product.
Chairman of the Apache James project.
Mail: btellier@linagora.com
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
On Sep 25, 2026 12:51 PM, from Felix Auringer <fe...@giz.berlin>Hello,
For some time now, I have seen a timeout error in the postgres executor:
o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
This was mainly happening in TMail and was reported here: github.com/linagora/tmail-backend/issues/1599
I have now found a way to reproduce this reliably with the Docker image built from the current James master (bf100353ac3c4fc76d9ecced7472528b43e1ca21).
When using opensearch as the search backend and triggering a full reindex with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1', I observe the following behavior:
2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter - WebAdmin request received
2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer - Starting a full reindex
2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter - WebAdmin response received
2026-09-25T10:29:43.178Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
2026-09-25T10:29:44.120Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
2026-09-25T10:29:45.125Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
2026-09-25T10:29:46.128Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
2026-09-25T10:29:47.131Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
2026-09-25T10:29:48.135Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
2026-09-25T10:29:49.137Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
2026-09-25T10:29:50.147Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
2026-09-25T10:29:51.143Z [INFO ] o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance.
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer - Full reindex finished
2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker - Task was partially performed. Check logs for more details. Taskid : TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
This error always happens 10 seconds after starting the reindex task.
Therefore, exactly 9 batches of the size specified with messagesPerSecond are processed.
When indexing 2 messages per second, the indexer manages to index the first 18 messages, when indexing 5 messages per second, the first 45 messages are indexed, and so on...
I don't know much about this part of the codebase and would be happy about help to debug this.
Did somebody else ever see this error or can somebody else reproduce it?
Best regards,
Felix
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
giz.berlin; felix.auringer@giz.berlin
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org