You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "moguchev (via GitHub)" <gi...@apache.org> on 2025/11/27 15:41:09 UTC
[I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
moguchev opened a new issue, #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919
Version of driver: **v2.0.0** (latest)
Go version: **1.24**
Cassandra version: **5.0.4**
If set global `Session` `Timeout` equal `1*time.Second` or less in logs we can see a lot of messages:
```
"message":"Pool connection error.","addr":"10.239.171.143:9042","err":"read tcp 10.221.55.21:53299->10.229.171.143:9042: i/o timeout"
```
The same thing happens with long timeouts, but less often.
I tracked the full error flow:
`Conn.serve` -> `Conn.recv` -> `Conn.processFrame` -> `readHeader` -> `connReader.Read` -> `Conn.serve` -> `Conn.closeWithError` ->`hostConnPool.HandleError `
So the driver keeps getting read/timeout errors and closes the connection with error and establishes a new one (and so on all the time).
The most surprising thing is that Cassandra's response time doesn't exceed 40ms at 99.9p. The go client isn't under any load.
Also I've found, the Heartbeat connection is established with a general timeout, not a Connect timeout (which isn't entirely accurate). However, debugging has shown that this isn't the reason the connection is closed.
Constant client-side reconnects put a strain on Cassandra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "moguchev (via GitHub)" <gi...@apache.org>.
moguchev commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3790513438
I experimented. I added a patch to Cassandra `5.0.4` where I set a 1-second sleep in `PasswordAuthenticator`.
In the driver, I set ConnectTimeout = `5s` and Timeout = `200ms` in Session.
As a driver user, I expect `5s` to be used for connection requests and reconnects.
But I end up getting the following error:
```txt
Control connection error.
read tcp 10.201.125.63:46758->10.239.214.7:9042: i/o timeout.
```
I decided to check what your colleagues are doing with the ScyllaDB gocql fork. They don't have this problem: ConnectTimeout is always used to establish connections, both in control and in pool.
See: https://github.com/scylladb/gocql/releases/tag/v1.15.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] CASSGO-125 Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis closed issue #1919: CASSGO-125 Many "Pool connection error" with small Session.Timeout
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4489377324
The problem isn't submitting a PR, it's finding time (at least for me) to properly think through what we want to do with the timeout setting. Right now I'm working on different projects so I can't work on this right now. I believe I will be able to focus more on gocql in a couple of weeks. Maybe @worryg0d can help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3587034749
I think the main issue is that we are using the `Timeout` setting for both request level timeouts and connection read timeout. The connection read timeout should in general be quite high and it should be fine to close the connection when those timeouts happen. Users should only care about the request level timeout but there doesn't seem to be a session wide timeout config field just for those if I'm understanding correctly...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3938853752
Yeah what they did is aligned with what other drivers do and what we should probably do in gocql as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] CASSGO-125 Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4604507412
> I tracked the full error flow:
Conn.serve -> Conn.recv -> Conn.processFrame -> readHeader -> connReader.Read -> Conn.serve -> Conn.closeWithError ->hostConnPool.HandleError
Are you forcing protocol v4? This would be incorrect if you're using v5 (which would be the default when using C* >= 4.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "moguchev (via GitHub)" <gi...@apache.org>.
moguchev commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3586634931
> Did you start observing this behavior only after you upgraded to gocql 2.0.0?
No, this can also be observed on `v2.0.0-rc1`.
> Where are you measuring this? Are you sure that is the latency that gocql itself is seeing?
Response time is determined by Cassandra cluster monitoring data. The latency seen by gocql itself is unknown. I'll have to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "worryg0d (via GitHub)" <gi...@apache.org>.
worryg0d commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4497096538
I write a simple fix that reverts the previous behavior here https://github.com/worryg0d/gocql/commit/0b48aa7b70421859fb14ae502c3ccbee7a2ff4cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3587037638
But you're right that if we already have a heartbeat mechanism we probably should not close the connection when the connection read times out...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4500004179
I would prefer a different solution. Get rid of socket read deadline altogether and just use the config.Timeout value for the `callReq.timer`. Maybe we need to keep the deadline for the initial setup requests though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3586559318
> The most surprising thing is that Cassandra's response time doesn't exceed 40ms at 99.9p.
Where are you measuring this? Are you sure that is the latency that gocql itself is seeing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "moguchev (via GitHub)" <gi...@apache.org>.
moguchev commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4488055352
These changes from scylladb/gocql should resolve this problem:
- https://github.com/scylladb/gocql/pull/529 (issue: https://github.com/scylladb/gocql/issues/521)
- https://github.com/scylladb/gocql/pull/531 (issue: https://github.com/scylladb/gocql/issues/532)
- https://github.com/scylladb/gocql/pull/535
- https://github.com/scylladb/gocql/pull/540
Small Timeout on Session also can lead to `Control connection error.` and a lot of recconections in backgroud.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] CASSGO-125 Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4605508368
> I experimented. I added a patch to Cassandra `5.0.4` where I set a 1-second sleep in `PasswordAuthenticator`.
>
> In the driver, I set ConnectTimeout = `5s` and Timeout = `200ms` in Session.
>
> As a driver user, I expect `5s` to be used for connection requests and reconnects.
>
> But I end up getting the following error:
>
> Control connection error.
> read tcp 10.201.125.63:46758->10.239.214.7:9042: i/o timeout.
>
> I decided to check what your colleagues are doing with the ScyllaDB gocql fork. They don't have this problem: ConnectTimeout is always used to establish connections, both in control and in pool.
>
> See: https://github.com/scylladb/gocql/releases/tag/v1.15.3
And are you sure that this specific error didn't occur before you added the sleep to the authenticator? That error message could be for the normal recv() call after the initialization of the connection is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "moguchev (via GitHub)" <gi...@apache.org>.
moguchev commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4488086515
@joao-r-reis, when is this issue planned to be resolved? Or would it be better to submit my pull request for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "worryg0d (via GitHub)" <gi...@apache.org>.
worryg0d commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4489816319
I'm not familiar with the issue right now. I'll take a look at this tommorrow to see what we should do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "moguchev (via GitHub)" <gi...@apache.org>.
moguchev commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3586663230
What worries me most is that the driver has a hertbeat mechanism that removes connections from the pool due to errors (including timeouts). This mechanism should be the only one to close the connection in such cases. But it turns out that somewhere at the Conn level, an I/O timeout occurs, causing it to be closed...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3586563738
Did you start observing this behavior only after you upgraded to gocql 2.0.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "moguchev (via GitHub)" <gi...@apache.org>.
moguchev commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-3586780056
May be problem somewhere here:
```go
func (c *Conn) processFrame(ctx context.Context, r io.Reader) error {
// not safe for concurrent reads
// read a full header, ignore timeouts, as this is being ran in a loop
// TODO: TCP level deadlines? or just query level deadlines?
if c.r.GetTimeout() > 0 {
c.r.SetReadDeadline(time.Time{}) //nolint:errcheck
}
headStartTime := time.Now()
// were just reading headers over and over and copy bodies
head, err := readHeader(r, c.headerBuf[:])
headEndTime := time.Now()
if err != nil {
return err // here we get error i/o timeout
}
```
We set no timeout `c.r.SetReadDeadline(time.Time{})` to read a full header, ignore timeouts. But `readHeader` call `io.ReadFull(c.r)` which calls `(c *connReader) Read`:
```go
func (c *connReader) Read(p []byte) (n int, err error) {
const maxAttempts = 5
for i := 0; i < maxAttempts; i++ {
var nn int
if c.timeout > 0 {
c.conn.SetReadDeadline(time.Now().Add(c.timeout)) //nolint:errcheck
}
nn, err = io.ReadFull(c.r, p[n:])
n += nn
if err == nil {
break
}
if verr, ok := err.(net.Error); !ok || !verr.Temporary() { //nolint:staticcheck
break
}
}
return
}
```
`connReader.timeout` is not zero so we `SetReadDeadline` again...
May be it's ok logic now, I don't know...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "worryg0d (via GitHub)" <gi...@apache.org>.
worryg0d commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4496951288
The main problem right now is that the driver overrides the no-timeout contract for reading frame/segment headers as @moguchev pointed.
Before gocql v2, reading the response from conn was split into two steps:
1. Read the frame header directly from the raw conn without any timeout so `Conn.recv->readHeader()` hangs for data from the server
2. Read the rest of the frame using connReader with timeouts (there wasn't actually a connReader wrapper before; it was introduced in CASSGO-1)
To quickly fix this, we should revert the previous behavior in the first place.
> I think the main issue is that we are using the Timeout setting for both request level timeouts and connection read timeout.
It should not be a problem, connReader retries for 5 times before throwing io timeout so it should take 5x Timeout time to read data from the conn while the request timeout is 1x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
Re: [I] Many "Pool connection error" with small Session.Timeout [cassandra-gocql-driver]
Posted by "moguchev (via GitHub)" <gi...@apache.org>.
moguchev commented on issue #1919:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1919#issuecomment-4501821466
I spent a lot of time testing timeouts in the `scylladb/gocql` and `cassandra-gocql-driver` drivers on real clusters. I think `scylladb/gocql` has a rather interesting, flexible, and, most importantly, resilient implementation to low client `config.Timeout` values. I tried to bring the same logic here, adapting it along the way.
You can find it here: https://github.com/apache/cassandra-gocql-driver/pull/1953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org