You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "ut0mt8 (via GitHub)" <gi...@apache.org> on 2024/08/30 10:23:51 UTC

[I] Trouble with DCAware or RackAware HostSelectionPolicy [cassandra-gocql-driver]

ut0mt8 opened a new issue, #1807:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1807

   ### What version of Cassandra are you using?
   Cassandra 3.11.14 | CQL spec 3.4.4 | Native protocol v4
   
   ### What version of Gocql are you using?
   v1.6.0
   
   ### What version of Go are you using?
   go version go1.22.2
   
   ### What did you do?
   I got a C$ worlwide cluster divide in 3 regions (DC) and multiple AZ (Rack).
   I want to connect and read the most locally possible (rack)
   So I'm testing ; just connecting and few select to test where read and connection are distributed
   
   ### What did you expect to see?
   I'm expecting that query only goes to either my local DC or local Rack.
   
   ### What did you see instead?
   instead they are randomly distributed among the worldwide nodes randomly
   Note that with Hostfilter it's work (but it's not convenient at all). It's like the driver didn't understand my cluster topology; see clues in debug (2 below nodes IP means global if read correctly the code)
   
   ---
   
   ### Describe your Cassandra cluster
   please provide the following information
   
   - output of `nodetool status`
   
   `````
   Datacenter: ap-southeast-1
   ==========================
   Status=Up/Down
   |/ State=Normal/Leaving/Joining/Moving
   --  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
   UN  10.80.197.48   84.27 GiB  256          17.1%             4d1a1159-fc5d-431c-a420-8bff72ef2484  ap-southeast-1b
   UN  10.80.111.193  79.07 GiB  256          16.6%             0a69d219-f42f-4de9-9c8d-7efc06355e59  ap-southeast-1a
   UN  10.80.197.197  85.49 GiB  256          17.7%             bb87ff84-4b68-46d9-87a3-e1e8d0d6fe07  ap-southeast-1b
   UN  10.80.197.198  78.92 GiB  256          17.6%             db91859b-4759-4d69-ab99-65ffe97a9deb  ap-southeast-1b
   UN  10.80.197.103  79.37 GiB  256          16.5%             ab22efbd-585a-4c4e-adb9-b0ce4a56e8f0  ap-southeast-1b
   UN  10.80.197.104  66.52 GiB  256          14.6%             552d1660-fec5-4205-bfa6-992a30ec702c  ap-southeast-1b
   UN  10.80.197.121  77.52 GiB  256          16.6%             d326d322-ca5e-4d6d-b052-2971c6ea57cf  ap-southeast-1b
   UN  10.80.111.234  77.78 GiB  256          16.1%             e0974eb1-7b9b-487c-bcbe-8e16f50d6508  ap-southeast-1a
   UN  10.80.111.235  75.12 GiB  256          16.7%             29d1bcc3-4a60-477d-8980-d6b7083d2541  ap-southeast-1a
   UN  10.80.111.189  84.49 GiB  256          17.5%             6d7ea647-5a14-4462-9118-ce20ce6b4602  ap-southeast-1a
   UN  10.80.111.126  84.62 GiB  256          17.1%             544ce9c7-71ea-47c7-80c7-32dc06ed6b5e  ap-southeast-1a
   UN  10.80.111.190  74.33 GiB  256          16.1%             650975be-09cf-4db5-9a45-3f99a03bdfea  ap-southeast-1a
   Datacenter: eu-west-1
   =====================
   Status=Up/Down
   |/ State=Normal/Leaving/Joining/Moving
   --  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
   UN  10.0.111.32    99.37 GiB  256          33.6%             7f54a4b0-b30a-4043-854a-7efec3604463  eu-west-1a
   UN  10.0.111.145   96.86 GiB  256          32.3%             1222059b-bf71-4df9-ab78-2d9fd420f29b  eu-west-1a
   UN  10.0.111.17    105.54 GiB  256          35.0%             42d01053-1bbe-4fef-aed8-8f3bc201fab5  eu-west-1a
   UN  10.0.111.65    86.03 GiB  256          32.0%             f4a7b146-80a1-408e-a31a-893e3465a271  eu-west-1a
   UN  10.0.111.146   91.01 GiB  256          32.9%             485334d7-11c6-4716-bbe1-4f09db7c41eb  eu-west-1a
   UN  10.0.111.147   98.04 GiB  256          33.9%             62377c8a-61bd-4bb2-b434-c607a5eba1d2  eu-west-1a
   UN  10.0.197.246   95.4 GiB   256          32.4%             31905c62-dd32-4acc-b247-90899f8069ca  eu-west-1b
   UN  10.0.197.88    98.83 GiB  256          34.6%             8b42330a-e209-4e43-9174-e7b8c23eaa80  eu-west-1b
   UN  10.0.197.122   106.38 GiB  256          34.8%             1336edb6-5dec-4826-b263-4263b7ff24d3  eu-west-1b
   UN  10.0.197.74    94.17 GiB  256          32.4%             fe63b3cb-f129-4bd8-b879-63b74730d324  eu-west-1b
   UN  10.0.197.123   92.88 GiB  256          33.7%             440d4fa5-2f45-472d-8721-fbda44401804  eu-west-1b
   UN  10.0.197.124   89.06 GiB  256          32.4%             45ba9039-14e4-4082-9243-e5b7465be36f  eu-west-1b
   Datacenter: us-east-1
   =====================
   Status=Up/Down
   |/ State=Normal/Leaving/Joining/Moving
   --  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
   UN  10.66.201.210  102.07 GiB  256          24.8%             7f369bcf-12c1-4d3a-aae2-53fe06bbff06  us-east-1c
   UN  10.66.200.192  104.88 GiB  256          25.3%             dbd379f8-545a-412c-82e1-c218f19906c3  us-east-1d
   UN  10.66.111.5    102.18 GiB  256          24.4%             0148c92a-482a-4b5d-98df-d1ac88b40ffe  us-east-1a
   UN  10.66.200.74   113.37 GiB  256          27.0%             ec84e8b6-fc78-44df-9f18-aeb937858a69  us-east-1d
   UN  10.66.201.43   98.25 GiB  256          23.0%             0ba00e53-b02e-4423-94ce-3a5974098d16  us-east-1c
   UN  10.66.201.139  108.57 GiB  256          25.8%             8719fbc4-590c-49a1-b664-2be9ecd22d94  us-east-1c
   UN  10.66.200.40   107.59 GiB  256          25.8%             3f95a54f-d9a1-42e6-acdb-3661dfe858f7  us-east-1d
   UN  10.66.111.206  104.56 GiB  256          25.0%             68000a79-dfe7-4732-8aa1-4efc6199ff9b  us-east-1a
   UN  10.66.200.191  92.95 GiB  256          21.8%             d86d1e9d-dbd6-43a1-b20d-022bf301fcef  us-east-1d
   UN  10.66.201.140  111.44 GiB  256          26.5%             4e5b023d-ff58-4841-b3cc-ca799eeabc4c  us-east-1c
   UN  10.66.111.13   109.17 GiB  256          26.1%             0839e0af-5b21-4efe-aa21-27f67f0f5d07  us-east-1a
   UN  10.66.111.205  103.45 GiB  256          24.4%             9bf4075f-1563-4c7f-b812-89a31a95dc57  us-east-1a
   
   `````
   
   - output of `SELECT peer, rpc_address FROM system.peers`
   
   ``````
    peer          | rpc_address
   ---------------+---------------
     10.0.111.145 |  10.0.111.145
     10.66.111.13 |  10.66.111.13
    10.66.201.140 | 10.66.201.140
     10.66.200.40 |  10.66.200.40
     10.80.197.48 |  10.80.197.48
     10.0.197.124 |  10.0.197.124
      10.0.111.32 |   10.0.111.32
    10.80.197.198 | 10.80.197.198
    10.80.111.189 | 10.80.111.189
    10.66.200.192 | 10.66.200.192
    10.80.197.121 | 10.80.197.121
     10.66.200.74 |  10.66.200.74
     10.66.201.43 |  10.66.201.43
    10.66.201.210 | 10.66.201.210
    10.66.111.205 | 10.66.111.205
    10.80.111.126 | 10.80.111.126
    10.66.201.139 | 10.66.201.139
     10.0.197.122 |  10.0.197.122
      10.0.197.74 |   10.0.197.74
      10.66.111.5 |   10.66.111.5
     10.0.111.147 |  10.0.111.147
      10.0.197.88 |   10.0.197.88
    10.80.197.197 | 10.80.197.197
     10.0.197.123 |  10.0.197.123
      10.0.111.65 |   10.0.111.65
    10.80.197.104 | 10.80.197.104
    10.80.111.235 | 10.80.111.235
    10.66.111.206 | 10.66.111.206
     10.0.197.246 |  10.0.197.246
     10.0.111.146 |  10.0.111.146
    10.80.111.190 | 10.80.111.190
    10.80.111.193 | 10.80.111.193
    10.66.200.191 | 10.66.200.191
    10.80.111.234 | 10.80.111.234
    10.80.197.103 | 10.80.197.103
   
   ``````
   
   Code
   ```````
   package main
   
   import (
       "flag"
       "time"
   
       log "github.com/cihub/seelog"
       "github.com/gocql/gocql"
   )
   
   
   func main() {
   
       seeds := flag.String("seeds", "10.0.111.65", "seeds cnx")
       dc := flag.String("dc", "eu-west-1", "local dc")
       rack := flag.String("rack", "eu-west-1a", "local rack")
       flag.Parse()
   
       cluster := gocql.NewCluster(*seeds)
       cluster.Keyspace = "ks_user_db"
       cluster.Consistency = gocql.LocalOne
       cluster.ProtoVersion = 4
       cluster.CQLVersion = "3.4.4"
       cluster.PoolConfig.HostSelectionPolicy = gocql.TokenAwareHostPolicy(gocql.RackAwareRoundRobinPolicy(*dc, *rack))
       cluster.Authenticator = gocql.PasswordAuthenticator{Username: "cassandra", Password: "cassandra"}
   
       session, err := cluster.CreateSession()
       if err != nil {
           log.Critical(err)
       }
       defer session.Close()
   
       ids := []string{
           "cc7cb092ef52b3e9b2fdd5a8e382c5db589fe963",
           "809823a219b206462656be68b67f07e69f329773",
           "c395181acb6a24a5e8c0188c5f3587b0d1e37910",
       }
   
       for {
           for _, id := range ids {
               qry := session.Query("SELECT * FROM ks_user_db.user_db WHERE device_id=? and os =?", id, "ANDROID")
               output := qry.Iter()
               qry.Release()
               log.Infof("query result: %v %s\n", output)
           }
   
       }
   }
   ```````
   
   - rebuild your application with the `gocql_debug` tag and post the output
   
   output
   ``````
   
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 control: error executing "\n\t\tSELECT durable_writes, strategy_class, strategy_options\n\t\tFROM system.schema_keyspaces\n\t\tWHERE keyspace_name = ?": unconfigured table schema_keyspaces
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.111.65:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.111.32:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.197.123:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.111.145:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.197.246:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.197.74:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.111.147:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.111.146:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.197.122:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.111.17:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.197.88:9042
   2024/08/30 11:59:13 gocql: Session.handleNodeConnected: 10.0.197.124:9042
   2024/08/30 11:59:13 gocql: conns of pool after stopped "10.0.111.65": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.197.123": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.111.32": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.111.147": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.197.246": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.111.146": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.111.145": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.197.74": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.111.17": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.197.122": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.197.88": 2
   2024/08/30 11:59:14 gocql: conns of pool after stopped "10.0.197.124": 2
   2024/08/30 11:59:23 gocql: connection failed "10.66.201.43": dial tcp 10.66.201.43:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.197.198": dial tcp 10.80.197.198:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.201.210": dial tcp 10.66.201.210:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.111.189": dial tcp 10.80.111.189:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.200.74": dial tcp 10.66.200.74:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.111.234": dial tcp 10.80.111.234:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.111.5": dial tcp 10.66.111.5:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.200.192": dial tcp 10.66.200.192:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.201.139": dial tcp 10.66.201.139:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.111.205": dial tcp 10.66.111.205:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.197.103": dial tcp 10.80.197.103:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.197.197": dial tcp 10.80.197.197:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.197.104": dial tcp 10.80.197.104:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.111.193": dial tcp 10.80.111.193:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.111.13": dial tcp 10.66.111.13:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.200.40": dial tcp 10.66.200.40:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.201.140": dial tcp 10.66.201.140:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.197.48": dial tcp 10.80.197.48:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.200.191": dial tcp 10.66.200.191:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.197.121": dial tcp 10.80.197.121:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.111.126": dial tcp 10.80.111.126:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.66.111.206": dial tcp 10.66.111.206:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.111.235": dial tcp 10.80.111.235:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:23 gocql: connection failed "10.80.111.190": dial tcp 10.80.111.190:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.111.13": dial tcp 10.66.111.13:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.111.126": dial tcp 10.80.111.126:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.200.191": dial tcp 10.66.200.191:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.200.74": dial tcp 10.66.200.74:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.201.139": dial tcp 10.66.201.139:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.111.190": dial tcp 10.80.111.190:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.200.192": dial tcp 10.66.200.192:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.197.103": dial tcp 10.80.197.103:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.111.206": dial tcp 10.66.111.206:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.197.197": dial tcp 10.80.197.197:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.200.40": dial tcp 10.66.200.40:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.197.121": dial tcp 10.80.197.121:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.111.205": dial tcp 10.66.111.205:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.197.198": dial tcp 10.80.197.198:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.201.43": dial tcp 10.66.201.43:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.111.189": dial tcp 10.80.111.189:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.201.210": dial tcp 10.66.201.210:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.111.5": dial tcp 10.66.111.5:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.111.234": dial tcp 10.80.111.234:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.111.235": dial tcp 10.80.111.235:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.197.104": dial tcp 10.80.197.104:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.111.193": dial tcp 10.80.111.193:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.80.197.48": dial tcp 10.80.197.48:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:34 gocql: connection failed "10.66.201.140": dial tcp 10.66.201.140:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.111.13": dial tcp 10.66.111.13:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.111.205": dial tcp 10.66.111.205:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.197.104": dial tcp 10.80.197.104:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.111.234": dial tcp 10.80.111.234:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.111.206": dial tcp 10.66.111.206:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.200.191": dial tcp 10.66.200.191:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.201.139": dial tcp 10.66.201.139:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.111.189": dial tcp 10.80.111.189:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.111.190": dial tcp 10.80.111.190:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.201.140": dial tcp 10.66.201.140:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.197.197": dial tcp 10.80.197.197:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.197.198": dial tcp 10.80.197.198:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.200.40": dial tcp 10.66.200.40:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.201.210": dial tcp 10.66.201.210:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.197.121": dial tcp 10.80.197.121:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.197.103": dial tcp 10.80.197.103:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.201.43": dial tcp 10.66.201.43:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.111.5": dial tcp 10.66.111.5:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.200.74": dial tcp 10.66.200.74:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.197.48": dial tcp 10.80.197.48:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.111.193": dial tcp 10.80.111.193:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.111.235": dial tcp 10.80.111.235:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.66.200.192": dial tcp 10.66.200.192:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:45 gocql: connection failed "10.80.111.126": dial tcp 10.80.111.126:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.111.13\" peer=\"10.66.111.13\" rpc_address=\"10.66.111.13\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.111.13\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1a\" host_id=\"0839e0af-5b21-4efe-aa21-27f67f0f5d07\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.111.13:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.111.13": dial tcp 10.66.111.13:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.111.205\" peer=\"10.66.111.205\" rpc_address=\"10.66.111.205\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.111.205\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1a\" host_id=\"9bf4075f-1563-4c7f-b812-89a31a95dc57\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.111.205:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.111.205": dial tcp 10.66.111.205:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.111.126\" peer=\"10.80.111.126\" rpc_address=\"10.80.111.126\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.111.126\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1a\" host_id=\"544ce9c7-71ea-47c7-80c7-32dc06ed6b5e\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.111.126:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.111.126": dial tcp 10.80.111.126:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.197.198\" peer=\"10.80.197.198\" rpc_address=\"10.80.197.198\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.197.198\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1b\" host_id=\"db91859b-4759-4d69-ab99-65ffe97a9deb\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.197.198:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.197.198": dial tcp 10.80.197.198:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.200.40\" peer=\"10.66.200.40\" rpc_address=\"10.66.200.40\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.200.40\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1d\" host_id=\"3f95a54f-d9a1-42e6-acdb-3661dfe858f7\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.200.40:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.200.40": dial tcp 10.66.200.40:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.201.210\" peer=\"10.66.201.210\" rpc_address=\"10.66.201.210\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.201.210\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1c\" host_id=\"7f369bcf-12c1-4d3a-aae2-53fe06bbff06\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.201.210:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.201.210": dial tcp 10.66.201.210:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.201.139\" peer=\"10.66.201.139\" rpc_address=\"10.66.201.139\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.201.139\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1c\" host_id=\"8719fbc4-590c-49a1-b664-2be9ecd22d94\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.201.139:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.201.139": dial tcp 10.66.201.139:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.197.121\" peer=\"10.80.197.121\" rpc_address=\"10.80.197.121\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.197.121\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1b\" host_id=\"d326d322-ca5e-4d6d-b052-2971c6ea57cf\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.197.121:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.197.121": dial tcp 10.80.197.121:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.197.104\" peer=\"10.80.197.104\" rpc_address=\"10.80.197.104\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.197.104\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1b\" host_id=\"552d1660-fec5-4205-bfa6-992a30ec702c\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.197.104:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.197.104": dial tcp 10.80.197.104:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.197.103\" peer=\"10.80.197.103\" rpc_address=\"10.80.197.103\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.197.103\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1b\" host_id=\"ab22efbd-585a-4c4e-adb9-b0ce4a56e8f0\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.197.103:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.197.103": dial tcp 10.80.197.103:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.111.234\" peer=\"10.80.111.234\" rpc_address=\"10.80.111.234\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.111.234\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1a\" host_id=\"e0974eb1-7b9b-487c-bcbe-8e16f50d6508\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.111.234:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.111.234": dial tcp 10.80.111.234:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.111.193\" peer=\"10.80.111.193\" rpc_address=\"10.80.111.193\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.111.193\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1a\" host_id=\"0a69d219-f42f-4de9-9c8d-7efc06355e59\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.111.193:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.111.193": dial tcp 10.80.111.193:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.197.48\" peer=\"10.80.197.48\" rpc_address=\"10.80.197.48\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.197.48\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1b\" host_id=\"4d1a1159-fc5d-431c-a420-8bff72ef2484\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.197.48:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.197.48": dial tcp 10.80.197.48:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.111.206\" peer=\"10.66.111.206\" rpc_address=\"10.66.111.206\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.111.206\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1a\" host_id=\"68000a79-dfe7-4732-8aa1-4efc6199ff9b\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.111.206:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.111.206": dial tcp 10.66.111.206:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.111.235\" peer=\"10.80.111.235\" rpc_address=\"10.80.111.235\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.111.235\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1a\" host_id=\"29d1bcc3-4a60-477d-8980-d6b7083d2541\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.111.235:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.111.235": dial tcp 10.80.111.235:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.201.43\" peer=\"10.66.201.43\" rpc_address=\"10.66.201.43\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.201.43\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1c\" host_id=\"0ba00e53-b02e-4423-94ce-3a5974098d16\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.201.43:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.201.43": dial tcp 10.66.201.43:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.200.191\" peer=\"10.66.200.191\" rpc_address=\"10.66.200.191\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.200.191\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1d\" host_id=\"d86d1e9d-dbd6-43a1-b20d-022bf301fcef\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.200.191:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.200.191": dial tcp 10.66.200.191:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.200.192\" peer=\"10.66.200.192\" rpc_address=\"10.66.200.192\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.200.192\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1d\" host_id=\"dbd379f8-545a-412c-82e1-c218f19906c3\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.200.192:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.200.192": dial tcp 10.66.200.192:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.111.5\" peer=\"10.66.111.5\" rpc_address=\"10.66.111.5\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.111.5\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1a\" host_id=\"0148c92a-482a-4b5d-98df-d1ac88b40ffe\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.111.5:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.111.5": dial tcp 10.66.111.5:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.111.190\" peer=\"10.80.111.190\" rpc_address=\"10.80.111.190\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.111.190\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1a\" host_id=\"650975be-09cf-4db5-9a45-3f99a03bdfea\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.111.190:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.111.190": dial tcp 10.80.111.190:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.201.140\" peer=\"10.66.201.140\" rpc_address=\"10.66.201.140\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.201.140\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1c\" host_id=\"4e5b023d-ff58-4841-b3cc-ca799eeabc4c\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.201.140:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.201.140": dial tcp 10.66.201.140:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.66.200.74\" peer=\"10.66.200.74\" rpc_address=\"10.66.200.74\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.66.200.74\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"us-east-1\" rack=\"us-east-1d\" host_id=\"ec84e8b6-fc78-44df-9f18-aeb937858a69\" version=\"v3.11.14\" state=UP num_tokens=256]": dial tcp 10.66.200.74:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.66.200.74": dial tcp 10.66.200.74:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.111.189\" peer=\"10.80.111.189\" rpc_address=\"10.80.111.189\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.111.189\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1a\" host_id=\"6d7ea647-5a14-4462-9118-ce20ce6b4602\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.111.189:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.111.189": dial tcp 10.80.111.189:9042: i/o timeout
   2024/08/30 11:59:46 gocql: unable to dial "[HostInfo hostname=\"\" connectAddress=\"10.80.197.197\" peer=\"10.80.197.197\" rpc_address=\"10.80.197.197\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.80.197.197\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"ap-southeast-1\" rack=\"ap-southeast-1b\" host_id=\"bb87ff84-4b68-46d9-87a3-e1e8d0d6fe07\" version=\"v3.11.16\" state=UP num_tokens=256]": dial tcp 10.80.197.197:9042: i/o timeout
   2024/08/30 11:59:46 gocql: filling stopped "10.80.197.197": dial tcp 10.80.197.197:9042: i/o timeout
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.201.43": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.201.43:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.201.210": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.201.210:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.111.193": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.111.193:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.200.74": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.200.74:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.200.40": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.200.40:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.197.198": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.197.198:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.197.48": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.197.48:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.200.192": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.200.192:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.111.5": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.111.5:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.197.197": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.197.197:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.111.126": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.111.126:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.197.103": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.197.103:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.200.191": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.200.191:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.201.140": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.201.140:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.111.235": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.111.235:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.201.139": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.201.139:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.197.104": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.197.104:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.111.190": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.111.190:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.111.234": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.111.234:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.111.205": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.111.205:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.197.121": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.197.121:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.80.111.189": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.80.111.189:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.111.13": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.111.13:9042
   2024/08/30 11:59:46 gocql: conns of pool after stopped "10.66.111.206": 0
   2024/08/30 11:59:46 gocql: Session.handleNodeDown: 10.66.111.206:9042
   1725011987191345405 [Info] query result: &{<nil> 0 {1 [] [{ks_user_db user_db device_id {4 13 }} {ks_user_db user_db os {4 13 }} {ks_user_db user_db age_group {4 13 }} {ks_user_db user_db carrier {4 1
   3 }} {ks_user_db user_db device_model {4 13 }} {ks_user_db user_db gender {4 13 }} {ks_user_db user_db home_city {4 13 }} {ks_user_db user_db home_country {4 13 }} {ks_user_db user_db rtb_frequency {4
    9 }}] 9 9} 0 <nil> 0xc00034ad80 0xc00019a180 0} %!s(MISSING)
   
   1725011987487125615 [Info] query result: &{<nil> 0 {1 [] [{ks_user_db user_db device_id {4 13 }} {ks_user_db user_db os {4 13 }} {ks_user_db user_db age_group {4 13 }} {ks_user_db user_db carrier {4 1
   3 }} {ks_user_db user_db device_model {4 13 }} {ks_user_db user_db gender {4 13 }} {ks_user_db user_db home_city {4 13 }} {ks_user_db user_db home_country {4 13 }} {ks_user_db user_db rtb_frequency {4
    9 }}] 9 9} 0 <nil> 0xc00034a180 0xc00019a480 0} %!s(MISSING)
   
   ``````
   
   lot of things to say on this trace:
   1/ we can see that the system schema is wrongly guesses !?
   2/ then it connects to all EU nodes (successfully) but see they do not appear as local (2) number that seems indicating other if I read the code correctly
   3/ then lot of timeout to cql port on other regions (there are close, it's what make our clients being datacenter local)
   
   I'm wondering what I'm doing bad


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [I] Trouble with DCAware or RackAware HostSelectionPolicy [cassandra-gocql-driver]

Posted by "ut0mt8 (via GitHub)" <gi...@apache.org>.
ut0mt8 commented on issue #1807:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1807#issuecomment-2454912174

   Oups sorry for not having closed this issue. Indeed I tested it more in depth and it works perfectly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [I] Trouble with DCAware or RackAware HostSelectionPolicy [cassandra-gocql-driver]

Posted by "lukasz-antoniak (via GitHub)" <gi...@apache.org>.
lukasz-antoniak commented on issue #1807:
URL: https://github.com/apache/cassandra-gocql-driver/issues/1807#issuecomment-2454547902

   i have tested the `RackAwareRoundRobinPolicy` locally with 9 node cluster across 3 DCs, it works as expected. What makes you conclude that requests are routed randomly to remote DCs? Regarding second point, "2" represents here number of connections in the pool to certain node.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Re: [I] Trouble with DCAware or RackAware HostSelectionPolicy [cassandra-gocql-driver]

Posted by "ut0mt8 (via GitHub)" <gi...@apache.org>.
ut0mt8 closed issue #1807: Trouble with DCAware or RackAware HostSelectionPolicy
URL: https://github.com/apache/cassandra-gocql-driver/issues/1807


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org