You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Andrija Panic <an...@gmail.com> on 2015/06/08 14:42:49 UTC
[HELP needed] haproxy vs mysql client timeout
Hi,
I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
seems it's loosing connection to the haproxy/mysql, since we are using
haproxy VIP to in db.properties, and this haproxy is using 1 galera node in
the backend section...(2 others are backup etc).
haproxy timeout is 100sec.
mysql timeouts are all defaults, except innodb_lock_wait_timeout.
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 600 |
| innodb_rollback_on_timeout | ON |
| interactive_timeout | 28800 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
To make things more interesting, all 3 servers (acs mgmt, haproxy,
galera-node1) are on the same physical host, so I rule out network
issues... and I dont see anything interesting in the haproxy logs, also no
backend downtimes etc...
The errors are like folowing, from mgmt logs, but after that mgmt server
continues to work fine...
Any suggestions on tuning timeouts ?
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet successfully received from the server was 137,744
milliseconds ago. The last packet sent successfully to the server was 1
milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
at
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
... 61 more
Caused by: java.io.EOFException: Can not read response from server.
Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
lost.
... 76 more
Thanks,
--
Andrija Panić
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Andrija Panic <an...@gmail.com>.
Actually, no...
On haproxy I have allowed all traffic with input interface matching the
internal interface (eth0), rest of nodes no firewall whatsoever (which will
be changed)..
On 8 June 2015 at 15:17, Rafael Fonseca <rs...@gmail.com> wrote:
> Do you have a firewall in between that might be closing long or idle
> sessions?
>
> On Mon, Jun 8, 2015 at 3:14 PM, Andrija Panic <an...@gmail.com>
> wrote:
>
> > Will try Simon.
> >
> > I expect no latency issues, since these 3 VMs are all on same
> > bridge/physical host. Other 2 Galera nodes are remote, but still low
> > latency - and zero seconds backend downtime...
> >
> > Not sure where to dig further...
> >
> > Thx
> >
> >
> >
> > On 8 June 2015 at 15:05, Simon Weller <sw...@ena.com> wrote:
> >
> > > Can you point the mgmt server directly at a single Galera node
> (removing
> > > haproxy), and see if the problem persists?
> > >
> > > Is your Galera cluster healthy? Could you be seeing latency on commit?
> > >
> > >
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Monday, June 8, 2015 7:42 AM
> > > To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> > > Subject: [HELP needed] haproxy vs mysql client timeout
> > >
> > > Hi,
> > >
> > > I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
> > > seems it's loosing connection to the haproxy/mysql, since we are using
> > > haproxy VIP to in db.properties, and this haproxy is using 1 galera
> node
> > in
> > > the backend section...(2 others are backup etc).
> > >
> > >
> > > haproxy timeout is 100sec.
> > > mysql timeouts are all defaults, except innodb_lock_wait_timeout.
> > >
> > > +-----------------------------+----------+
> > > | Variable_name | Value |
> > > +-----------------------------+----------+
> > > | connect_timeout | 10 |
> > > | delayed_insert_timeout | 300 |
> > > | have_statement_timeout | YES |
> > > | innodb_flush_log_at_timeout | 1 |
> > > | innodb_lock_wait_timeout | 600 |
> > > | innodb_rollback_on_timeout | ON |
> > > | interactive_timeout | 28800 |
> > > | net_read_timeout | 30 |
> > > | net_write_timeout | 60 |
> > > | thread_pool_idle_timeout | 60 |
> > > | wait_timeout | 28800 |
> > > +-----------------------------+----------+
> > >
> > > To make things more interesting, all 3 servers (acs mgmt, haproxy,
> > > galera-node1) are on the same physical host, so I rule out network
> > > issues... and I dont see anything interesting in the haproxy logs, also
> > no
> > > backend downtimes etc...
> > >
> > >
> > > The errors are like folowing, from mgmt logs, but after that mgmt
> server
> > > continues to work fine...
> > >
> > > Any suggestions on tuning timeouts ?
> > >
> > >
> > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > > Communications link failure
> > >
> > > The last packet successfully received from the server was 137,744
> > > milliseconds ago. The last packet sent successfully to the server was
> 1
> > > milliseconds ago.
> > > at
> > sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
> > > Source)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > > at
> > >
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> > > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
> > > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
> > > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
> > > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> > > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > > at
> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
> > > at
> > >
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
> > > at
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
> > > at
> > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > > at
> > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > > at
> > > com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
> > > ... 61 more
> > > Caused by: java.io.EOFException: Can not read response from server.
> > > Expected to read 4 bytes, read 0 bytes before connection was
> unexpectedly
> > > lost.
> > > ... 76 more
> > >
> > > Thanks,
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
--
Andrija Panić
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Andrija Panic <an...@gmail.com>.
Actually, no...
On haproxy I have allowed all traffic with input interface matching the
internal interface (eth0), rest of nodes no firewall whatsoever (which will
be changed)..
On 8 June 2015 at 15:17, Rafael Fonseca <rs...@gmail.com> wrote:
> Do you have a firewall in between that might be closing long or idle
> sessions?
>
> On Mon, Jun 8, 2015 at 3:14 PM, Andrija Panic <an...@gmail.com>
> wrote:
>
> > Will try Simon.
> >
> > I expect no latency issues, since these 3 VMs are all on same
> > bridge/physical host. Other 2 Galera nodes are remote, but still low
> > latency - and zero seconds backend downtime...
> >
> > Not sure where to dig further...
> >
> > Thx
> >
> >
> >
> > On 8 June 2015 at 15:05, Simon Weller <sw...@ena.com> wrote:
> >
> > > Can you point the mgmt server directly at a single Galera node
> (removing
> > > haproxy), and see if the problem persists?
> > >
> > > Is your Galera cluster healthy? Could you be seeing latency on commit?
> > >
> > >
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Monday, June 8, 2015 7:42 AM
> > > To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> > > Subject: [HELP needed] haproxy vs mysql client timeout
> > >
> > > Hi,
> > >
> > > I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
> > > seems it's loosing connection to the haproxy/mysql, since we are using
> > > haproxy VIP to in db.properties, and this haproxy is using 1 galera
> node
> > in
> > > the backend section...(2 others are backup etc).
> > >
> > >
> > > haproxy timeout is 100sec.
> > > mysql timeouts are all defaults, except innodb_lock_wait_timeout.
> > >
> > > +-----------------------------+----------+
> > > | Variable_name | Value |
> > > +-----------------------------+----------+
> > > | connect_timeout | 10 |
> > > | delayed_insert_timeout | 300 |
> > > | have_statement_timeout | YES |
> > > | innodb_flush_log_at_timeout | 1 |
> > > | innodb_lock_wait_timeout | 600 |
> > > | innodb_rollback_on_timeout | ON |
> > > | interactive_timeout | 28800 |
> > > | net_read_timeout | 30 |
> > > | net_write_timeout | 60 |
> > > | thread_pool_idle_timeout | 60 |
> > > | wait_timeout | 28800 |
> > > +-----------------------------+----------+
> > >
> > > To make things more interesting, all 3 servers (acs mgmt, haproxy,
> > > galera-node1) are on the same physical host, so I rule out network
> > > issues... and I dont see anything interesting in the haproxy logs, also
> > no
> > > backend downtimes etc...
> > >
> > >
> > > The errors are like folowing, from mgmt logs, but after that mgmt
> server
> > > continues to work fine...
> > >
> > > Any suggestions on tuning timeouts ?
> > >
> > >
> > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > > Communications link failure
> > >
> > > The last packet successfully received from the server was 137,744
> > > milliseconds ago. The last packet sent successfully to the server was
> 1
> > > milliseconds ago.
> > > at
> > sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
> > > Source)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > > at
> > >
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> > > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
> > > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
> > > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
> > > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> > > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > > at
> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
> > > at
> > >
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
> > > at
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
> > > at
> > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > > at
> > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > > at
> > > com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
> > > ... 61 more
> > > Caused by: java.io.EOFException: Can not read response from server.
> > > Expected to read 4 bytes, read 0 bytes before connection was
> unexpectedly
> > > lost.
> > > ... 76 more
> > >
> > > Thanks,
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
--
Andrija Panić
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Rafael Fonseca <rs...@gmail.com>.
Do you have a firewall in between that might be closing long or idle
sessions?
On Mon, Jun 8, 2015 at 3:14 PM, Andrija Panic <an...@gmail.com>
wrote:
> Will try Simon.
>
> I expect no latency issues, since these 3 VMs are all on same
> bridge/physical host. Other 2 Galera nodes are remote, but still low
> latency - and zero seconds backend downtime...
>
> Not sure where to dig further...
>
> Thx
>
>
>
> On 8 June 2015 at 15:05, Simon Weller <sw...@ena.com> wrote:
>
> > Can you point the mgmt server directly at a single Galera node (removing
> > haproxy), and see if the problem persists?
> >
> > Is your Galera cluster healthy? Could you be seeing latency on commit?
> >
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Monday, June 8, 2015 7:42 AM
> > To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> > Subject: [HELP needed] haproxy vs mysql client timeout
> >
> > Hi,
> >
> > I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
> > seems it's loosing connection to the haproxy/mysql, since we are using
> > haproxy VIP to in db.properties, and this haproxy is using 1 galera node
> in
> > the backend section...(2 others are backup etc).
> >
> >
> > haproxy timeout is 100sec.
> > mysql timeouts are all defaults, except innodb_lock_wait_timeout.
> >
> > +-----------------------------+----------+
> > | Variable_name | Value |
> > +-----------------------------+----------+
> > | connect_timeout | 10 |
> > | delayed_insert_timeout | 300 |
> > | have_statement_timeout | YES |
> > | innodb_flush_log_at_timeout | 1 |
> > | innodb_lock_wait_timeout | 600 |
> > | innodb_rollback_on_timeout | ON |
> > | interactive_timeout | 28800 |
> > | net_read_timeout | 30 |
> > | net_write_timeout | 60 |
> > | thread_pool_idle_timeout | 60 |
> > | wait_timeout | 28800 |
> > +-----------------------------+----------+
> >
> > To make things more interesting, all 3 servers (acs mgmt, haproxy,
> > galera-node1) are on the same physical host, so I rule out network
> > issues... and I dont see anything interesting in the haproxy logs, also
> no
> > backend downtimes etc...
> >
> >
> > The errors are like folowing, from mgmt logs, but after that mgmt server
> > continues to work fine...
> >
> > Any suggestions on tuning timeouts ?
> >
> >
> > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > Communications link failure
> >
> > The last packet successfully received from the server was 137,744
> > milliseconds ago. The last packet sent successfully to the server was 1
> > milliseconds ago.
> > at
> sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
> > Source)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > at
> > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > at
> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
> > at
> >
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
> > at
> >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > at
> >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > at
> > com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
> > ... 61 more
> > Caused by: java.io.EOFException: Can not read response from server.
> > Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
> > lost.
> > ... 76 more
> >
> > Thanks,
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Rafael Fonseca <rs...@gmail.com>.
Do you have a firewall in between that might be closing long or idle
sessions?
On Mon, Jun 8, 2015 at 3:14 PM, Andrija Panic <an...@gmail.com>
wrote:
> Will try Simon.
>
> I expect no latency issues, since these 3 VMs are all on same
> bridge/physical host. Other 2 Galera nodes are remote, but still low
> latency - and zero seconds backend downtime...
>
> Not sure where to dig further...
>
> Thx
>
>
>
> On 8 June 2015 at 15:05, Simon Weller <sw...@ena.com> wrote:
>
> > Can you point the mgmt server directly at a single Galera node (removing
> > haproxy), and see if the problem persists?
> >
> > Is your Galera cluster healthy? Could you be seeing latency on commit?
> >
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Monday, June 8, 2015 7:42 AM
> > To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> > Subject: [HELP needed] haproxy vs mysql client timeout
> >
> > Hi,
> >
> > I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
> > seems it's loosing connection to the haproxy/mysql, since we are using
> > haproxy VIP to in db.properties, and this haproxy is using 1 galera node
> in
> > the backend section...(2 others are backup etc).
> >
> >
> > haproxy timeout is 100sec.
> > mysql timeouts are all defaults, except innodb_lock_wait_timeout.
> >
> > +-----------------------------+----------+
> > | Variable_name | Value |
> > +-----------------------------+----------+
> > | connect_timeout | 10 |
> > | delayed_insert_timeout | 300 |
> > | have_statement_timeout | YES |
> > | innodb_flush_log_at_timeout | 1 |
> > | innodb_lock_wait_timeout | 600 |
> > | innodb_rollback_on_timeout | ON |
> > | interactive_timeout | 28800 |
> > | net_read_timeout | 30 |
> > | net_write_timeout | 60 |
> > | thread_pool_idle_timeout | 60 |
> > | wait_timeout | 28800 |
> > +-----------------------------+----------+
> >
> > To make things more interesting, all 3 servers (acs mgmt, haproxy,
> > galera-node1) are on the same physical host, so I rule out network
> > issues... and I dont see anything interesting in the haproxy logs, also
> no
> > backend downtimes etc...
> >
> >
> > The errors are like folowing, from mgmt logs, but after that mgmt server
> > continues to work fine...
> >
> > Any suggestions on tuning timeouts ?
> >
> >
> > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > Communications link failure
> >
> > The last packet successfully received from the server was 137,744
> > milliseconds ago. The last packet sent successfully to the server was 1
> > milliseconds ago.
> > at
> sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
> > Source)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > at
> > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > at
> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
> > at
> >
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
> > at
> >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > at
> >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > at
> > com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
> > ... 61 more
> > Caused by: java.io.EOFException: Can not read response from server.
> > Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
> > lost.
> > ... 76 more
> >
> > Thanks,
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Andrija Panic <an...@gmail.com>.
Will try Simon.
I expect no latency issues, since these 3 VMs are all on same
bridge/physical host. Other 2 Galera nodes are remote, but still low
latency - and zero seconds backend downtime...
Not sure where to dig further...
Thx
On 8 June 2015 at 15:05, Simon Weller <sw...@ena.com> wrote:
> Can you point the mgmt server directly at a single Galera node (removing
> haproxy), and see if the problem persists?
>
> Is your Galera cluster healthy? Could you be seeing latency on commit?
>
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Monday, June 8, 2015 7:42 AM
> To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> Subject: [HELP needed] haproxy vs mysql client timeout
>
> Hi,
>
> I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
> seems it's loosing connection to the haproxy/mysql, since we are using
> haproxy VIP to in db.properties, and this haproxy is using 1 galera node in
> the backend section...(2 others are backup etc).
>
>
> haproxy timeout is 100sec.
> mysql timeouts are all defaults, except innodb_lock_wait_timeout.
>
> +-----------------------------+----------+
> | Variable_name | Value |
> +-----------------------------+----------+
> | connect_timeout | 10 |
> | delayed_insert_timeout | 300 |
> | have_statement_timeout | YES |
> | innodb_flush_log_at_timeout | 1 |
> | innodb_lock_wait_timeout | 600 |
> | innodb_rollback_on_timeout | ON |
> | interactive_timeout | 28800 |
> | net_read_timeout | 30 |
> | net_write_timeout | 60 |
> | thread_pool_idle_timeout | 60 |
> | wait_timeout | 28800 |
> +-----------------------------+----------+
>
> To make things more interesting, all 3 servers (acs mgmt, haproxy,
> galera-node1) are on the same physical host, so I rule out network
> issues... and I dont see anything interesting in the haproxy logs, also no
> backend downtimes etc...
>
>
> The errors are like folowing, from mgmt logs, but after that mgmt server
> continues to work fine...
>
> Any suggestions on tuning timeouts ?
>
>
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>
> The last packet successfully received from the server was 137,744
> milliseconds ago. The last packet sent successfully to the server was 1
> milliseconds ago.
> at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
> Source)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
> at
>
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
> at
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
> at
>
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> at
>
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> at
> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
> ... 61 more
> Caused by: java.io.EOFException: Can not read response from server.
> Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
> lost.
> ... 76 more
>
> Thanks,
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Andrija Panic <an...@gmail.com>.
Will try Simon.
I expect no latency issues, since these 3 VMs are all on same
bridge/physical host. Other 2 Galera nodes are remote, but still low
latency - and zero seconds backend downtime...
Not sure where to dig further...
Thx
On 8 June 2015 at 15:05, Simon Weller <sw...@ena.com> wrote:
> Can you point the mgmt server directly at a single Galera node (removing
> haproxy), and see if the problem persists?
>
> Is your Galera cluster healthy? Could you be seeing latency on commit?
>
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Monday, June 8, 2015 7:42 AM
> To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> Subject: [HELP needed] haproxy vs mysql client timeout
>
> Hi,
>
> I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
> seems it's loosing connection to the haproxy/mysql, since we are using
> haproxy VIP to in db.properties, and this haproxy is using 1 galera node in
> the backend section...(2 others are backup etc).
>
>
> haproxy timeout is 100sec.
> mysql timeouts are all defaults, except innodb_lock_wait_timeout.
>
> +-----------------------------+----------+
> | Variable_name | Value |
> +-----------------------------+----------+
> | connect_timeout | 10 |
> | delayed_insert_timeout | 300 |
> | have_statement_timeout | YES |
> | innodb_flush_log_at_timeout | 1 |
> | innodb_lock_wait_timeout | 600 |
> | innodb_rollback_on_timeout | ON |
> | interactive_timeout | 28800 |
> | net_read_timeout | 30 |
> | net_write_timeout | 60 |
> | thread_pool_idle_timeout | 60 |
> | wait_timeout | 28800 |
> +-----------------------------+----------+
>
> To make things more interesting, all 3 servers (acs mgmt, haproxy,
> galera-node1) are on the same physical host, so I rule out network
> issues... and I dont see anything interesting in the haproxy logs, also no
> backend downtimes etc...
>
>
> The errors are like folowing, from mgmt logs, but after that mgmt server
> continues to work fine...
>
> Any suggestions on tuning timeouts ?
>
>
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>
> The last packet successfully received from the server was 137,744
> milliseconds ago. The last packet sent successfully to the server was 1
> milliseconds ago.
> at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
> Source)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
> at
>
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
> at
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
> at
>
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> at
>
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> at
> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
> ... 61 more
> Caused by: java.io.EOFException: Can not read response from server.
> Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
> lost.
> ... 76 more
>
> Thanks,
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Simon Weller <sw...@ena.com>.
Can you point the mgmt server directly at a single Galera node (removing haproxy), and see if the problem persists?
Is your Galera cluster healthy? Could you be seeing latency on commit?
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Monday, June 8, 2015 7:42 AM
To: users@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: [HELP needed] haproxy vs mysql client timeout
Hi,
I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
seems it's loosing connection to the haproxy/mysql, since we are using
haproxy VIP to in db.properties, and this haproxy is using 1 galera node in
the backend section...(2 others are backup etc).
haproxy timeout is 100sec.
mysql timeouts are all defaults, except innodb_lock_wait_timeout.
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 600 |
| innodb_rollback_on_timeout | ON |
| interactive_timeout | 28800 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
To make things more interesting, all 3 servers (acs mgmt, haproxy,
galera-node1) are on the same physical host, so I rule out network
issues... and I dont see anything interesting in the haproxy logs, also no
backend downtimes etc...
The errors are like folowing, from mgmt logs, but after that mgmt server
continues to work fine...
Any suggestions on tuning timeouts ?
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet successfully received from the server was 137,744
milliseconds ago. The last packet sent successfully to the server was 1
milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
at
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
... 61 more
Caused by: java.io.EOFException: Can not read response from server.
Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
lost.
... 76 more
Thanks,
--
Andrija Panić
Re: [HELP needed] haproxy vs mysql client timeout
Posted by Simon Weller <sw...@ena.com>.
Can you point the mgmt server directly at a single Galera node (removing haproxy), and see if the problem persists?
Is your Galera cluster healthy? Could you be seeing latency on commit?
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Monday, June 8, 2015 7:42 AM
To: users@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: [HELP needed] haproxy vs mysql client timeout
Hi,
I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
seems it's loosing connection to the haproxy/mysql, since we are using
haproxy VIP to in db.properties, and this haproxy is using 1 galera node in
the backend section...(2 others are backup etc).
haproxy timeout is 100sec.
mysql timeouts are all defaults, except innodb_lock_wait_timeout.
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 600 |
| innodb_rollback_on_timeout | ON |
| interactive_timeout | 28800 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
To make things more interesting, all 3 servers (acs mgmt, haproxy,
galera-node1) are on the same physical host, so I rule out network
issues... and I dont see anything interesting in the haproxy logs, also no
backend downtimes etc...
The errors are like folowing, from mgmt logs, but after that mgmt server
continues to work fine...
Any suggestions on tuning timeouts ?
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet successfully received from the server was 137,744
milliseconds ago. The last packet sent successfully to the server was 1
milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
at
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
... 61 more
Caused by: java.io.EOFException: Can not read response from server.
Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
lost.
... 76 more
Thanks,
--
Andrija Panić