You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Gary Dixon <Ga...@quadris.co.uk.INVALID> on 2022/12/20 13:55:13 UTC
database connection lost
Hi all
CS v4.15.2
Hypervisor = KVM
We have started to experience an issue on a frequent basis where the CS mgmt. service is losing the connection to the database and believe it is being caused by mysqldumps locking the tables.
Looking at the db.properties file there are tuning parameters :
# CloudStack database tuning parameters
db.cloud.maxActive=250
db.cloud.maxIdle=30
db.cloud.maxWait=10000
db.cloud.validationQuery=SELECT 1
db.cloud.testOnBorrow=true
db.cloud.testWhileIdle=true
db.cloud.timeBetweenEvictionRunsMillis=40000
db.cloud.minEvictableIdleTimeMillis=240000
db.cloud.poolPreparedStatements=false
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
would changing the 'maxwait' value to a higher value stop the mgmt. service losing the db connection whilst dumps are taken ?
BR
Gary
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: vms@quadris-support.com
W: www.quadris.co.uk
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
RE: database connection lost
Posted by Gary Dixon <Ga...@quadris.co.uk.INVALID>.
Thanks all
I just checked and mysql wait timeout is set to 28800
The strange thing is we take mysqldumps every 15 minutes - but we see the db connection only gets lost around every 7 and half hours apart not every 15 minutes.
I am going to set the mysql wait_timeout to 3600 as suggested in the last post in the article that Wei posted https://github.com/apache/cloudstack/issues/4657
Subject: Re: database connection lost
Hi Gary,
We had the same problem and increasing the wait timeout solved it.
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: vms@quadris-support.com
W: www.quadris.co.uk
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
________________________________
From: Gary Dixon <Ga...@quadris.co.uk.INVALID>
Sent: Tuesday, December 20, 2022, 9:31 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: database connection lost
Thanks Wei
Yes we are seeing the exact same errors in the cs mgmt. logs - we will set the MySQL global wait timeout to a higher value and see if it resolves our issue
What's the error message ?
There was a github issue may help you:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcloudstack%2Fissues%2F4657&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7C605bb13402834b9e42bd08dae297f382%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638071439517503224%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XUINQ77haPeXi7%2BsRx%2BZvnT2OGxmqh0LIpLaB4y7wOM%3D&reserved=0
-Wei
On Tuesday, 20 December 2022, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi Vivek
>
> We are using the --single-transaction flag and --skip-lock-tables flag
> Could there be something other than mysqldumps causing the cs mgmt.
> server to be losing the connection to the database ?
>
>
> BR
>
> Gary
>
>
> Hello,
>
> You can use --single-transaction
> Option while taking the dump or you can schedule to take dump from a
> slave server if you have.
>
> Regards,
> Vivek Kumar
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> %2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7C605bb13402834b9e42bd08
> dae297f382%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63807143951765
> 9442%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT
> iI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lxW6c1uPtRfxmKZxhu6Jmg
> PJOlnC0hhKOOuMhIM3zO8%3D&reserved=0
> uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e
> 4149de9c5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63
> 8071430711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ClDbnphfgf
> WJtD37XnVSa2U1n6EQOtTVYBYbUBCsAJc%3D&reserved=0
> The information contained in this e-mail from Quadris may be
> confidential and privileged for the private use of the named
> recipient. The contents of this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please
> destroy any hard copies and delete this message.
>
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: v<tel:+44%207989717661>ms@quadris‑support.com
W: https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.quadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7C605bb13402834b9e42bd08dae297f382%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638071439517659442%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=trP%2BLqxu6N08d0Yu600pNE3y2y1Wuv0pb44t2n%2BTf6E%3D&reserved=0
[cid:image011498.png@FB84B645.38DDA861]
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
On Tue, Dec 20, 2022, 7:25 PM Gary Dixon <Gary.Dixon@quadris.co.uk.
> invalid>
> wrote:
>
> > Hi all
> >
> >
> >
> > CS v4.15.2
> >
> > Hypervisor = KVM
> >
> >
> >
> > We have started to experience an issue on a frequent basis where the
> > CS mgmt. service is losing the connection to the database and
> > believe it is being caused by mysqldumps locking the tables.
> >
> > Looking at the db.properties file there are tuning parameters :
> >
> > # CloudStack database tuning parameters
> >
> > db.cloud.maxActive=250
> >
> > db.cloud.maxIdle=30
> >
> > db.cloud.maxWait=10000
> >
> > db.cloud.validationQuery=SELECT 1
> >
> > db.cloud.testOnBorrow=true
> >
> > db.cloud.testWhileIdle=true
> >
> > db.cloud.timeBetweenEvictionRunsMillis=40000
> >
> > db.cloud.minEvictableIdleTimeMillis=240000
> >
> > db.cloud.poolPreparedStatements=false
> >
> >
> > db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessio
> > nV
> > ariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,
> > ER
> > ROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
> >
> >
> >
> > would changing the ‘maxwait’ value to a higher value stop the mgmt.
> > service losing the db connection whilst dumps are taken ?
> >
> > BR
> >
> >
> >
> > Gary
> >
> >
> >
> >
> > Gary Dixon
> > Senior Technical Consultant
> > T: +44 161 537 4990
> > E: *v* <+44%207989717661>ms@quadris‑support.com
> > W:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww
> > .q%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e4149de9c
> > 5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63807143
> > 0711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
> > zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mt9MlK3EMSW1
> > shNbTq%2B4Z9JseyTJw1Ynah305yJeq4s%3D&reserved=0
> > uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc5084879
> > 20
> > c948fa441908dae2949e38%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C
> > 63
> > 8071425209591727%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo
> > iV
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SAj0nkOY
> > 53
> > 6UUokgQPzXfdGH6Jp2pJ1hxlqA5hAJiwY%3D&reserved=0
> > The information contained in this e-mail from Quadris may be
> > confidential and privileged for the private use of the named
> > recipient. The contents of this e-mail may not necessarily represent
> > the
> official views of Quadris.
> > If you have received this information in error you must not copy,
> > distribute or take any action or reliance on its contents. Please
> > destroy any hard copies and delete this message.
> >
>
> --
> This message is intended only for the use of the individual or entity
> to which it is addressed and may contain confidential and/or
> privileged information. If you are not the intended recipient, please
> delete the original message and any copy of it from your computer
> system. You are hereby notified that any dissemination, distribution
> or copying of this communication is strictly prohibited unless proper
> authorization has been obtained for such action. If you have received
> this communication in error, please notify the sender immediately.
> Although IndiQus attempts to sweep e-mail and attachments for viruses,
> it does not guarantee that both are virus-free and accepts no
> liability for any damage sustained as a result of viruses.
>
Re: database connection lost
Posted by Ricardo Pertuz <ri...@kuasar.co>.
Hi Gary,
We had the same problem and increasing the wait timeout solved it.
________________________________
From: Gary Dixon <Ga...@quadris.co.uk.INVALID>
Sent: Tuesday, December 20, 2022, 9:31 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: database connection lost
Thanks Wei
Yes we are seeing the exact same errors in the cs mgmt. logs - we will set the MySQL global wait timeout to a higher value and see if it resolves our issue
What's the error message ?
There was a github issue may help you:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcloudstack%2Fissues%2F4657&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e4149de9c5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638071430711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u%2BEUWskmwM4y%2FA4%2FEwmJOBEOsdr0ZtYKcmz2Z5X9OGg%3D&reserved=0
-Wei
On Tuesday, 20 December 2022, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi Vivek
>
> We are using the --single-transaction flag and --skip-lock-tables flag
> Could there be something other than mysqldumps causing the cs mgmt.
> server to be losing the connection to the database ?
>
>
> BR
>
> Gary
>
>
> Hello,
>
> You can use --single-transaction
> Option while taking the dump or you can schedule to take dump from a
> slave server if you have.
>
> Regards,
> Vivek Kumar
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e
> 4149de9c5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63
> 8071430711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ClDbnphfgf
> WJtD37XnVSa2U1n6EQOtTVYBYbUBCsAJc%3D&reserved=0
> The information contained in this e-mail from Quadris may be
> confidential and privileged for the private use of the named
> recipient. The contents of this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please
> destroy any hard copies and delete this message.
>
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: v<tel:+44%207989717661>ms@quadris‑support.com
W: www.quadris.co.uk
[cid:image011498.png@FB84B645.38DDA861]
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
On Tue, Dec 20, 2022, 7:25 PM Gary Dixon <Gary.Dixon@quadris.co.uk.
> invalid>
> wrote:
>
> > Hi all
> >
> >
> >
> > CS v4.15.2
> >
> > Hypervisor = KVM
> >
> >
> >
> > We have started to experience an issue on a frequent basis where the
> > CS mgmt. service is losing the connection to the database and
> > believe it is being caused by mysqldumps locking the tables.
> >
> > Looking at the db.properties file there are tuning parameters :
> >
> > # CloudStack database tuning parameters
> >
> > db.cloud.maxActive=250
> >
> > db.cloud.maxIdle=30
> >
> > db.cloud.maxWait=10000
> >
> > db.cloud.validationQuery=SELECT 1
> >
> > db.cloud.testOnBorrow=true
> >
> > db.cloud.testWhileIdle=true
> >
> > db.cloud.timeBetweenEvictionRunsMillis=40000
> >
> > db.cloud.minEvictableIdleTimeMillis=240000
> >
> > db.cloud.poolPreparedStatements=false
> >
> >
> > db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessio
> > nV
> > ariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,
> > ER
> > ROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
> >
> >
> >
> > would changing the ‘maxwait’ value to a higher value stop the mgmt.
> > service losing the db connection whilst dumps are taken ?
> >
> > BR
> >
> >
> >
> > Gary
> >
> >
> >
> >
> > Gary Dixon
> > Senior Technical Consultant
> > T: +44 161 537 4990
> > E: *v* <+44%207989717661>ms@quadris‑support.com
> > W:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww
> > .q%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e4149de9c
> > 5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63807143
> > 0711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
> > zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mt9MlK3EMSW1
> > shNbTq%2B4Z9JseyTJw1Ynah305yJeq4s%3D&reserved=0
> > uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc5084879
> > 20
> > c948fa441908dae2949e38%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C
> > 63
> > 8071425209591727%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo
> > iV
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SAj0nkOY
> > 53
> > 6UUokgQPzXfdGH6Jp2pJ1hxlqA5hAJiwY%3D&reserved=0
> > The information contained in this e-mail from Quadris may be
> > confidential and privileged for the private use of the named
> > recipient. The contents of this e-mail may not necessarily represent
> > the
> official views of Quadris.
> > If you have received this information in error you must not copy,
> > distribute or take any action or reliance on its contents. Please
> > destroy any hard copies and delete this message.
> >
>
> --
> This message is intended only for the use of the individual or entity
> to which it is addressed and may contain confidential and/or
> privileged information. If you are not the intended recipient, please
> delete the original message and any copy of it from your computer
> system. You are hereby notified that any dissemination, distribution
> or copying of this communication is strictly prohibited unless proper
> authorization has been obtained for such action. If you have received
> this communication in error, please notify the sender immediately.
> Although IndiQus attempts to sweep e-mail and attachments for viruses,
> it does not guarantee that both are virus-free and accepts no
> liability for any damage sustained as a result of viruses.
>
RE: database connection lost
Posted by Gary Dixon <Ga...@quadris.co.uk.INVALID>.
Thanks Wei
Yes we are seeing the exact same errors in the cs mgmt. logs - we will set the MySQL global wait timeout to a higher value and see if it resolves our issue
What's the error message ?
There was a github issue may help you:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcloudstack%2Fissues%2F4657&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e4149de9c5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638071430711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u%2BEUWskmwM4y%2FA4%2FEwmJOBEOsdr0ZtYKcmz2Z5X9OGg%3D&reserved=0
-Wei
On Tuesday, 20 December 2022, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi Vivek
>
> We are using the --single-transaction flag and --skip-lock-tables flag
> Could there be something other than mysqldumps causing the cs mgmt.
> server to be losing the connection to the database ?
>
>
> BR
>
> Gary
>
>
> Hello,
>
> You can use --single-transaction
> Option while taking the dump or you can schedule to take dump from a
> slave server if you have.
>
> Regards,
> Vivek Kumar
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e
> 4149de9c5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63
> 8071430711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ClDbnphfgf
> WJtD37XnVSa2U1n6EQOtTVYBYbUBCsAJc%3D&reserved=0
> The information contained in this e-mail from Quadris may be
> confidential and privileged for the private use of the named
> recipient. The contents of this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please
> destroy any hard copies and delete this message.
> On Tue, Dec 20, 2022, 7:25 PM Gary Dixon <Gary.Dixon@quadris.co.uk.
> invalid>
> wrote:
>
> > Hi all
> >
> >
> >
> > CS v4.15.2
> >
> > Hypervisor = KVM
> >
> >
> >
> > We have started to experience an issue on a frequent basis where the
> > CS mgmt. service is losing the connection to the database and
> > believe it is being caused by mysqldumps locking the tables.
> >
> > Looking at the db.properties file there are tuning parameters :
> >
> > # CloudStack database tuning parameters
> >
> > db.cloud.maxActive=250
> >
> > db.cloud.maxIdle=30
> >
> > db.cloud.maxWait=10000
> >
> > db.cloud.validationQuery=SELECT 1
> >
> > db.cloud.testOnBorrow=true
> >
> > db.cloud.testWhileIdle=true
> >
> > db.cloud.timeBetweenEvictionRunsMillis=40000
> >
> > db.cloud.minEvictableIdleTimeMillis=240000
> >
> > db.cloud.poolPreparedStatements=false
> >
> >
> > db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessio
> > nV
> > ariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,
> > ER
> > ROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
> >
> >
> >
> > would changing the ‘maxwait’ value to a higher value stop the mgmt.
> > service losing the db connection whilst dumps are taken ?
> >
> > BR
> >
> >
> >
> > Gary
> >
> >
> >
> >
> > Gary Dixon
> > Senior Technical Consultant
> > T: +44 161 537 4990
> > E: *v* <+44%207989717661>ms@quadris‑support.com
> > W:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww
> > .q%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc0e6e3c68e4149de9c
> > 5308dae295e6b3%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63807143
> > 0711725069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
> > zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mt9MlK3EMSW1
> > shNbTq%2B4Z9JseyTJw1Ynah305yJeq4s%3D&reserved=0
> > uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc5084879
> > 20
> > c948fa441908dae2949e38%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C
> > 63
> > 8071425209591727%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo
> > iV
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SAj0nkOY
> > 53
> > 6UUokgQPzXfdGH6Jp2pJ1hxlqA5hAJiwY%3D&reserved=0
> > The information contained in this e-mail from Quadris may be
> > confidential and privileged for the private use of the named
> > recipient. The contents of this e-mail may not necessarily represent
> > the
> official views of Quadris.
> > If you have received this information in error you must not copy,
> > distribute or take any action or reliance on its contents. Please
> > destroy any hard copies and delete this message.
> >
>
> --
> This message is intended only for the use of the individual or entity
> to which it is addressed and may contain confidential and/or
> privileged information. If you are not the intended recipient, please
> delete the original message and any copy of it from your computer
> system. You are hereby notified that any dissemination, distribution
> or copying of this communication is strictly prohibited unless proper
> authorization has been obtained for such action. If you have received
> this communication in error, please notify the sender immediately.
> Although IndiQus attempts to sweep e-mail and attachments for viruses,
> it does not guarantee that both are virus-free and accepts no
> liability for any damage sustained as a result of viruses.
>
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: vms@quadris-support.com
W: www.quadris.co.uk
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
Re: database connection lost
Posted by Wei ZHOU <us...@gmail.com>.
What's the error message ?
There was a github issue may help you:
https://github.com/apache/cloudstack/issues/4657
-Wei
On Tuesday, 20 December 2022, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi Vivek
>
> We are using the --single-transaction flag and --skip-lock-tables flag
> Could there be something other than mysqldumps causing the cs mgmt. server
> to be losing the connection to the database ?
>
>
> BR
>
> Gary
>
>
> Hello,
>
> You can use --single-transaction
> Option while taking the dump or you can schedule to take dump from a slave
> server if you have.
>
> Regards,
> Vivek Kumar
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient. The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please destroy
> any hard copies and delete this message.
> On Tue, Dec 20, 2022, 7:25 PM Gary Dixon <Gary.Dixon@quadris.co.uk.
> invalid>
> wrote:
>
> > Hi all
> >
> >
> >
> > CS v4.15.2
> >
> > Hypervisor = KVM
> >
> >
> >
> > We have started to experience an issue on a frequent basis where the
> > CS mgmt. service is losing the connection to the database and believe
> > it is being caused by mysqldumps locking the tables.
> >
> > Looking at the db.properties file there are tuning parameters :
> >
> > # CloudStack database tuning parameters
> >
> > db.cloud.maxActive=250
> >
> > db.cloud.maxIdle=30
> >
> > db.cloud.maxWait=10000
> >
> > db.cloud.validationQuery=SELECT 1
> >
> > db.cloud.testOnBorrow=true
> >
> > db.cloud.testWhileIdle=true
> >
> > db.cloud.timeBetweenEvictionRunsMillis=40000
> >
> > db.cloud.minEvictableIdleTimeMillis=240000
> >
> > db.cloud.poolPreparedStatements=false
> >
> >
> > db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionV
> > ariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ER
> > ROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
> >
> >
> >
> > would changing the ‘maxwait’ value to a higher value stop the mgmt.
> > service losing the db connection whilst dumps are taken ?
> >
> > BR
> >
> >
> >
> > Gary
> >
> >
> >
> >
> > Gary Dixon
> > Senior Technical Consultant
> > T: +44 161 537 4990
> > E: *v* <+44%207989717661>ms@quadris‑support.com
> > W:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> > uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc508487920
> > c948fa441908dae2949e38%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63
> > 8071425209591727%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SAj0nkOY53
> > 6UUokgQPzXfdGH6Jp2pJ1hxlqA5hAJiwY%3D&reserved=0
> > The information contained in this e-mail from Quadris may be
> > confidential and privileged for the private use of the named
> > recipient. The contents of this e-mail may not necessarily represent the
> official views of Quadris.
> > If you have received this information in error you must not copy,
> > distribute or take any action or reliance on its contents. Please
> > destroy any hard copies and delete this message.
> >
>
> --
> This message is intended only for the use of the individual or entity to
> which it is addressed and may contain confidential and/or privileged
> information. If you are not the intended recipient, please delete the
> original message and any copy of it from your computer system. You are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited unless proper authorization has been
> obtained for such action. If you have received this communication in error,
> please notify the sender immediately. Although IndiQus attempts to sweep
> e-mail and attachments for viruses, it does not guarantee that both are
> virus-free and accepts no liability for any damage sustained as a result of
> viruses.
>
RE: database connection lost
Posted by Gary Dixon <Ga...@quadris.co.uk.INVALID>.
Hi Vivek
We are using the --single-transaction flag and --skip-lock-tables flag
Could there be something other than mysqldumps causing the cs mgmt. server to be losing the connection to the database ?
BR
Gary
Hello,
You can use --single-transaction
Option while taking the dump or you can schedule to take dump from a slave server if you have.
Regards,
Vivek Kumar
On Tue, Dec 20, 2022, 7:25 PM Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi all
>
>
>
> CS v4.15.2
>
> Hypervisor = KVM
>
>
>
> We have started to experience an issue on a frequent basis where the
> CS mgmt. service is losing the connection to the database and believe
> it is being caused by mysqldumps locking the tables.
>
> Looking at the db.properties file there are tuning parameters :
>
> # CloudStack database tuning parameters
>
> db.cloud.maxActive=250
>
> db.cloud.maxIdle=30
>
> db.cloud.maxWait=10000
>
> db.cloud.validationQuery=SELECT 1
>
> db.cloud.testOnBorrow=true
>
> db.cloud.testWhileIdle=true
>
> db.cloud.timeBetweenEvictionRunsMillis=40000
>
> db.cloud.minEvictableIdleTimeMillis=240000
>
> db.cloud.poolPreparedStatements=false
>
>
> db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionV
> ariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ER
> ROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
>
>
>
> would changing the ‘maxwait’ value to a higher value stop the mgmt.
> service losing the db connection whilst dumps are taken ?
>
> BR
>
>
>
> Gary
>
>
>
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cc508487920
> c948fa441908dae2949e38%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63
> 8071425209591727%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SAj0nkOY53
> 6UUokgQPzXfdGH6Jp2pJ1hxlqA5hAJiwY%3D&reserved=0
> The information contained in this e-mail from Quadris may be
> confidential and privileged for the private use of the named
> recipient. The contents of this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please
> destroy any hard copies and delete this message.
>
--
This message is intended only for the use of the individual or entity to which it is addressed and may contain confidential and/or privileged information. If you are not the intended recipient, please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: vms@quadris-support.com
W: www.quadris.co.uk
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
Re: database connection lost
Posted by Vivek Kumar <vi...@indiqus.com.INVALID>.
Hello,
You can use --single-transaction
Option while taking the dump or you can schedule to take dump from a slave
server if you have.
Regards,
Vivek Kumar
On Tue, Dec 20, 2022, 7:25 PM Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi all
>
>
>
> CS v4.15.2
>
> Hypervisor = KVM
>
>
>
> We have started to experience an issue on a frequent basis where the CS
> mgmt. service is losing the connection to the database and believe it is
> being caused by mysqldumps locking the tables.
>
> Looking at the db.properties file there are tuning parameters :
>
> # CloudStack database tuning parameters
>
> db.cloud.maxActive=250
>
> db.cloud.maxIdle=30
>
> db.cloud.maxWait=10000
>
> db.cloud.validationQuery=SELECT 1
>
> db.cloud.testOnBorrow=true
>
> db.cloud.testWhileIdle=true
>
> db.cloud.timeBetweenEvictionRunsMillis=40000
>
> db.cloud.minEvictableIdleTimeMillis=240000
>
> db.cloud.poolPreparedStatements=false
>
>
> db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
>
>
>
> would changing the ‘maxwait’ value to a higher value stop the mgmt.
> service losing the db connection whilst dumps are taken ?
>
> BR
>
>
>
> Gary
>
>
>
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient. The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please destroy
> any hard copies and delete this message.
>
--
This message is intended only for the use of the individual or entity to
which it is addressed and may contain confidential and/or privileged
information. If you are not the intended recipient, please delete the
original message and any copy of it from your computer system. You are
hereby notified that any dissemination, distribution or copying of this
communication is strictly prohibited unless proper authorization has been
obtained for such action. If you have received this communication in error,
please notify the sender immediately. Although IndiQus attempts to sweep
e-mail and attachments for viruses, it does not guarantee that both are
virus-free and accepts no liability for any damage sustained as a result of
viruses.
RE: database connection lost
Posted by Gary Dixon <Ga...@quadris.co.uk.INVALID>.
Hi Wei
We thought that without locking tables the mysql dumps may be inconsistent ?
Hi,
You can dump databases without locking tables.
-Wei
On Tuesday, 20 December 2022, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi all
>
>
>
> CS v4.15.2
>
> Hypervisor = KVM
>
>
>
> We have started to experience an issue on a frequent basis where the
> CS mgmt. service is losing the connection to the database and believe
> it is being caused by mysqldumps locking the tables.
>
> Looking at the db.properties file there are tuning parameters :
>
> # CloudStack database tuning parameters
>
> db.cloud.maxActive=250
>
> db.cloud.maxIdle=30
>
> db.cloud.maxWait=10000
>
> db.cloud.validationQuery=SELECT 1
>
> db.cloud.testOnBorrow=true
>
> db.cloud.testWhileIdle=true
>
> db.cloud.timeBetweenEvictionRunsMillis=40000
>
> db.cloud.minEvictableIdleTimeMillis=240000
>
> db.cloud.poolPreparedStatements=false
>
> db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&
> sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_
> IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
> '&
> serverTimezone=UTC
>
>
>
> would changing the ‘maxwait’ value to a higher value stop the mgmt.
> service losing the db connection whilst dumps are taken ?
>
> BR
>
>
>
> Gary
>
>
>
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Ccea023f90a
> c8411f3f4108dae2930167%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63
> 8071418262847712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4koYUIbCcb
> ARylkd5DOg1ZP7gY%2B6YkNyRRjOipuM%2Brw%3D&reserved=0
> The information contained in this e-mail from Quadris may be
> confidential and privileged for the private use of the named
> recipient. The contents of this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please
> destroy any hard copies and delete this message.
>
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: vms@quadris-support.com
W: www.quadris.co.uk
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
Re: database connection lost
Posted by Wei ZHOU <us...@gmail.com>.
Hi,
You can dump databases without locking tables.
-Wei
On Tuesday, 20 December 2022, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi all
>
>
>
> CS v4.15.2
>
> Hypervisor = KVM
>
>
>
> We have started to experience an issue on a frequent basis where the CS
> mgmt. service is losing the connection to the database and believe it is
> being caused by mysqldumps locking the tables.
>
> Looking at the db.properties file there are tuning parameters :
>
> # CloudStack database tuning parameters
>
> db.cloud.maxActive=250
>
> db.cloud.maxIdle=30
>
> db.cloud.maxWait=10000
>
> db.cloud.validationQuery=SELECT 1
>
> db.cloud.testOnBorrow=true
>
> db.cloud.testWhileIdle=true
>
> db.cloud.timeBetweenEvictionRunsMillis=40000
>
> db.cloud.minEvictableIdleTimeMillis=240000
>
> db.cloud.poolPreparedStatements=false
>
> db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&
> sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_
> IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&
> serverTimezone=UTC
>
>
>
> would changing the ‘maxwait’ value to a higher value stop the mgmt.
> service losing the db connection whilst dumps are taken ?
>
> BR
>
>
>
> Gary
>
>
>
>
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient. The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please destroy
> any hard copies and delete this message.
>