You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Erik Weber <te...@gmail.com> on 2015/03/20 21:17:33 UTC
Ways to monitor Virtual Router disk space
I've had a few incidents where conntrack logging has filled the /var
partition, and break provisioning of new VMs (unable to save password).
And this got me thinking that there must be a way to monitor VR disk space..
Some things to consider:
- I'm using XenServer, so AFAIK the VR ssh is only available from the XS
link-local network.
- I'd prefer to not build my own systemvm templates, ie. I'd like to use
them as is
- The monitoring should eventually end up in Nagios
My idea so far is to make a collectd script or similar, that uses the
CloudStack API to fetch a list of routers and its XS host, connect to the
VR through the right XS Host and then push the data to graphite or similar.
Then use the check_graphite plugin in Nagios to monitor the disk space.
Does anyone have any insight to this, experience with doing something
similar or other inputs?
--
Erik
Re: Ways to monitor Virtual Router disk space
Posted by Erik Weber <te...@gmail.com>.
On Sat, Mar 21, 2015 at 2:20 PM, Rene Moser <ma...@renemoser.net> wrote:
> Hi Erik
>
> On 03/20/2015 09:17 PM, Erik Weber wrote:
>
>> I've had a few incidents where conntrack logging has filled the /var
>> partition, and break provisioning of new VMs (unable to save password).
>>
>> And this got me thinking that there must be a way to monitor VR disk
>> space..
>>
>
> We have had the same problem.
>
> We created some tools a while ago for that like
> https://github.com/swisstxt/cloudstack-nagios which helps you monitor
> CloudStack VRs in nagios or icinga.
>
> But recently we switch to Ansible for managing (security updates, config
> changes, package install) the running VRs. So you can basically make a
> playbook where you can setup the monitoring on the VRs.
>
> I created a example project. It uses a "dynamic inventory" by fetching all
> the routers using the API. See https://github.com/resmo/
> ansible-cloudstack-routers
>
> You can run the playbooks scheduled by a cronjob or manually, using check
> mode (aka dry-run) to see what would have changed and you are also able to
> limit the targets like updating the backup routers first, and then the
> masters, etc.
>
> Hope that helps :)
>
>
That's some wicked tools, thanks for sharing!
I'll have to spend some time to get them to know a bit better, and report
my findings :-)
--
Erik
Re: Ways to monitor Virtual Router disk space
Posted by Erik Weber <te...@gmail.com>.
Sorry for top-posting.. Mobile..
I'm using xenserver and linklocal is using the 169.254 apipa range. All xs
servers have the 169.254.0.1 iphone.
I've been looking at improving it to proxy through the right xs server, but
face a few ansible issues i have to clear out
Erik
Den mandag 23. mars 2015 skrev Rene Moser <ma...@renemoser.net> følgende:
>
>
> On 22.03.2015 22:35, Erik Weber wrote:
>
> > This is running on CCP 4.3.2 if it matters, and is on production so I'm
> > hesitant to mess with it..
>
> I am on CCP 4.2.1
>
> > Any idea if this is how it's supposed to be or if something if fubar in
> my
> > setup? If this is how it's supposed to be, how does other access their
> > systemvms outside the hypervisor?
>
> You should be able to SSH into the systemvm from the cloudstack
> management system (using the right private key and user root).
>
> If you like to have access from your workstation, you can make use of
> "proxycommand" if you have a recent openssh. Copy the private key from
> cloudstack management to ~/.ssh/id_rsa_cloudstack
>
> scp root@cloudstack:/var/lib/cloud/management/.ssh/id_rsa
> .ssh/id_rsa_cloudstack
>
> and configure your client ssh config like, note that you must use the so
> called "linklocal ip" of the router. In my case these are IPs
> 10.100.9.0/24 and 10.100.209.0/24
>
> $ cat .ssh/config
> Host 10.100.9.* 10.100.209.*
> User root
> IdentityFile ~/.ssh/id_rsa_cloudstack
> StrictHostKeyChecking no
> ProxyCommand ssh -q -o ControlPersist=no cloudstack.example.com
>
> you can simply ssh in:
>
> ssh 10.100.9.45
>
> So how to find the linklocal ip you may ask:
>
> A easy way is using cs package from pip (which you will also need for
> using ansible). It also has a cli interface
>
> pip install cs
>
> cs listRouters listall=true | grep linklocalip
>
> Yours
> René
>
Re: Ways to monitor Virtual Router disk space
Posted by Rene Moser <ma...@renemoser.net>.
On 22.03.2015 22:35, Erik Weber wrote:
> This is running on CCP 4.3.2 if it matters, and is on production so I'm
> hesitant to mess with it..
I am on CCP 4.2.1
> Any idea if this is how it's supposed to be or if something if fubar in my
> setup? If this is how it's supposed to be, how does other access their
> systemvms outside the hypervisor?
You should be able to SSH into the systemvm from the cloudstack
management system (using the right private key and user root).
If you like to have access from your workstation, you can make use of
"proxycommand" if you have a recent openssh. Copy the private key from
cloudstack management to ~/.ssh/id_rsa_cloudstack
scp root@cloudstack:/var/lib/cloud/management/.ssh/id_rsa
.ssh/id_rsa_cloudstack
and configure your client ssh config like, note that you must use the so
called "linklocal ip" of the router. In my case these are IPs
10.100.9.0/24 and 10.100.209.0/24
$ cat .ssh/config
Host 10.100.9.* 10.100.209.*
User root
IdentityFile ~/.ssh/id_rsa_cloudstack
StrictHostKeyChecking no
ProxyCommand ssh -q -o ControlPersist=no cloudstack.example.com
you can simply ssh in:
ssh 10.100.9.45
So how to find the linklocal ip you may ask:
A easy way is using cs package from pip (which you will also need for
using ansible). It also has a cli interface
pip install cs
cs listRouters listall=true | grep linklocalip
Yours
René
Re: Ways to monitor Virtual Router disk space
Posted by Erik Weber <te...@gmail.com>.
That was a rather empty reply, did you hit 'send' by accident? :-)
--
Erik
2015-03-23 12:27 GMT+01:00 Rene Moser <ma...@renemoser.net>:
>
Re: Ways to monitor Virtual Router disk space
Posted by Erik Weber <te...@gmail.com>.
On Sat, Mar 21, 2015 at 2:20 PM, Rene Moser <ma...@renemoser.net> wrote:
> Hi Erik
>
> On 03/20/2015 09:17 PM, Erik Weber wrote:
>
>> I've had a few incidents where conntrack logging has filled the /var
>> partition, and break provisioning of new VMs (unable to save password).
>>
>> And this got me thinking that there must be a way to monitor VR disk
>> space..
>>
>
> We have had the same problem.
>
> We created some tools a while ago for that like
> https://github.com/swisstxt/cloudstack-nagios which helps you monitor
> CloudStack VRs in nagios or icinga.
>
> But recently we switch to Ansible for managing (security updates, config
> changes, package install) the running VRs. So you can basically make a
> playbook where you can setup the monitoring on the VRs.
>
> I created a example project. It uses a "dynamic inventory" by fetching all
> the routers using the API. See https://github.com/resmo/
> ansible-cloudstack-routers
>
> You can run the playbooks scheduled by a cronjob or manually, using check
> mode (aka dry-run) to see what would have changed and you are also able to
> limit the targets like updating the backup routers first, and then the
> masters, etc.
>
> Hope that helps :)
>
>
I do have a small problem though, I'm not entirely sure if it's my setup or
if it usually is like this, but here goes..
My hypervisors have 6 interfaces, eth0-eth5. They are bonded in pairs in
the following way:
eth0 + eth1 = xapi2, label=cloud-private, usage=management network on
native vlan, public network on tagged vlan
eth2 + eth3 = xapi0, label=cloud-backup, usage=guest network, currently not
in use
eth4 + eth5 = xapi1, label=cloud-guest, usage=guest network, vlan tagged
additionally I have the xapi3 bridge, which only consists of virtual
interfaces (ie. systemvm interfaces), and no physical interfaces.
Which makes it really hard to access any systemvm from anything else than
the actual hypervisor host that is running the vm.
This is running on CCP 4.3.2 if it matters, and is on production so I'm
hesitant to mess with it..
Any idea if this is how it's supposed to be or if something if fubar in my
setup? If this is how it's supposed to be, how does other access their
systemvms outside the hypervisor?
--
Erik
Re: Ways to monitor Virtual Router disk space
Posted by Rene Moser <ma...@renemoser.net>.
Hi Erik
On 03/20/2015 09:17 PM, Erik Weber wrote:
> I've had a few incidents where conntrack logging has filled the /var
> partition, and break provisioning of new VMs (unable to save password).
>
> And this got me thinking that there must be a way to monitor VR disk space..
We have had the same problem.
We created some tools a while ago for that like
https://github.com/swisstxt/cloudstack-nagios which helps you monitor
CloudStack VRs in nagios or icinga.
But recently we switch to Ansible for managing (security updates, config
changes, package install) the running VRs. So you can basically make a
playbook where you can setup the monitoring on the VRs.
I created a example project. It uses a "dynamic inventory" by fetching
all the routers using the API. See
https://github.com/resmo/ansible-cloudstack-routers
You can run the playbooks scheduled by a cronjob or manually, using
check mode (aka dry-run) to see what would have changed and you are also
able to limit the targets like updating the backup routers first, and
then the masters, etc.
Hope that helps :)
Yours
René