You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Osama Lotf <en...@gmail.com> on 2013/04/04 09:48:53 UTC
failing to create new instances
Hi
I have cs 3.02 (no upgrades is allowed if you dont remeber :P) and KVM
after creating 1000 instances and they finished normally and been
destroyed, I cant start any single instance at all
attached is a portion from log file
Thank you in advance..
--
Eng. Osamah Lutf Barakat
Master Candidate
Computer security
Faculty of Engineering
University Putra Malaysia
Re: failing to create new instances
Posted by Osama Lotf <en...@gmail.com>.
hi
Thank you for your replay,
1. No, there were New servers..
2. here are the logs i found in one of them
http://paste.cloudstack.org/F3Cp/
Thank you for your great support and time
2013/4/9 Somesh Naidu <So...@citrix.com>
> Were the 2 hosts being added fresh installs or were they previously part
> of CS at some point and were being re-used?
>
> As I said, more details should be present in the agent logs.
>
> Regards,
> Somesh
>
>
> -----Original Message-----
> From: Osama Lotf [mailto:engbarakat2007@gmail.com]
> Sent: Friday, April 05, 2013 5:48 PM
> To: users@cloudstack.apache.org
> Subject: Re: failing to create new instances
>
> woow thats amazing >>>
>
> well, what happened in details is like this
>
>
> 1. I ran dataset which invokes 1000 instances and then destroy them.
>
> 2. I noticed that the 2 newly added hosts are offline..
>
> 3. when i tried to turn the network up, it failed to up bridge clourbr0
>
> 4. I tried to fix that as every time it said bridge with name xxx is
> already exits and cannot create with same name...
>
> 5. when I deleted the bridge file from
> /etc/sysconfig/network-scripts/ifcfg-cloudbr0 every thing went up
> normally, and can connect and ssh to those agents..
>
>
> 6. with this error (in log), what do you think I should do?
>
> Thank you
>
>
> 2013/4/5 Somesh Naidu <So...@citrix.com>
>
> > The VM in question was initially being deployed on
> >
> Dest[Zone(1)-Pod(1)-Cluster(3)-Host(6)-Storage(Volume(59228|ROOT-->Pool(202))].
> >
> > While the volume was successfully created for this VM on storage pool 202
> > associated with Cluster 3, the attempt to start the VM in this cluster
> > failed due to the following error,
> > 2013-04-04 15:16:33,687 WARN [cloud.vm.VirtualMachineManagerImpl]
> > (AgentManager-Handler-6:null) Cleanup failed due to Unable to add bridge
> > cloudbr0 port vnet0: Operation not supported
> >
> > Not sure about the exact cause of this error, needs further
> investigation.
> > I see you using KVM. More details can be found in the agent logs for KVM
> > (/var/log/cloud/agent/agent.log). You should grep for "6-447676435" in
> the
> > logs and paste the output @ http://paste.cloudstack.org/ for us to
> review.
> >
> > Management server then tried to identify other hosts on this cluster
> > (id=3) to start this VM but found none suitable. So the management server
> > eventually picked up a host in a different cluster (id 5) to deploy this
> > VM. However, since the volume was created in the first attempt on
> cluster 3
> > and since by default all user volumes have "recreatable" attribute set to
> > false this operation failed with the following error:
> > 2013-04-04 15:16:46,585 DEBUG [cloud.storage.StorageManagerImpl]
> > (Job-Executor-1:job-6899) Volume Vol[59228|vm=59221|ROOT] is not
> > recreatable! Cannot recreate on storagepool: Pool[203|NetworkFilesystem]
> >
> > This is due to a defect which, if I remember correctly, has been fixed in
> > the future releases.
> >
> > Regards,
> > Somesh
> >
> > -----Original Message-----
> > From: Osama Lotf [mailto:engbarakat2007@gmail.com]
> > Sent: Friday, April 05, 2013 1:44 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: failing to create new instances
> >
> > hi
> >
> > sorry being late
> >
> > http://paste.cloudstack.org/nXGg/ try this to go through log
> >
> > I run ssvm script and every test is ok ...
> >
> >
> > Thank you
> >
> >
> > 2013/4/4 Nitin Mehta <Ni...@citrix.com>
> >
> > > Can you please paste the logs @ http://paste.cloudstack.org/
> > > ML doesn't allow you to attach files.
> > >
> > > From: Osama Lotf <engbarakat2007@gmail.com<mailto:
> > engbarakat2007@gmail.com
> > > >>
> > > Reply-To: "users@cloudstack.apache.org<mailto:
> > users@cloudstack.apache.org>"
> > > <us...@cloudstack.apache.org>>
> > > Date: Thursday 4 April 2013 1:18 PM
> > > To: "cloudstack-users@incubator.apache.org<mailto:
> > > cloudstack-users@incubator.apache.org>" <
> > > cloudstack-users@incubator.apache.org<mailto:
> > > cloudstack-users@incubator.apache.org>>
> > > Subject: failing to create new instances
> > >
> > > Hi
> > >
> > > I have cs 3.02 (no upgrades is allowed if you dont remeber :P) and KVM
> > >
> > > after creating 1000 instances and they finished normally and been
> > > destroyed, I cant start any single instance at all
> > >
> > > attached is a portion from log file
> > > Thank you in advance..
> > >
> > > --
> > > Eng. Osamah Lutf Barakat
> > > Master Candidate
> > > Computer security
> > > Faculty of Engineering
> > > University Putra Malaysia
> > >
> >
> >
> >
> > --
> > Eng. Osamah Lutf Barakat
> > Master Candidate
> > Computer security
> > Faculty of Engineering
> > University Putra Malaysia
> >
>
>
>
> --
> Eng. Osamah Lutf Barakat
> Master Candidate
> Computer security
> Faculty of Engineering
> University Putra Malaysia
>
--
Eng. Osamah Lutf Barakat
Master Candidate
Computer security
Faculty of Engineering
University Putra Malaysia
RE: failing to create new instances
Posted by Somesh Naidu <So...@citrix.com>.
Were the 2 hosts being added fresh installs or were they previously part of CS at some point and were being re-used?
As I said, more details should be present in the agent logs.
Regards,
Somesh
-----Original Message-----
From: Osama Lotf [mailto:engbarakat2007@gmail.com]
Sent: Friday, April 05, 2013 5:48 PM
To: users@cloudstack.apache.org
Subject: Re: failing to create new instances
woow thats amazing >>>
well, what happened in details is like this
1. I ran dataset which invokes 1000 instances and then destroy them.
2. I noticed that the 2 newly added hosts are offline..
3. when i tried to turn the network up, it failed to up bridge clourbr0
4. I tried to fix that as every time it said bridge with name xxx is
already exits and cannot create with same name...
5. when I deleted the bridge file from
/etc/sysconfig/network-scripts/ifcfg-cloudbr0 every thing went up
normally, and can connect and ssh to those agents..
6. with this error (in log), what do you think I should do?
Thank you
2013/4/5 Somesh Naidu <So...@citrix.com>
> The VM in question was initially being deployed on
> Dest[Zone(1)-Pod(1)-Cluster(3)-Host(6)-Storage(Volume(59228|ROOT-->Pool(202))].
>
> While the volume was successfully created for this VM on storage pool 202
> associated with Cluster 3, the attempt to start the VM in this cluster
> failed due to the following error,
> 2013-04-04 15:16:33,687 WARN [cloud.vm.VirtualMachineManagerImpl]
> (AgentManager-Handler-6:null) Cleanup failed due to Unable to add bridge
> cloudbr0 port vnet0: Operation not supported
>
> Not sure about the exact cause of this error, needs further investigation.
> I see you using KVM. More details can be found in the agent logs for KVM
> (/var/log/cloud/agent/agent.log). You should grep for "6-447676435" in the
> logs and paste the output @ http://paste.cloudstack.org/ for us to review.
>
> Management server then tried to identify other hosts on this cluster
> (id=3) to start this VM but found none suitable. So the management server
> eventually picked up a host in a different cluster (id 5) to deploy this
> VM. However, since the volume was created in the first attempt on cluster 3
> and since by default all user volumes have "recreatable" attribute set to
> false this operation failed with the following error:
> 2013-04-04 15:16:46,585 DEBUG [cloud.storage.StorageManagerImpl]
> (Job-Executor-1:job-6899) Volume Vol[59228|vm=59221|ROOT] is not
> recreatable! Cannot recreate on storagepool: Pool[203|NetworkFilesystem]
>
> This is due to a defect which, if I remember correctly, has been fixed in
> the future releases.
>
> Regards,
> Somesh
>
> -----Original Message-----
> From: Osama Lotf [mailto:engbarakat2007@gmail.com]
> Sent: Friday, April 05, 2013 1:44 PM
> To: users@cloudstack.apache.org
> Subject: Re: failing to create new instances
>
> hi
>
> sorry being late
>
> http://paste.cloudstack.org/nXGg/ try this to go through log
>
> I run ssvm script and every test is ok ...
>
>
> Thank you
>
>
> 2013/4/4 Nitin Mehta <Ni...@citrix.com>
>
> > Can you please paste the logs @ http://paste.cloudstack.org/
> > ML doesn't allow you to attach files.
> >
> > From: Osama Lotf <engbarakat2007@gmail.com<mailto:
> engbarakat2007@gmail.com
> > >>
> > Reply-To: "users@cloudstack.apache.org<mailto:
> users@cloudstack.apache.org>"
> > <us...@cloudstack.apache.org>>
> > Date: Thursday 4 April 2013 1:18 PM
> > To: "cloudstack-users@incubator.apache.org<mailto:
> > cloudstack-users@incubator.apache.org>" <
> > cloudstack-users@incubator.apache.org<mailto:
> > cloudstack-users@incubator.apache.org>>
> > Subject: failing to create new instances
> >
> > Hi
> >
> > I have cs 3.02 (no upgrades is allowed if you dont remeber :P) and KVM
> >
> > after creating 1000 instances and they finished normally and been
> > destroyed, I cant start any single instance at all
> >
> > attached is a portion from log file
> > Thank you in advance..
> >
> > --
> > Eng. Osamah Lutf Barakat
> > Master Candidate
> > Computer security
> > Faculty of Engineering
> > University Putra Malaysia
> >
>
>
>
> --
> Eng. Osamah Lutf Barakat
> Master Candidate
> Computer security
> Faculty of Engineering
> University Putra Malaysia
>
--
Eng. Osamah Lutf Barakat
Master Candidate
Computer security
Faculty of Engineering
University Putra Malaysia
Re: failing to create new instances
Posted by Osama Lotf <en...@gmail.com>.
woow thats amazing >>>
well, what happened in details is like this
1. I ran dataset which invokes 1000 instances and then destroy them.
2. I noticed that the 2 newly added hosts are offline..
3. when i tried to turn the network up, it failed to up bridge clourbr0
4. I tried to fix that as every time it said bridge with name xxx is
already exits and cannot create with same name...
5. when I deleted the bridge file from
/etc/sysconfig/network-scripts/ifcfg-cloudbr0 every thing went up
normally, and can connect and ssh to those agents..
6. with this error (in log), what do you think I should do?
Thank you
2013/4/5 Somesh Naidu <So...@citrix.com>
> The VM in question was initially being deployed on
> Dest[Zone(1)-Pod(1)-Cluster(3)-Host(6)-Storage(Volume(59228|ROOT-->Pool(202))].
>
> While the volume was successfully created for this VM on storage pool 202
> associated with Cluster 3, the attempt to start the VM in this cluster
> failed due to the following error,
> 2013-04-04 15:16:33,687 WARN [cloud.vm.VirtualMachineManagerImpl]
> (AgentManager-Handler-6:null) Cleanup failed due to Unable to add bridge
> cloudbr0 port vnet0: Operation not supported
>
> Not sure about the exact cause of this error, needs further investigation.
> I see you using KVM. More details can be found in the agent logs for KVM
> (/var/log/cloud/agent/agent.log). You should grep for "6-447676435" in the
> logs and paste the output @ http://paste.cloudstack.org/ for us to review.
>
> Management server then tried to identify other hosts on this cluster
> (id=3) to start this VM but found none suitable. So the management server
> eventually picked up a host in a different cluster (id 5) to deploy this
> VM. However, since the volume was created in the first attempt on cluster 3
> and since by default all user volumes have "recreatable" attribute set to
> false this operation failed with the following error:
> 2013-04-04 15:16:46,585 DEBUG [cloud.storage.StorageManagerImpl]
> (Job-Executor-1:job-6899) Volume Vol[59228|vm=59221|ROOT] is not
> recreatable! Cannot recreate on storagepool: Pool[203|NetworkFilesystem]
>
> This is due to a defect which, if I remember correctly, has been fixed in
> the future releases.
>
> Regards,
> Somesh
>
> -----Original Message-----
> From: Osama Lotf [mailto:engbarakat2007@gmail.com]
> Sent: Friday, April 05, 2013 1:44 PM
> To: users@cloudstack.apache.org
> Subject: Re: failing to create new instances
>
> hi
>
> sorry being late
>
> http://paste.cloudstack.org/nXGg/ try this to go through log
>
> I run ssvm script and every test is ok ...
>
>
> Thank you
>
>
> 2013/4/4 Nitin Mehta <Ni...@citrix.com>
>
> > Can you please paste the logs @ http://paste.cloudstack.org/
> > ML doesn't allow you to attach files.
> >
> > From: Osama Lotf <engbarakat2007@gmail.com<mailto:
> engbarakat2007@gmail.com
> > >>
> > Reply-To: "users@cloudstack.apache.org<mailto:
> users@cloudstack.apache.org>"
> > <us...@cloudstack.apache.org>>
> > Date: Thursday 4 April 2013 1:18 PM
> > To: "cloudstack-users@incubator.apache.org<mailto:
> > cloudstack-users@incubator.apache.org>" <
> > cloudstack-users@incubator.apache.org<mailto:
> > cloudstack-users@incubator.apache.org>>
> > Subject: failing to create new instances
> >
> > Hi
> >
> > I have cs 3.02 (no upgrades is allowed if you dont remeber :P) and KVM
> >
> > after creating 1000 instances and they finished normally and been
> > destroyed, I cant start any single instance at all
> >
> > attached is a portion from log file
> > Thank you in advance..
> >
> > --
> > Eng. Osamah Lutf Barakat
> > Master Candidate
> > Computer security
> > Faculty of Engineering
> > University Putra Malaysia
> >
>
>
>
> --
> Eng. Osamah Lutf Barakat
> Master Candidate
> Computer security
> Faculty of Engineering
> University Putra Malaysia
>
--
Eng. Osamah Lutf Barakat
Master Candidate
Computer security
Faculty of Engineering
University Putra Malaysia
RE: failing to create new instances
Posted by Somesh Naidu <So...@citrix.com>.
The VM in question was initially being deployed on Dest[Zone(1)-Pod(1)-Cluster(3)-Host(6)-Storage(Volume(59228|ROOT-->Pool(202))].
While the volume was successfully created for this VM on storage pool 202 associated with Cluster 3, the attempt to start the VM in this cluster failed due to the following error,
2013-04-04 15:16:33,687 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-6:null) Cleanup failed due to Unable to add bridge cloudbr0 port vnet0: Operation not supported
Not sure about the exact cause of this error, needs further investigation. I see you using KVM. More details can be found in the agent logs for KVM (/var/log/cloud/agent/agent.log). You should grep for "6-447676435" in the logs and paste the output @ http://paste.cloudstack.org/ for us to review.
Management server then tried to identify other hosts on this cluster (id=3) to start this VM but found none suitable. So the management server eventually picked up a host in a different cluster (id 5) to deploy this VM. However, since the volume was created in the first attempt on cluster 3 and since by default all user volumes have "recreatable" attribute set to false this operation failed with the following error:
2013-04-04 15:16:46,585 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-1:job-6899) Volume Vol[59228|vm=59221|ROOT] is not recreatable! Cannot recreate on storagepool: Pool[203|NetworkFilesystem]
This is due to a defect which, if I remember correctly, has been fixed in the future releases.
Regards,
Somesh
-----Original Message-----
From: Osama Lotf [mailto:engbarakat2007@gmail.com]
Sent: Friday, April 05, 2013 1:44 PM
To: users@cloudstack.apache.org
Subject: Re: failing to create new instances
hi
sorry being late
http://paste.cloudstack.org/nXGg/ try this to go through log
I run ssvm script and every test is ok ...
Thank you
2013/4/4 Nitin Mehta <Ni...@citrix.com>
> Can you please paste the logs @ http://paste.cloudstack.org/
> ML doesn't allow you to attach files.
>
> From: Osama Lotf <engbarakat2007@gmail.com<mailto:engbarakat2007@gmail.com
> >>
> Reply-To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> <us...@cloudstack.apache.org>>
> Date: Thursday 4 April 2013 1:18 PM
> To: "cloudstack-users@incubator.apache.org<mailto:
> cloudstack-users@incubator.apache.org>" <
> cloudstack-users@incubator.apache.org<mailto:
> cloudstack-users@incubator.apache.org>>
> Subject: failing to create new instances
>
> Hi
>
> I have cs 3.02 (no upgrades is allowed if you dont remeber :P) and KVM
>
> after creating 1000 instances and they finished normally and been
> destroyed, I cant start any single instance at all
>
> attached is a portion from log file
> Thank you in advance..
>
> --
> Eng. Osamah Lutf Barakat
> Master Candidate
> Computer security
> Faculty of Engineering
> University Putra Malaysia
>
--
Eng. Osamah Lutf Barakat
Master Candidate
Computer security
Faculty of Engineering
University Putra Malaysia
Re: failing to create new instances
Posted by Osama Lotf <en...@gmail.com>.
hi
sorry being late
http://paste.cloudstack.org/nXGg/ try this to go through log
I run ssvm script and every test is ok ...
Thank you
2013/4/4 Nitin Mehta <Ni...@citrix.com>
> Can you please paste the logs @ http://paste.cloudstack.org/
> ML doesn't allow you to attach files.
>
> From: Osama Lotf <engbarakat2007@gmail.com<mailto:engbarakat2007@gmail.com
> >>
> Reply-To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> <us...@cloudstack.apache.org>>
> Date: Thursday 4 April 2013 1:18 PM
> To: "cloudstack-users@incubator.apache.org<mailto:
> cloudstack-users@incubator.apache.org>" <
> cloudstack-users@incubator.apache.org<mailto:
> cloudstack-users@incubator.apache.org>>
> Subject: failing to create new instances
>
> Hi
>
> I have cs 3.02 (no upgrades is allowed if you dont remeber :P) and KVM
>
> after creating 1000 instances and they finished normally and been
> destroyed, I cant start any single instance at all
>
> attached is a portion from log file
> Thank you in advance..
>
> --
> Eng. Osamah Lutf Barakat
> Master Candidate
> Computer security
> Faculty of Engineering
> University Putra Malaysia
>
--
Eng. Osamah Lutf Barakat
Master Candidate
Computer security
Faculty of Engineering
University Putra Malaysia
Re: failing to create new instances
Posted by Nitin Mehta <Ni...@citrix.com>.
Can you please paste the logs @ http://paste.cloudstack.org/
ML doesn't allow you to attach files.
From: Osama Lotf <en...@gmail.com>>
Reply-To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Date: Thursday 4 April 2013 1:18 PM
To: "cloudstack-users@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>
Subject: failing to create new instances
Hi
I have cs 3.02 (no upgrades is allowed if you dont remeber :P) and KVM
after creating 1000 instances and they finished normally and been destroyed, I cant start any single instance at all
attached is a portion from log file
Thank you in advance..
--
Eng. Osamah Lutf Barakat
Master Candidate
Computer security
Faculty of Engineering
University Putra Malaysia