You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Prasanna Santhanam <ts...@apache.org> on 2013/06/22 19:19:24 UTC

Master broken on systemVM start

Unable to bring up the systemVMs on latest master. There is very
little detail in the logs: posted CLOUDSTACK-3137

Can someone please take a look? The regression tests running on the
weekend were blocked because of this.

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Re: Master broken on systemVM start

Posted by Prasanna Santhanam <ts...@apache.org>.
I did a mvn -Pdeveloper -Dsystemvm clean install and those
systemvm.iso(s) went away. So it must be within devcloud somewhere
that the iso existed. 

On Fri, Jun 28, 2013 at 10:41:39AM -0700, Hugo Trippaers wrote:
> Hey Prasanna,
> 
> I just did a quick test, these systemvm.iso files are generated by
> the default maven build (mvm -Dnonoss -Psystemvm install)
> 
> ./client/target/cloud-client-ui-4.2.0-SNAPSHOT/WEB-INF/classes/vms/systemvm.iso
> ./client/target/generated-webapp/WEB-INF/classes/vms/systemvm.iso
> ./services/console-proxy/server/dist/systemvm.iso
> 
> After cleaning (mvm -Dnonoss -Psystemvm clean) these files were removed as well.
> 
> Which targets did you run?
> 
> Cheers,
> 
> Hugo
> 
> 
> On Jun 28, 2013, at 10:21 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> 
> > I deleted all traces of systemvm.iso from my codebase
> > $ find . -name systemvm.iso | xargs -n1 rm -f
> > 
> > Then reverted my devcloud snapshot to old state and don't see this
> > error again.
> > 
> > I wonder by mvn doesn't do a good job of cleaning though :/
> > 
> > Thanks for the fixes to the bugs and the upcoming fixes for the new
> > ones. I'll be switching over the test infra to run object store backed
> > secondary only from next week so we should have more results. 
> > 
> > -- 
> > Prasanna.,
> > 
> > On Fri, Jun 28, 2013 at 04:35:29PM +0000, Min Chen wrote:
> >> Prasanna, glad to know that CLOUDSTACK-3137 is resolved. I fixed
> >> CLOUDSTACK-3249 yesterday for a corner case reported by Alena, but I could
> >> not reproduce CLOUDSTACK-3137 on my environment at all. For the JSON
> >> issue, I can only tell that it is inconsistent systemvm issue between
> >> management server and system vm. In 4.2, we changed ArrayTypeAdaptor to
> >> return full qualified command class name, so I don't quite get why you are
> >> still seeing the abbreviated command name in your log. Yes,
> >> StartupSecondaryStorageCommand is still there. From your log, it seems
> >> that your ssvm has the old code.
> >> 
> >> Thanks
> >> -min
> >> 
> >> On 6/28/13 12:03 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
> >> 
> >>> On Fri, Jun 28, 2013 at 12:17:55PM +0530, Prasanna Santhanam wrote:
> >>>> On Tue, Jun 25, 2013 at 04:07:08PM +0000, Min Chen wrote:
> >>>>> Json deserialization issue is caused by out-of-dated systemvm.iso on
> >>>> your
> >>>>> hypervisor host. You need rebuilding systemvm.iso and deployed to your
> >>>>> hypervisor host.
> >>>>> 
> >>>> 
> >>>> CLOUDSTACK-3137 exists and seems to be a problem with HA and
> >>>> systemVMs. The JSON serialization also exists for me despite
> >>>> rebuidling the systemvm with -Dsystemvm.
> >>> 
> >>> Ok - CLOUDSTACK-3137 _may_ have been fixed. the last few test-matrix
> >>> jobs have succeeded. Perhaps related to fix for CLOUDSTACK-3249.
> >>> 
> >>>> 
> >>>> I see the deserialization error when the ssvm agent sends this to
> >>>> management server:
> >>>> 
> >>>> 2013-06-28 06:42:50,609 DEBUG [cloud.agent.Agent] (Agent-Handler-1:)
> >>>> Sending Startup: Seq -1-2:  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags:
> >>>> 101, 
> >>>> [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter
> >>>> ":"1","pod":"1","guid":"s-1-VM-NfsSecondaryStorageResource","name":"s-1-V
> >>>> M","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"192.168.56
> >>>> .144","publicNetmask":"255.255.255.0","publicMacAddress":"06:5c:ac:00:00:
> >>>> 42","privateIpAddress":"192.168.56.217","privateMacAddress":"06:aa:84:00:
> >>>> 00:12","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.56.21
> >>>> 7","storageNetmask":"255.255.255.0","storageMacAddress":"06:aa:84:00:00:1
> >>>> 2","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
> >>>> 
> >>>> This happens both when I have a s3 store added and a regular nfs
> >>>> storage added.
> >>>> Is the startupcommand still reqd?
> >>>> 
> >>>> -- 
> >>>> Prasanna.,
> >>>> 
> >>>> ------------------------
> >>>> Powered by BigRock.com
> >>> 
> >>> -- 
> >>> Prasanna.,
> >>> 
> >>> ------------------------
> >>> Powered by BigRock.com
> >>> 
> > 
> > 
> > ------------------------
> > Powered by BigRock.com
> > 

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Re: Master broken on systemVM start

Posted by Hugo Trippaers <hu...@trippaers.nl>.
Hey Prasanna,

I just did a quick test, these systemvm.iso files are generated by the default maven build (mvm -Dnonoss -Psystemvm install)

./client/target/cloud-client-ui-4.2.0-SNAPSHOT/WEB-INF/classes/vms/systemvm.iso
./client/target/generated-webapp/WEB-INF/classes/vms/systemvm.iso
./services/console-proxy/server/dist/systemvm.iso

After cleaning (mvm -Dnonoss -Psystemvm clean) these files were removed as well.

Which targets did you run?

Cheers,

Hugo


On Jun 28, 2013, at 10:21 AM, Prasanna Santhanam <ts...@apache.org> wrote:

> I deleted all traces of systemvm.iso from my codebase
> $ find . -name systemvm.iso | xargs -n1 rm -f
> 
> Then reverted my devcloud snapshot to old state and don't see this
> error again.
> 
> I wonder by mvn doesn't do a good job of cleaning though :/
> 
> Thanks for the fixes to the bugs and the upcoming fixes for the new
> ones. I'll be switching over the test infra to run object store backed
> secondary only from next week so we should have more results. 
> 
> -- 
> Prasanna.,
> 
> On Fri, Jun 28, 2013 at 04:35:29PM +0000, Min Chen wrote:
>> Prasanna, glad to know that CLOUDSTACK-3137 is resolved. I fixed
>> CLOUDSTACK-3249 yesterday for a corner case reported by Alena, but I could
>> not reproduce CLOUDSTACK-3137 on my environment at all. For the JSON
>> issue, I can only tell that it is inconsistent systemvm issue between
>> management server and system vm. In 4.2, we changed ArrayTypeAdaptor to
>> return full qualified command class name, so I don't quite get why you are
>> still seeing the abbreviated command name in your log. Yes,
>> StartupSecondaryStorageCommand is still there. From your log, it seems
>> that your ssvm has the old code.
>> 
>> Thanks
>> -min
>> 
>> On 6/28/13 12:03 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>> 
>>> On Fri, Jun 28, 2013 at 12:17:55PM +0530, Prasanna Santhanam wrote:
>>>> On Tue, Jun 25, 2013 at 04:07:08PM +0000, Min Chen wrote:
>>>>> Json deserialization issue is caused by out-of-dated systemvm.iso on
>>>> your
>>>>> hypervisor host. You need rebuilding systemvm.iso and deployed to your
>>>>> hypervisor host.
>>>>> 
>>>> 
>>>> CLOUDSTACK-3137 exists and seems to be a problem with HA and
>>>> systemVMs. The JSON serialization also exists for me despite
>>>> rebuidling the systemvm with -Dsystemvm.
>>> 
>>> Ok - CLOUDSTACK-3137 _may_ have been fixed. the last few test-matrix
>>> jobs have succeeded. Perhaps related to fix for CLOUDSTACK-3249.
>>> 
>>>> 
>>>> I see the deserialization error when the ssvm agent sends this to
>>>> management server:
>>>> 
>>>> 2013-06-28 06:42:50,609 DEBUG [cloud.agent.Agent] (Agent-Handler-1:)
>>>> Sending Startup: Seq -1-2:  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags:
>>>> 101, 
>>>> [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter
>>>> ":"1","pod":"1","guid":"s-1-VM-NfsSecondaryStorageResource","name":"s-1-V
>>>> M","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"192.168.56
>>>> .144","publicNetmask":"255.255.255.0","publicMacAddress":"06:5c:ac:00:00:
>>>> 42","privateIpAddress":"192.168.56.217","privateMacAddress":"06:aa:84:00:
>>>> 00:12","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.56.21
>>>> 7","storageNetmask":"255.255.255.0","storageMacAddress":"06:aa:84:00:00:1
>>>> 2","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
>>>> 
>>>> This happens both when I have a s3 store added and a regular nfs
>>>> storage added.
>>>> Is the startupcommand still reqd?
>>>> 
>>>> -- 
>>>> Prasanna.,
>>>> 
>>>> ------------------------
>>>> Powered by BigRock.com
>>> 
>>> -- 
>>> Prasanna.,
>>> 
>>> ------------------------
>>> Powered by BigRock.com
>>> 
> 
> 
> ------------------------
> Powered by BigRock.com
> 


Re: Master broken on systemVM start

Posted by Prasanna Santhanam <ts...@apache.org>.
On Fri, Jun 28, 2013 at 05:38:19PM +0000, Edison Su wrote:
> Devcloud has a trick to clean up systemvm.iso:
> There is a bug in xapi on Ubuntu/debian(not sure the bug still there
> in the latest debian or not), which can't attach an ISO to a VM. I
> worked around the bug by creating a VDI on primary storage, copy
> systemvm iso into that VDI, then attach the vdi into system vm. The
> downside of this workaround is that, the VDI is not synced with
> systemvm iso.
> In order to clean up devcloud, need to either clean up primary
> storage, or restore devcloud from snapshot for each run in your test
> infra.
> 
Ok I'll have to explore that. I don't do a vdi reset each time so that
must be the problem.

-- 
Prasanna.,

------------------------
Powered by BigRock.com


RE: Master broken on systemVM start

Posted by Edison Su <Ed...@citrix.com>.
Devcloud has a trick to clean up systemvm.iso:
There is a bug in xapi on Ubuntu/debian(not sure the bug still there in the latest debian or not), which can't attach an ISO to a VM. I worked around the bug by creating a VDI on primary storage, copy systemvm iso into that VDI, then attach the vdi into system vm. The downside of this workaround is that, the VDI is not synced with systemvm iso.
In order to clean up devcloud, need to either clean up primary storage, or restore devcloud from snapshot for each run in your test infra.

> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: Friday, June 28, 2013 10:22 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Master broken on systemVM start
> 
> I deleted all traces of systemvm.iso from my codebase $ find . -name
> systemvm.iso | xargs -n1 rm -f
> 
> Then reverted my devcloud snapshot to old state and don't see this error
> again.
> 
> I wonder by mvn doesn't do a good job of cleaning though :/
> 
> Thanks for the fixes to the bugs and the upcoming fixes for the new ones. I'll
> be switching over the test infra to run object store backed secondary only
> from next week so we should have more results.
> 
> --
> Prasanna.,
> 
> On Fri, Jun 28, 2013 at 04:35:29PM +0000, Min Chen wrote:
> > Prasanna, glad to know that CLOUDSTACK-3137 is resolved. I fixed
> > CLOUDSTACK-3249 yesterday for a corner case reported by Alena, but I
> > could not reproduce CLOUDSTACK-3137 on my environment at all. For the
> > JSON issue, I can only tell that it is inconsistent systemvm issue
> > between management server and system vm. In 4.2, we changed
> > ArrayTypeAdaptor to return full qualified command class name, so I
> > don't quite get why you are still seeing the abbreviated command name
> > in your log. Yes, StartupSecondaryStorageCommand is still there. From
> > your log, it seems that your ssvm has the old code.
> >
> > Thanks
> > -min
> >
> > On 6/28/13 12:03 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
> >
> > >On Fri, Jun 28, 2013 at 12:17:55PM +0530, Prasanna Santhanam wrote:
> > >> On Tue, Jun 25, 2013 at 04:07:08PM +0000, Min Chen wrote:
> > >> > Json deserialization issue is caused by out-of-dated systemvm.iso
> > >> > on
> > >>your
> > >> > hypervisor host. You need rebuilding systemvm.iso and deployed to
> > >> > your hypervisor host.
> > >> >
> > >>
> > >> CLOUDSTACK-3137 exists and seems to be a problem with HA and
> > >> systemVMs. The JSON serialization also exists for me despite
> > >> rebuidling the systemvm with -Dsystemvm.
> > >
> > >Ok - CLOUDSTACK-3137 _may_ have been fixed. the last few test-matrix
> > >jobs have succeeded. Perhaps related to fix for CLOUDSTACK-3249.
> > >
> > >>
> > >> I see the deserialization error when the ssvm agent sends this to
> > >> management server:
> > >>
> > >> 2013-06-28 06:42:50,609 DEBUG [cloud.agent.Agent]
> > >>(Agent-Handler-1:) Sending Startup: Seq -1-2:  { Cmd , MgmtId: -1, via: -1,
> Ver: v1, Flags:
> > >>101,
> > >>[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","da
> taC
> > >>enter
> > >>":"1","pod":"1","guid":"s-1-VM-
> NfsSecondaryStorageResource","name":"
> > >>s-1-V
> > >>M","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"192.1
> > >>68.56
> > >>.144","publicNetmask":"255.255.255.0","publicMacAddress":"06:5c:ac:00
> :00:
> > >>42","privateIpAddress":"192.168.56.217","privateMacAddress":"06:aa:84
> :00:
> > >>00:12","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.
> > >>56.21
> > >>7","storageNetmask":"255.255.255.0","storageMacAddress":"06:aa:84:0
> 0
> > >>:00:1 2","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
> > >>
> > >> This happens both when I have a s3 store added and a regular nfs
> > >>storage added.
> > >> Is the startupcommand still reqd?
> > >>
> > >> --
> > >> Prasanna.,
> > >>
> > >> ------------------------
> > >> Powered by BigRock.com
> > >
> > >--
> > >Prasanna.,
> > >
> > >------------------------
> > >Powered by BigRock.com
> > >
> 
> 
> ------------------------
> Powered by BigRock.com


Re: Master broken on systemVM start

Posted by Prasanna Santhanam <ts...@apache.org>.
I deleted all traces of systemvm.iso from my codebase
$ find . -name systemvm.iso | xargs -n1 rm -f

Then reverted my devcloud snapshot to old state and don't see this
error again.

I wonder by mvn doesn't do a good job of cleaning though :/

Thanks for the fixes to the bugs and the upcoming fixes for the new
ones. I'll be switching over the test infra to run object store backed
secondary only from next week so we should have more results. 

-- 
Prasanna.,

On Fri, Jun 28, 2013 at 04:35:29PM +0000, Min Chen wrote:
> Prasanna, glad to know that CLOUDSTACK-3137 is resolved. I fixed
> CLOUDSTACK-3249 yesterday for a corner case reported by Alena, but I could
> not reproduce CLOUDSTACK-3137 on my environment at all. For the JSON
> issue, I can only tell that it is inconsistent systemvm issue between
> management server and system vm. In 4.2, we changed ArrayTypeAdaptor to
> return full qualified command class name, so I don't quite get why you are
> still seeing the abbreviated command name in your log. Yes,
> StartupSecondaryStorageCommand is still there. From your log, it seems
> that your ssvm has the old code.
> 
> Thanks
> -min
> 
> On 6/28/13 12:03 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
> 
> >On Fri, Jun 28, 2013 at 12:17:55PM +0530, Prasanna Santhanam wrote:
> >> On Tue, Jun 25, 2013 at 04:07:08PM +0000, Min Chen wrote:
> >> > Json deserialization issue is caused by out-of-dated systemvm.iso on
> >>your
> >> > hypervisor host. You need rebuilding systemvm.iso and deployed to your
> >> > hypervisor host.
> >> > 
> >> 
> >> CLOUDSTACK-3137 exists and seems to be a problem with HA and
> >> systemVMs. The JSON serialization also exists for me despite
> >> rebuidling the systemvm with -Dsystemvm.
> >
> >Ok - CLOUDSTACK-3137 _may_ have been fixed. the last few test-matrix
> >jobs have succeeded. Perhaps related to fix for CLOUDSTACK-3249.
> >
> >> 
> >> I see the deserialization error when the ssvm agent sends this to
> >> management server:
> >> 
> >> 2013-06-28 06:42:50,609 DEBUG [cloud.agent.Agent] (Agent-Handler-1:)
> >>Sending Startup: Seq -1-2:  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags:
> >>101, 
> >>[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter
> >>":"1","pod":"1","guid":"s-1-VM-NfsSecondaryStorageResource","name":"s-1-V
> >>M","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"192.168.56
> >>.144","publicNetmask":"255.255.255.0","publicMacAddress":"06:5c:ac:00:00:
> >>42","privateIpAddress":"192.168.56.217","privateMacAddress":"06:aa:84:00:
> >>00:12","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.56.21
> >>7","storageNetmask":"255.255.255.0","storageMacAddress":"06:aa:84:00:00:1
> >>2","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
> >> 
> >> This happens both when I have a s3 store added and a regular nfs
> >>storage added.
> >> Is the startupcommand still reqd?
> >> 
> >> -- 
> >> Prasanna.,
> >> 
> >> ------------------------
> >> Powered by BigRock.com
> >
> >-- 
> >Prasanna.,
> >
> >------------------------
> >Powered by BigRock.com
> >


------------------------
Powered by BigRock.com


Re: Master broken on systemVM start

Posted by Min Chen <mi...@citrix.com>.
Prasanna, glad to know that CLOUDSTACK-3137 is resolved. I fixed
CLOUDSTACK-3249 yesterday for a corner case reported by Alena, but I could
not reproduce CLOUDSTACK-3137 on my environment at all. For the JSON
issue, I can only tell that it is inconsistent systemvm issue between
management server and system vm. In 4.2, we changed ArrayTypeAdaptor to
return full qualified command class name, so I don't quite get why you are
still seeing the abbreviated command name in your log. Yes,
StartupSecondaryStorageCommand is still there. From your log, it seems
that your ssvm has the old code.

Thanks
-min

On 6/28/13 12:03 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:

>On Fri, Jun 28, 2013 at 12:17:55PM +0530, Prasanna Santhanam wrote:
>> On Tue, Jun 25, 2013 at 04:07:08PM +0000, Min Chen wrote:
>> > Json deserialization issue is caused by out-of-dated systemvm.iso on
>>your
>> > hypervisor host. You need rebuilding systemvm.iso and deployed to your
>> > hypervisor host.
>> > 
>> 
>> CLOUDSTACK-3137 exists and seems to be a problem with HA and
>> systemVMs. The JSON serialization also exists for me despite
>> rebuidling the systemvm with -Dsystemvm.
>
>Ok - CLOUDSTACK-3137 _may_ have been fixed. the last few test-matrix
>jobs have succeeded. Perhaps related to fix for CLOUDSTACK-3249.
>
>> 
>> I see the deserialization error when the ssvm agent sends this to
>> management server:
>> 
>> 2013-06-28 06:42:50,609 DEBUG [cloud.agent.Agent] (Agent-Handler-1:)
>>Sending Startup: Seq -1-2:  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags:
>>101, 
>>[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter
>>":"1","pod":"1","guid":"s-1-VM-NfsSecondaryStorageResource","name":"s-1-V
>>M","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"192.168.56
>>.144","publicNetmask":"255.255.255.0","publicMacAddress":"06:5c:ac:00:00:
>>42","privateIpAddress":"192.168.56.217","privateMacAddress":"06:aa:84:00:
>>00:12","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.56.21
>>7","storageNetmask":"255.255.255.0","storageMacAddress":"06:aa:84:00:00:1
>>2","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
>> 
>> This happens both when I have a s3 store added and a regular nfs
>>storage added.
>> Is the startupcommand still reqd?
>> 
>> -- 
>> Prasanna.,
>> 
>> ------------------------
>> Powered by BigRock.com
>
>-- 
>Prasanna.,
>
>------------------------
>Powered by BigRock.com
>


Re: Master broken on systemVM start

Posted by Prasanna Santhanam <ts...@apache.org>.
On Fri, Jun 28, 2013 at 12:17:55PM +0530, Prasanna Santhanam wrote:
> On Tue, Jun 25, 2013 at 04:07:08PM +0000, Min Chen wrote:
> > Json deserialization issue is caused by out-of-dated systemvm.iso on your
> > hypervisor host. You need rebuilding systemvm.iso and deployed to your
> > hypervisor host.
> > 
> 
> CLOUDSTACK-3137 exists and seems to be a problem with HA and
> systemVMs. The JSON serialization also exists for me despite
> rebuidling the systemvm with -Dsystemvm.

Ok - CLOUDSTACK-3137 _may_ have been fixed. the last few test-matrix
jobs have succeeded. Perhaps related to fix for CLOUDSTACK-3249.

> 
> I see the deserialization error when the ssvm agent sends this to
> management server:
> 
> 2013-06-28 06:42:50,609 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Sending Startup: Seq -1-2:  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-1-VM-NfsSecondaryStorageResource","name":"s-1-VM","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"192.168.56.144","publicNetmask":"255.255.255.0","publicMacAddress":"06:5c:ac:00:00:42","privateIpAddress":"192.168.56.217","privateMacAddress":"06:aa:84:00:00:12","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.56.217","storageNetmask":"255.255.255.0","storageMacAddress":"06:aa:84:00:00:12","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
> 
> This happens both when I have a s3 store added and a regular nfs storage added.
> Is the startupcommand still reqd?
> 
> -- 
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Re: Master broken on systemVM start

Posted by Prasanna Santhanam <ts...@apache.org>.
On Tue, Jun 25, 2013 at 04:07:08PM +0000, Min Chen wrote:
> Json deserialization issue is caused by out-of-dated systemvm.iso on your
> hypervisor host. You need rebuilding systemvm.iso and deployed to your
> hypervisor host.
> 

CLOUDSTACK-3137 exists and seems to be a problem with HA and
systemVMs. The JSON serialization also exists for me despite
rebuidling the systemvm with -Dsystemvm.

I see the deserialization error when the ssvm agent sends this to
management server:

2013-06-28 06:42:50,609 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Sending Startup: Seq -1-2:  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-1-VM-NfsSecondaryStorageResource","name":"s-1-VM","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"192.168.56.144","publicNetmask":"255.255.255.0","publicMacAddress":"06:5c:ac:00:00:42","privateIpAddress":"192.168.56.217","privateMacAddress":"06:aa:84:00:00:12","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.56.217","storageNetmask":"255.255.255.0","storageMacAddress":"06:aa:84:00:00:12","resourceName":"NfsSecondaryStorageResource","wait":0}}] }

This happens both when I have a s3 store added and a regular nfs storage added.
Is the startupcommand still reqd?

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Re: Master broken on systemVM start

Posted by Min Chen <mi...@citrix.com>.
Json deserialization issue is caused by out-of-dated systemvm.iso on your
hypervisor host. You need rebuilding systemvm.iso and deployed to your
hypervisor host.

Thanks
-min

On 6/25/13 12:03 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:

>Just to be clear, the issue 3137 is related to systemVM HA and isn't
>related to the JSON serialization. The JSON serialization is probably
>because you will need to upgrade your agent to the latest. I don't see
>the problem with KVM, Xen and XCP on the test infrastructure.
>
>On Tue, Jun 25, 2013 at 06:58:52AM +0000, Rajesh Battala wrote:
>> Am facing the same issue while adding the kvm host.
>> 
>> 
>>"localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images
>>","poolType":"Filesystem","capacityBytes":52844687360,"availableBytes":49
>>413353472},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage
>>","dataCenter":"1","pod":"1","guid":"eb98ae45-6462-3878-9dfe-376b05a1bdfe
>>-LibvirtComputingResource","name":"kvm56","version":"4.2.0-SNAPSHOT","res
>>ourceName":"LibvirtComputingResource","contextMap":{},"wait":0}}] given
>>the type class [Lcom.cloud.agent.api.Command;
>>         at 
>>com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserial
>>izerExceptionWrapper.java:64)
>>         at 
>>com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonD
>>eserializationVisitor.java:92)
>>         at 
>>com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDe
>>serializationVisitor.java:80)
>>         at 
>>com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
>>         at 
>>com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeser
>>ializationContextDefault.java:67)
>>         at 
>>com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeseria
>>lizationContextDefault.java:52)
>>         at com.google.gson.Gson.fromJson(Gson.java:551)
>>         at com.google.gson.Gson.fromJson(Gson.java:498)
>>         at 
>>com.cloud.agent.transport.Request.getCommands(Request.java:235)
>>         at 
>>com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(Agen
>>tManagerImpl.java:1221)
>>         at 
>>com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManager
>>Impl.java:1374)
>>         at 
>>com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.d
>>oTask(ClusteredAgentManagerImpl.java:659)
>>         at com.cloud.utils.nio.Task.run(Task.java:83)
>>         at 
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:1110)
>>         at 
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a:603)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
>>com.cloud.agent.api.com.cloud.agent.api.StartupRoutingCommand
>>         at 
>>com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.j
>>ava:79)
>>         at 
>>com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.j
>>ava:37)
>>         at 
>>com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserial
>>izerExceptionWrapper.java:51)
>>         ... 15 more
>> 
>-- 
>Prasanna.,
>
>------------------------
>Powered by BigRock.com
>


Re: Master broken on systemVM start

Posted by Prasanna Santhanam <ts...@apache.org>.
Just to be clear, the issue 3137 is related to systemVM HA and isn't
related to the JSON serialization. The JSON serialization is probably
because you will need to upgrade your agent to the latest. I don't see
the problem with KVM, Xen and XCP on the test infrastructure.

On Tue, Jun 25, 2013 at 06:58:52AM +0000, Rajesh Battala wrote:
> Am facing the same issue while adding the kvm host. 
> 
> "localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":52844687360,"availableBytes":49413353472},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"eb98ae45-6462-3878-9dfe-376b05a1bdfe-LibvirtComputingResource","name":"kvm56","version":"4.2.0-SNAPSHOT","resourceName":"LibvirtComputingResource","contextMap":{},"wait":0}}] given the type class [Lcom.cloud.agent.api.Command;
>         at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:64)
>         at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
>         at com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
>         at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
>         at com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
>         at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
>         at com.google.gson.Gson.fromJson(Gson.java:551)
>         at com.google.gson.Gson.fromJson(Gson.java:498)
>         at com.cloud.agent.transport.Request.getCommands(Request.java:235)
>         at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1221)
>         at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
>         at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:659)
>         at com.cloud.utils.nio.Task.run(Task.java:83)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find com.cloud.agent.api.com.cloud.agent.api.StartupRoutingCommand
>         at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:79)
>         at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:37)
>         at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
>         ... 15 more
> 
-- 
Prasanna.,

------------------------
Powered by BigRock.com


RE: Master broken on systemVM start

Posted by Rajesh Battala <ra...@citrix.com>.
Am facing the same issue while adding the kvm host. 

"localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":52844687360,"availableBytes":49413353472},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"eb98ae45-6462-3878-9dfe-376b05a1bdfe-LibvirtComputingResource","name":"kvm56","version":"4.2.0-SNAPSHOT","resourceName":"LibvirtComputingResource","contextMap":{},"wait":0}}] given the type class [Lcom.cloud.agent.api.Command;
        at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:64)
        at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:498)
        at com.cloud.agent.transport.Request.getCommands(Request.java:235)
        at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1221)
        at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
        at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:659)
        at com.cloud.utils.nio.Task.run(Task.java:83)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find com.cloud.agent.api.com.cloud.agent.api.StartupRoutingCommand
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:79)
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:37)
        at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        ... 15 more

> -----Original Message-----
> From: Rajesh Battala [mailto:rajesh.battala@citrix.com]
> Sent: Tuesday, June 25, 2013 12:24 PM
> To: dev@cloudstack.apache.org
> Subject: RE: Master broken on systemVM start
> 
> I guess after object store merge there are changes with json library. The
> same json deserialization is happening with kvm agent.
> 
> > -----Original Message-----
> > From: Nitin Mehta [mailto:Nitin.Mehta@citrix.com]
> > Sent: Tuesday, June 25, 2013 12:21 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: Master broken on systemVM start
> >
> > After object store merge did you guys built the systemvm to get the
> > new systemvm iso ? I haven't tried it myself but contemplating that.
> >
> > On 25/06/13 12:15 PM, "Pranav Saxena" <pr...@citrix.com>
> wrote:
> >
> > >+1 , I am getting it too !
> > >
> > >-----Original Message-----
> > >From: Sanjay Tripathi [mailto:sanjay.tripathi@citrix.com]
> > >Sent: Tuesday, June 25, 2013 12:03 PM
> > >To: dev@cloudstack.apache.org
> > >Subject: RE: Master broken on systemVM start
> > >
> > >I am also getting this exception on latest master:
> > >
> > >WARN  [utils.nio.Task] (AgentManager-Handler-2:) Caught the following
> > >exception but pushing on
> > >com.google.gson.JsonParseException: The JsonDeserializer
> > >com.cloud.agent.transport.ArrayTypeAdaptor@18e6ae2 failed to
> > >deserialize json object
> >
> >[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCe
> > nter"
> > >:"1","pod":"1","guid":"s-3-VM-NfsSecondaryStorageResource","name":"s-
> > >3-
> > VM"
> > >,"version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"10.102.1
> > >96
> > >.13
> > >0","publicNetmask":"255.255.255.0","publicMacAddress":"06:a3:68:00:00
> > >:1
> > >5",
> > >"privateIpAddress":"10.102.192.91","privateMacAddress":"06:f5:18:00:0
> > >0:02
> > "
> > >,"privateNetmask":"255.255.252.0","storageIpAddress":"10.102.192.91",
> > >"s
> > >tor
> > >ageNetmask":"255.255.252.0","storageMacAddress":"06:f5:18:00:00:02","
> > >re sou
> > >rceName":"NfsSecondaryStorageResource","contextMap":{},"wait":0}}]
> > >given the type class [Lcom.cloud.agent.api.Command;
> > >	at
> > >com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDese
> > >ri
> > >ali
> > >zerExceptionWrapper.java:64)
> > >	at
> > >com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(J
> > >so
> > >nDe
> > >serializationVisitor.java:92)
> > >	at
> > >com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(Js
> > >on
> > >Des
> > >erializationVisitor.java:80)
> > >	at
> > com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
> > >	at
> > >com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonD
> > >es
> > >eri
> > >alizationContextDefault.java:67)
> > >	at
> > >com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDes
> > >er
> > >ial
> > >izationContextDefault.java:52)
> > >	at com.google.gson.Gson.fromJson(Gson.java:551)
> > >	at com.google.gson.Gson.fromJson(Gson.java:498)
> > >	at
> > com.cloud.agent.transport.Request.getCommands(Request.java:235)
> > >	at
> >
> >com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processReques
> > t(Ag
> > >ent
> > >ManagerImpl.java:1193)
> > >	at
> >
> >com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(Agent
> > Manag
> > >erI
> > >mpl.java:1346)
> > >	at
> >
> >com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHan
> > dler
> > >.do
> > >Task(ClusteredAgentManagerImpl.java:659)
> > >	at com.cloud.utils.nio.Task.run(Task.java:83)
> > >	at
> > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> > >ja
> > va:
> > >1145)
> > >	at
> > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> > >.j
> > >ava
> > >:615)
> > >	at java.lang.Thread.run(Thread.java:722)
> > >Caused by: com.cloud.utils.exception.CloudRuntimeException: can't
> > >find StartupSecondaryStorageCommand
> > >	at
> > >com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdapt
> > >or
> > >.ja
> > >va:77)
> > >	at
> > >com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdapt
> > >or
> > >.ja
> > >va:37)
> > >	at
> > >com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDese
> > >ri
> > >ali
> > >zerExceptionWrapper.java:51)
> > >	... 15 more
> > >
> > >--Sanjay
> > >
> > >> -----Original Message-----
> > >> From: Prasanna Santhanam [mailto:tsp@apache.org]
> > >> Sent: Saturday, June 22, 2013 10:49 PM
> > >> To: CloudStack Dev
> > >> Subject: Master broken on systemVM start
> > >>
> > >> Unable to bring up the systemVMs on latest master. There is very
> > >> little detail in the logs: posted CLOUDSTACK-3137
> > >>
> > >> Can someone please take a look? The regression tests running on the
> > >> weekend were blocked because of this.
> > >>
> > >> --
> > >> Prasanna.,
> > >>
> > >> ------------------------
> > >> Powered by BigRock.com
> > >


RE: Master broken on systemVM start

Posted by Rajesh Battala <ra...@citrix.com>.
I guess after object store merge there are changes with json library. The same json deserialization is happening with kvm agent.

> -----Original Message-----
> From: Nitin Mehta [mailto:Nitin.Mehta@citrix.com]
> Sent: Tuesday, June 25, 2013 12:21 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Master broken on systemVM start
> 
> After object store merge did you guys built the systemvm to get the new
> systemvm iso ? I haven't tried it myself but contemplating that.
> 
> On 25/06/13 12:15 PM, "Pranav Saxena" <pr...@citrix.com> wrote:
> 
> >+1 , I am getting it too !
> >
> >-----Original Message-----
> >From: Sanjay Tripathi [mailto:sanjay.tripathi@citrix.com]
> >Sent: Tuesday, June 25, 2013 12:03 PM
> >To: dev@cloudstack.apache.org
> >Subject: RE: Master broken on systemVM start
> >
> >I am also getting this exception on latest master:
> >
> >WARN  [utils.nio.Task] (AgentManager-Handler-2:) Caught the following
> >exception but pushing on
> >com.google.gson.JsonParseException: The JsonDeserializer
> >com.cloud.agent.transport.ArrayTypeAdaptor@18e6ae2 failed to
> >deserialize json object
> >[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCe
> nter"
> >:"1","pod":"1","guid":"s-3-VM-NfsSecondaryStorageResource","name":"s-3-
> VM"
> >,"version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"10.102.196
> >.13
> >0","publicNetmask":"255.255.255.0","publicMacAddress":"06:a3:68:00:00:1
> >5",
> >"privateIpAddress":"10.102.192.91","privateMacAddress":"06:f5:18:00:00:02
> "
> >,"privateNetmask":"255.255.252.0","storageIpAddress":"10.102.192.91","s
> >tor
> >ageNetmask":"255.255.252.0","storageMacAddress":"06:f5:18:00:00:02","re
> >sou rceName":"NfsSecondaryStorageResource","contextMap":{},"wait":0}}]
> >given the type class [Lcom.cloud.agent.api.Command;
> >	at
> >com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeseri
> >ali
> >zerExceptionWrapper.java:64)
> >	at
> >com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(Jso
> >nDe
> >serializationVisitor.java:92)
> >	at
> >com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(Json
> >Des
> >erializationVisitor.java:80)
> >	at
> com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
> >	at
> >com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDes
> >eri
> >alizationContextDefault.java:67)
> >	at
> >com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeser
> >ial
> >izationContextDefault.java:52)
> >	at com.google.gson.Gson.fromJson(Gson.java:551)
> >	at com.google.gson.Gson.fromJson(Gson.java:498)
> >	at
> com.cloud.agent.transport.Request.getCommands(Request.java:235)
> >	at
> >com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processReques
> t(Ag
> >ent
> >ManagerImpl.java:1193)
> >	at
> >com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(Agent
> Manag
> >erI
> >mpl.java:1346)
> >	at
> >com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHan
> dler
> >.do
> >Task(ClusteredAgentManagerImpl.java:659)
> >	at com.cloud.utils.nio.Task.run(Task.java:83)
> >	at
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> va:
> >1145)
> >	at
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> >ava
> >:615)
> >	at java.lang.Thread.run(Thread.java:722)
> >Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
> >StartupSecondaryStorageCommand
> >	at
> >com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor
> >.ja
> >va:77)
> >	at
> >com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor
> >.ja
> >va:37)
> >	at
> >com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeseri
> >ali
> >zerExceptionWrapper.java:51)
> >	... 15 more
> >
> >--Sanjay
> >
> >> -----Original Message-----
> >> From: Prasanna Santhanam [mailto:tsp@apache.org]
> >> Sent: Saturday, June 22, 2013 10:49 PM
> >> To: CloudStack Dev
> >> Subject: Master broken on systemVM start
> >>
> >> Unable to bring up the systemVMs on latest master. There is very
> >> little detail in the logs: posted CLOUDSTACK-3137
> >>
> >> Can someone please take a look? The regression tests running on the
> >> weekend were blocked because of this.
> >>
> >> --
> >> Prasanna.,
> >>
> >> ------------------------
> >> Powered by BigRock.com
> >


Re: Master broken on systemVM start

Posted by Nitin Mehta <Ni...@citrix.com>.
After object store merge did you guys built the systemvm to get the new
systemvm iso ? I haven't tried it myself but contemplating that.

On 25/06/13 12:15 PM, "Pranav Saxena" <pr...@citrix.com> wrote:

>+1 , I am getting it too !
>
>-----Original Message-----
>From: Sanjay Tripathi [mailto:sanjay.tripathi@citrix.com]
>Sent: Tuesday, June 25, 2013 12:03 PM
>To: dev@cloudstack.apache.org
>Subject: RE: Master broken on systemVM start
>
>I am also getting this exception on latest master:
>
>WARN  [utils.nio.Task] (AgentManager-Handler-2:) Caught the following
>exception but pushing on
>com.google.gson.JsonParseException: The JsonDeserializer
>com.cloud.agent.transport.ArrayTypeAdaptor@18e6ae2 failed to deserialize
>json object 
>[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter"
>:"1","pod":"1","guid":"s-3-VM-NfsSecondaryStorageResource","name":"s-3-VM"
>,"version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"10.102.196.13
>0","publicNetmask":"255.255.255.0","publicMacAddress":"06:a3:68:00:00:15",
>"privateIpAddress":"10.102.192.91","privateMacAddress":"06:f5:18:00:00:02"
>,"privateNetmask":"255.255.252.0","storageIpAddress":"10.102.192.91","stor
>ageNetmask":"255.255.252.0","storageMacAddress":"06:f5:18:00:00:02","resou
>rceName":"NfsSecondaryStorageResource","contextMap":{},"wait":0}}] given
>the type class [Lcom.cloud.agent.api.Command;
>	at 
>com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeseriali
>zerExceptionWrapper.java:64)
>	at 
>com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDe
>serializationVisitor.java:92)
>	at 
>com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDes
>erializationVisitor.java:80)
>	at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
>	at 
>com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeseri
>alizationContextDefault.java:67)
>	at 
>com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserial
>izationContextDefault.java:52)
>	at com.google.gson.Gson.fromJson(Gson.java:551)
>	at com.google.gson.Gson.fromJson(Gson.java:498)
>	at com.cloud.agent.transport.Request.getCommands(Request.java:235)
>	at 
>com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(Agent
>ManagerImpl.java:1193)
>	at 
>com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerI
>mpl.java:1346)
>	at 
>com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.do
>Task(ClusteredAgentManagerImpl.java:659)
>	at com.cloud.utils.nio.Task.run(Task.java:83)
>	at 
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1145)
>	at 
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:615)
>	at java.lang.Thread.run(Thread.java:722)
>Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
>StartupSecondaryStorageCommand
>	at 
>com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.ja
>va:77)
>	at 
>com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.ja
>va:37)
>	at 
>com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeseriali
>zerExceptionWrapper.java:51)
>	... 15 more
>
>--Sanjay
>
>> -----Original Message-----
>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>> Sent: Saturday, June 22, 2013 10:49 PM
>> To: CloudStack Dev
>> Subject: Master broken on systemVM start
>> 
>> Unable to bring up the systemVMs on latest master. There is very
>> little detail in the logs: posted CLOUDSTACK-3137
>> 
>> Can someone please take a look? The regression tests running on the
>> weekend were blocked because of this.
>> 
>> --
>> Prasanna.,
>> 
>> ------------------------
>> Powered by BigRock.com
>


RE: Master broken on systemVM start

Posted by Pranav Saxena <pr...@citrix.com>.
+1 , I am getting it too !

-----Original Message-----
From: Sanjay Tripathi [mailto:sanjay.tripathi@citrix.com] 
Sent: Tuesday, June 25, 2013 12:03 PM
To: dev@cloudstack.apache.org
Subject: RE: Master broken on systemVM start

I am also getting this exception on latest master:

WARN  [utils.nio.Task] (AgentManager-Handler-2:) Caught the following exception but pushing on
com.google.gson.JsonParseException: The JsonDeserializer com.cloud.agent.transport.ArrayTypeAdaptor@18e6ae2 failed to deserialize json object [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-3-VM-NfsSecondaryStorageResource","name":"s-3-VM","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"10.102.196.130","publicNetmask":"255.255.255.0","publicMacAddress":"06:a3:68:00:00:15","privateIpAddress":"10.102.192.91","privateMacAddress":"06:f5:18:00:00:02","privateNetmask":"255.255.252.0","storageIpAddress":"10.102.192.91","storageNetmask":"255.255.252.0","storageMacAddress":"06:f5:18:00:00:02","resourceName":"NfsSecondaryStorageResource","contextMap":{},"wait":0}}] given the type class [Lcom.cloud.agent.api.Command;
	at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:64)
	at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
	at com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
	at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
	at com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
	at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
	at com.google.gson.Gson.fromJson(Gson.java:551)
	at com.google.gson.Gson.fromJson(Gson.java:498)
	at com.cloud.agent.transport.Request.getCommands(Request.java:235)
	at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1193)
	at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1346)
	at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:659)
	at com.cloud.utils.nio.Task.run(Task.java:83)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find StartupSecondaryStorageCommand
	at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:77)
	at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:37)
	at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
	... 15 more

--Sanjay

> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: Saturday, June 22, 2013 10:49 PM
> To: CloudStack Dev
> Subject: Master broken on systemVM start
> 
> Unable to bring up the systemVMs on latest master. There is very 
> little detail in the logs: posted CLOUDSTACK-3137
> 
> Can someone please take a look? The regression tests running on the 
> weekend were blocked because of this.
> 
> --
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com


RE: Master broken on systemVM start

Posted by Sanjay Tripathi <sa...@citrix.com>.
I am also getting this exception on latest master:

WARN  [utils.nio.Task] (AgentManager-Handler-2:) Caught the following exception but pushing on
com.google.gson.JsonParseException: The JsonDeserializer com.cloud.agent.transport.ArrayTypeAdaptor@18e6ae2 failed to deserialize json object [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-3-VM-NfsSecondaryStorageResource","name":"s-3-VM","version":"4.2.0-SNAPSHOT","iqn":"NoIqn","publicIpAddress":"10.102.196.130","publicNetmask":"255.255.255.0","publicMacAddress":"06:a3:68:00:00:15","privateIpAddress":"10.102.192.91","privateMacAddress":"06:f5:18:00:00:02","privateNetmask":"255.255.252.0","storageIpAddress":"10.102.192.91","storageNetmask":"255.255.252.0","storageMacAddress":"06:f5:18:00:00:02","resourceName":"NfsSecondaryStorageResource","contextMap":{},"wait":0}}] given the type class [Lcom.cloud.agent.api.Command;
	at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:64)
	at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
	at com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
	at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
	at com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
	at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
	at com.google.gson.Gson.fromJson(Gson.java:551)
	at com.google.gson.Gson.fromJson(Gson.java:498)
	at com.cloud.agent.transport.Request.getCommands(Request.java:235)
	at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1193)
	at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1346)
	at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:659)
	at com.cloud.utils.nio.Task.run(Task.java:83)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find StartupSecondaryStorageCommand
	at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:77)
	at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:37)
	at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
	... 15 more

--Sanjay

> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: Saturday, June 22, 2013 10:49 PM
> To: CloudStack Dev
> Subject: Master broken on systemVM start
> 
> Unable to bring up the systemVMs on latest master. There is very little detail
> in the logs: posted CLOUDSTACK-3137
> 
> Can someone please take a look? The regression tests running on the
> weekend were blocked because of this.
> 
> --
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com