You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Natalia Costas Lago <na...@cesga.es> on 2018/05/16 15:44:00 UTC
Cloudtack manager UI returns Error 404
Dear all,
For some reason our cloudstack manager died and now we are not able to
make it work. Our cloudstack version is 4.9.0.
We tried to do a clean install of the manager and recover the database
from the last backup, but we were not able to recover the service.
At present we can see in the logs that it almost boot, but at the end we
get this error:
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
[o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null) (logid:)
Network Manager will run the NetworkGarbageCollector every '600' seconds.
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO [c.c.a.ApiServer]
(Thread-12:null) (logid:) ApiServer listening on port 8096
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.SecondaryStorageManagerImpl] (localhost-startStop-1:null)
(logid:) Start secondary storage vm manager
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No appenders
could be found for logger (com.cloud.utils.db.ConnectionConcierge).
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN Please
initialize the log4j system properly.
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in thread
"CapacityChecker" java.lang.NoClassDefFoundError:
org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
java.util.TimerThread.mainLoop(Timer.java:555)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
java.util.TimerThread.run(Timer.java:505)
May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
java.lang.ClassNotFoundException:
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1892)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1735)
May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in thread
"Timer-1" java.lang.NoClassDefFoundError:
org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
May 16 17:34:24 cldmanager.srv.cesga.es server: at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
May 16 17:34:24 cldmanager.srv.cesga.es server: at
java.util.TimerThread.mainLoop(Timer.java:555)
May 16 17:34:24 cldmanager.srv.cesga.es server: at
java.util.TimerThread.run(Timer.java:505)
Any help is appreciated..
Kind regards,
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: natalia@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.
Re: Cloudtack manager UI returns Error 404
Posted by Suresh Kumar Anaparti <su...@gmail.com>.
Hi Natalia,
Thanks for the update. We missed that context for root causing the issue.
-Suresh
On Tue, May 22, 2018 at 4:38 PM, Natalia Costas Lago <na...@cesga.es>
wrote:
>
> Hi,
>
> We finally were able to recover the manager :/
>
> The problem was due to a database modification made by hand a couple of
> months ago in order to eliminate one of the zombie virtual machines
> (machines that cannot be expunged because they are dependent on a network
> that was deleted before).
>
> The state of one of the machines was modified and Cloudstack did not like
> that change (setting a VM state as Destroyed), this eliminated the VM for
> the web interface but we did not detect the problem as we did not restart
> the management server. All backups since then were wrong and for some
> reason the management server crashed last week.
>
> Now it is fully operational :)
>
> Thank you,
> Kind regards,
>
>
> El 21/05/2018 a las 12:19, Natalia Costas Lago escribió:
>
>>
>> This is what I can see in localhost log:
>>
>> cat /var/log/cloudstack/management/localhost.2018-05-21.log
>>
>> [...]
>>
>> SEVERE: Exception sending context initialized event to listener instance
>> of class org.apache.cloudstack.spring.module.web.CloudStackContextLoa
>> derListener
>> java.lang.NullPointerException
>> at com.cloud.vm.UserVmManagerImpl.loadVmDetailsInMapForExternal
>> DhcpIp(UserVmManagerImpl.java:2019)
>> at com.cloud.vm.UserVmManagerImpl.start(UserVmManagerImpl.java:
>> 1999)
>> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycle$1.with(CloudStackExtendedLifeCycle.java:75)
>> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycle.with(CloudStackExtendedLifeCycle.java:154)
>> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycle.startBeans(CloudStackExtendedLifeCycle.java:72)
>> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycleStart.run(CloudStackExtendedLifeCycleStart.java:46)
>> at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet$1.with(DefaultModuleDefinitionSet.java:105)
>> at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
>> at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
>> at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
>> at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
>> at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.startContexts(DefaultModuleDefinitionSet.java:97)
>> at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.load(DefaultModuleDefinitionSet.java:80)
>> at org.apache.cloudstack.spring.module.factory.ModuleBasedConte
>> xtFactory.loadModules(ModuleBasedContextFactory.java:37)
>> at org.apache.cloudstack.spring.module.factory.CloudStackSpring
>> Context.init(CloudStackSpringContext.java:71)
>> at org.apache.cloudstack.spring.module.factory.CloudStackSpring
>> Context.<init>(CloudStackSpringContext.java:58)
>> at org.apache.cloudstack.spring.module.factory.CloudStackSpring
>> Context.<init>(CloudStackSpringContext.java:62)
>> at org.apache.cloudstack.spring.module.web.CloudStackContextLoa
>> derListener.contextInitialized(CloudStackContextLoaderListener.java:52)
>> at org.apache.catalina.core.StandardContext.listenerStart(Stand
>> ardContext.java:5118)
>> at org.apache.catalina.core.StandardContext.startInternal(Stand
>> ardContext.java:5634)
>> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.
>> java:145)
>> at org.apache.catalina.core.ContainerBase.addChildInternal(Cont
>> ainerBase.java:899)
>> at org.apache.catalina.core.ContainerBase.addChild(ContainerBas
>> e.java:875)
>> at org.apache.catalina.core.StandardHost.addChild(StandardHost.
>> java:652)
>> at org.apache.catalina.startup.HostConfig.deployDirectory(HostC
>> onfig.java:1260)
>> at org.apache.catalina.startup.HostConfig$DeployDirectory.run(
>> HostConfig.java:2002)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:473)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1152)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:622)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> El 21/05/2018 a las 9:21, Natalia Costas Lago escribió:
>>
>>>
>>> It has been working with 1.8 for long...
>>>
>>> I switched to java 1.7, but java errors persist.
>>>
>>> We recovered backups from a couple of days prior to the disaster, but we
>>> are still unlucky, it seems the problem is also in the backups copy.
>>>
>>> I installed a fresh cloudstack and copied a recent database and the
>>> manager still gives and Exception but to a later point in the execution:
>>>
>>>
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Configuring CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Done Configuring CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
>>> (logid:) Loaded module context [ipmitool] in 171 ms
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
>>> (logid:) Starting module [system]
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Starting CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Done Starting CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
>>> (logid:) Starting module [core]
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Starting CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.h.m.HypervManagerImpl] (localhost-startStop-1:null) (logid:) Cleanup
>>> mounted mount points used in previous session
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
>>> Grabbing lock to check for database integrity.
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
>>> Performing database integrity check
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Starting
>>> Cluster manager, msid : 2199193518866
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterServiceServletContainer] (Thread-10:null) (logid:)
>>> Cluster service servlet container listening on port 9090
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Management
>>> server 2199193518866, runId 1526887063086 is being started
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Management
>>> server (host id : 2) is being started at 10.38.3.18:9090
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Cluster
>>> manager was started successfully
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:null) (logid:)
>>> Starting work
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN No appenders
>>> could be found for logger (com.cloud.utils.db.ConnectionConcierge).
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN Please
>>> initialize the log4j system properly.
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN See
>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: Exception in thread
>>> "Thread-10" java.lang.NoClassDefFoundError:
>>> org/apache/log4j/spi/ThrowableInformation
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.log4j.Category.forcedLog(Category.java:391)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.log4j.Category.error(Category.java:322)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> com.cloud.cluster.ClusterServiceServletContainer$
>>> ListenerThread.run(ClusterServiceServletContainer.java:168)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: Caused by:
>>> java.lang.ClassNotFoundException: org.apache.log4j.spi.Throwable
>>> Information
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(W
>>> ebappClassLoaderBase.java:1892)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(W
>>> ebappClassLoaderBase.java:1735)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: ... 4 more
>>>
>>>
>>>
>>> El 17/05/2018 a las 7:39, ilya musayev escribió:
>>>
>>>> You must be running java 1.7 - I assume Suresh was alluding to this.
>>>> We’ve seen this error with 1.8...
>>>>
>>>> On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
>>>> sureshkumar.anaparti@gmail.com> wrote:
>>>>
>>>> Hi Natalia,
>>>>>
>>>>> Any changes in the java version or system configuration?
>>>>>
>>>>> -Suresh
>>>>>
>>>>> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo <
>>>>> Dag.Sonstebo@shapeblue.com>
>>>>> wrote:
>>>>>
>>>>> Hi Natalia,
>>>>>>
>>>>>> My guess is you have some sort of corruption on your original host.
>>>>>> Since
>>>>>> the management service itself is stateless it’s generally much quicker
>>>>>>
>>>>> just
>>>>>
>>>>>> building a new one than trying to recover the old one.
>>>>>>
>>>>>> So my suggestions would be:
>>>>>> - Recover your DB to a dedicated DB host if you haven’t done so
>>>>>> already.
>>>>>> - Build a new management server and just point this to the original
>>>>>> DB as
>>>>>> described in http://docs.cloudstack.apache.org/projects/cloudstack-
>>>>>> installation/en/4.9/management-server/index.html#
>>>>>> additional-management-servers
>>>>>> NOTE: do not use the “–deploy-as” option as this will write a new
>>>>>> blank
>>>>>> database. If you used encryption keys during the original setup you
>>>>>> also
>>>>>> need to specify these.
>>>>>>
>>>>>> Once you have these two speaking you should in theory be back up and
>>>>>> running again.
>>>>>>
>>>>>> Regards,
>>>>>> Dag Sonstebo
>>>>>> Cloud Architect
>>>>>> ShapeBlue
>>>>>>
>>>>>>
>>>>>> Dag.Sonstebo@shapeblue.com
>>>>>> www.shapeblue.com
>>>>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>>>>>> @shapeblue
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 16/05/2018, 16:44, "Natalia Costas Lago" <na...@cesga.es> wrote:
>>>>>>
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> For some reason our cloudstack manager died and now we are not
>>>>>> able
>>>>>>
>>>>> to
>>>>>
>>>>>> make it work. Our cloudstack version is 4.9.0.
>>>>>>
>>>>>> We tried to do a clean install of the manager and recover the
>>>>>>
>>>>> database
>>>>>
>>>>>> from the last backup, but we were not able to recover the
>>>>>> service.
>>>>>>
>>>>>> At present we can see in the logs that it almost boot, but at
>>>>>> the end
>>>>>> we
>>>>>> get this error:
>>>>>>
>>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>>> [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null)
>>>>>> (logid:)
>>>>>> Network Manager will run the NetworkGarbageCollector every '600'
>>>>>> seconds.
>>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>>> [c.c.a.ApiServer]
>>>>>> (Thread-12:null) (logid:) ApiServer listening on port 8096
>>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>>> [o.a.c.s.SecondaryStorageManagerImpl]
>>>>>> (localhost-startStop-1:null)
>>>>>> (logid:) Start secondary storage vm manager
>>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
>>>>>> appenders
>>>>>> could be found for logger (com.cloud.utils.db.Connection
>>>>>> Concierge).
>>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN
>>>>>> Please
>>>>>> initialize the log4j system properly.
>>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
>>>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
>>>>>> info.
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in
>>>>>> thread
>>>>>> "CapacityChecker" java.lang.NoClassDefFoundError:
>>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>>>> ManagedContextTimerTask.java:27)
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> java.util.TimerThread.run(Timer.java:505)
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
>>>>>> java.lang.ClassNotFoundException:
>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>>>> WebappClassLoaderBase.java:1892)
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>>>> WebappClassLoaderBase.java:1735)
>>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
>>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in
>>>>>> thread
>>>>>> "Timer-1" java.lang.NoClassDefFoundError:
>>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>>>> ManagedContextTimerTask.java:27)
>>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>>> java.util.TimerThread.run(Timer.java:505)
>>>>>>
>>>>>> Any help is appreciated..
>>>>>>
>>>>>> Kind regards,
>>>>>>
>>>>>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>>>>>> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>>>>>>
>>>>>> --
>>>>>> ====================================================
>>>>>> Natalia Costas Lago
>>>>>> Senior Communications Technician
>>>>>> Galicia Supercomputing Centre (CESGA)
>>>>>> (CESGA on Twitter | CESGA on Facebook)
>>>>>>
>>>>>> Avenida de Vigo, s/n (Campus Vida)
>>>>>> 15705 Santiago de Compostela - SPAIN
>>>>>>
>>>>>> E-mail: natalia@cesga.es
>>>>>> Cell: +34 981 56 98 10 (ext. 237)
>>>>>> Fax: +34 981 59 46 16
>>>>>> Web: https://www.cesga.es/
>>>>>> ====================================================
>>>>>> [IMPORTANTE] La información contenida en este mensaje y
>>>>>> sus posibles documentos adjuntos es privada y confidencial
>>>>>> y está dirigida únicamente a su destinatario/a. Si usted no
>>>>>> es el/la destinatario/a original de este mensaje, por favor
>>>>>> elimínelo. La distribución o copia de este mensaje no está
>>>>>> autorizada.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>>> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>>>
>>>
>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>>
>>
> NOTA: Para cualquier incidencia/consulta/petición que requiera
> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>
> --
> ====================================================
> Natalia Costas Lago
> Senior Communications Technician
> Galicia Supercomputing Centre (CESGA)
> (CESGA on Twitter | CESGA on Facebook)
>
> Avenida de Vigo, s/n (Campus Vida)
> 15705 Santiago de Compostela - SPAIN
>
> E-mail: natalia@cesga.es
> Cell: +34 981 56 98 10 (ext. 237)
> Fax: +34 981 59 46 16
> Web: https://www.cesga.es/
> ====================================================
> [IMPORTANTE] La información contenida en este mensaje y
> sus posibles documentos adjuntos es privada y confidencial
> y está dirigida únicamente a su destinatario/a. Si usted no
> es el/la destinatario/a original de este mensaje, por favor
> elimínelo. La distribución o copia de este mensaje no está
> autorizada.
>
>
Re: Cloudtack manager UI returns Error 404
Posted by Natalia Costas Lago <na...@cesga.es>.
Hi,
We finally were able to recover the manager :/
The problem was due to a database modification made by hand a couple of
months ago in order to eliminate one of the zombie virtual machines
(machines that cannot be expunged because they are dependent on a
network that was deleted before).
The state of one of the machines was modified and Cloudstack did not
like that change (setting a VM state as Destroyed), this eliminated the
VM for the web interface but we did not detect the problem as we did not
restart the management server. All backups since then were wrong and for
some reason the management server crashed last week.
Now it is fully operational :)
Thank you,
Kind regards,
El 21/05/2018 a las 12:19, Natalia Costas Lago escribió:
>
> This is what I can see in localhost log:
>
> cat /var/log/cloudstack/management/localhost.2018-05-21.log
>
> [...]
>
> SEVERE: Exception sending context initialized event to listener
> instance of class
> org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener
> java.lang.NullPointerException
> at
> com.cloud.vm.UserVmManagerImpl.loadVmDetailsInMapForExternalDhcpIp(UserVmManagerImpl.java:2019)
> at
> com.cloud.vm.UserVmManagerImpl.start(UserVmManagerImpl.java:1999)
> at
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$1.with(CloudStackExtendedLifeCycle.java:75)
> at
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:154)
> at
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.startBeans(CloudStackExtendedLifeCycle.java:72)
> at
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycleStart.run(CloudStackExtendedLifeCycleStart.java:46)
> at
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$1.with(DefaultModuleDefinitionSet.java:105)
> at
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
> at
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
> at
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
> at
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
> at
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.startContexts(DefaultModuleDefinitionSet.java:97)
> at
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:80)
> at
> org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
> at
> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
> at
> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58)
> at
> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62)
> at
> org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> at
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
> at
> org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> El 21/05/2018 a las 9:21, Natalia Costas Lago escribió:
>>
>> It has been working with 1.8 for long...
>>
>> I switched to java 1.7, but java errors persist.
>>
>> We recovered backups from a couple of days prior to the disaster, but
>> we are still unlucky, it seems the problem is also in the backups copy.
>>
>> I installed a fresh cloudstack and copied a recent database and the
>> manager still gives and Exception but to a later point in the execution:
>>
>>
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>> (logid:) Configuring CloudStack Components
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>> (logid:) Done Configuring CloudStack Components
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
>> (localhost-startStop-1:null) (logid:) Loaded module context
>> [ipmitool] in 171 ms
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
>> (localhost-startStop-1:null) (logid:) Starting module [system]
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>> (logid:) Starting CloudStack Components
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>> (logid:) Done Starting CloudStack Components
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
>> (localhost-startStop-1:null) (logid:) Starting module [core]
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>> (logid:) Starting CloudStack Components
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.h.h.m.HypervManagerImpl] (localhost-startStop-1:null) (logid:)
>> Cleanup mounted mount points used in previous session
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null)
>> (logid:) Grabbing lock to check for database integrity.
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null)
>> (logid:) Performing database integrity check
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
>> Starting Cluster manager, msid : 2199193518866
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.c.ClusterServiceServletContainer] (Thread-10:null) (logid:)
>> Cluster service servlet container listening on port 9090
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
>> Management server 2199193518866, runId 1526887063086 is being started
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
>> Management server (host id : 2) is being started at 10.38.3.18:9090
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
>> Cluster manager was started successfully
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:null) (logid:)
>> Starting work
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:null) (logid:)
>> Starting work
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:null) (logid:)
>> Starting work
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:null) (logid:)
>> Starting work
>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:null) (logid:)
>> Starting work
>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN No
>> appenders could be found for logger
>> (com.cloud.utils.db.ConnectionConcierge).
>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN Please
>> initialize the log4j system properly.
>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN See
>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
>> May 21 09:19:55 cldmanager.srv.cesga.es server: Exception in thread
>> "Thread-10" java.lang.NoClassDefFoundError:
>> org/apache/log4j/spi/ThrowableInformation
>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>> org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>> org.apache.log4j.Category.forcedLog(Category.java:391)
>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>> org.apache.log4j.Category.error(Category.java:322)
>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>> com.cloud.cluster.ClusterServiceServletContainer$ListenerThread.run(ClusterServiceServletContainer.java:168)
>> May 21 09:19:55 cldmanager.srv.cesga.es server: Caused by:
>> java.lang.ClassNotFoundException:
>> org.apache.log4j.spi.ThrowableInformation
>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1892)
>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1735)
>> May 21 09:19:55 cldmanager.srv.cesga.es server: ... 4 more
>>
>>
>>
>> El 17/05/2018 a las 7:39, ilya musayev escribió:
>>> You must be running java 1.7 - I assume Suresh was alluding to this.
>>> We’ve seen this error with 1.8...
>>>
>>> On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
>>> sureshkumar.anaparti@gmail.com> wrote:
>>>
>>>> Hi Natalia,
>>>>
>>>> Any changes in the java version or system configuration?
>>>>
>>>> -Suresh
>>>>
>>>> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo
>>>> <Da...@shapeblue.com>
>>>> wrote:
>>>>
>>>>> Hi Natalia,
>>>>>
>>>>> My guess is you have some sort of corruption on your original
>>>>> host. Since
>>>>> the management service itself is stateless it’s generally much
>>>>> quicker
>>>> just
>>>>> building a new one than trying to recover the old one.
>>>>>
>>>>> So my suggestions would be:
>>>>> - Recover your DB to a dedicated DB host if you haven’t done so
>>>>> already.
>>>>> - Build a new management server and just point this to the
>>>>> original DB as
>>>>> described in http://docs.cloudstack.apache.org/projects/cloudstack-
>>>>> installation/en/4.9/management-server/index.html#
>>>>> additional-management-servers
>>>>> NOTE: do not use the “–deploy-as” option as this will write a new
>>>>> blank
>>>>> database. If you used encryption keys during the original setup
>>>>> you also
>>>>> need to specify these.
>>>>>
>>>>> Once you have these two speaking you should in theory be back up and
>>>>> running again.
>>>>>
>>>>> Regards,
>>>>> Dag Sonstebo
>>>>> Cloud Architect
>>>>> ShapeBlue
>>>>>
>>>>>
>>>>> Dag.Sonstebo@shapeblue.com
>>>>> www.shapeblue.com
>>>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>>>>> @shapeblue
>>>>>
>>>>>
>>>>>
>>>>> On 16/05/2018, 16:44, "Natalia Costas Lago" <na...@cesga.es> wrote:
>>>>>
>>>>>
>>>>> Dear all,
>>>>>
>>>>> For some reason our cloudstack manager died and now we are
>>>>> not able
>>>> to
>>>>> make it work. Our cloudstack version is 4.9.0.
>>>>>
>>>>> We tried to do a clean install of the manager and recover the
>>>> database
>>>>> from the last backup, but we were not able to recover the
>>>>> service.
>>>>>
>>>>> At present we can see in the logs that it almost boot, but at
>>>>> the end
>>>>> we
>>>>> get this error:
>>>>>
>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>> [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null)
>>>>> (logid:)
>>>>> Network Manager will run the NetworkGarbageCollector every '600'
>>>>> seconds.
>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>> [c.c.a.ApiServer]
>>>>> (Thread-12:null) (logid:) ApiServer listening on port 8096
>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>> [o.a.c.s.SecondaryStorageManagerImpl]
>>>>> (localhost-startStop-1:null)
>>>>> (logid:) Start secondary storage vm manager
>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
>>>>> appenders
>>>>> could be found for logger
>>>>> (com.cloud.utils.db.ConnectionConcierge).
>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN
>>>>> Please
>>>>> initialize the log4j system properly.
>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
>>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for
>>>>> more info.
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in
>>>>> thread
>>>>> "CapacityChecker" java.lang.NoClassDefFoundError:
>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>>> ManagedContextTimerTask.java:27)
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>> java.util.TimerThread.run(Timer.java:505)
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>>> WebappClassLoaderBase.java:1892)
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>>> WebappClassLoaderBase.java:1735)
>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in
>>>>> thread
>>>>> "Timer-1" java.lang.NoClassDefFoundError:
>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>>> ManagedContextTimerTask.java:27)
>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>> java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> Any help is appreciated..
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>>>>> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>>>>>
>>>>> --
>>>>> ====================================================
>>>>> Natalia Costas Lago
>>>>> Senior Communications Technician
>>>>> Galicia Supercomputing Centre (CESGA)
>>>>> (CESGA on Twitter | CESGA on Facebook)
>>>>>
>>>>> Avenida de Vigo, s/n (Campus Vida)
>>>>> 15705 Santiago de Compostela - SPAIN
>>>>>
>>>>> E-mail: natalia@cesga.es
>>>>> Cell: +34 981 56 98 10 (ext. 237)
>>>>> Fax: +34 981 59 46 16
>>>>> Web: https://www.cesga.es/
>>>>> ====================================================
>>>>> [IMPORTANTE] La información contenida en este mensaje y
>>>>> sus posibles documentos adjuntos es privada y confidencial
>>>>> y está dirigida únicamente a su destinatario/a. Si usted no
>>>>> es el/la destinatario/a original de este mensaje, por favor
>>>>> elimínelo. La distribución o copia de este mensaje no está
>>>>> autorizada.
>>>>>
>>>>>
>>>>>
>>>>>
>>
>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>>
>
> NOTA: Para cualquier incidencia/consulta/petición que requiera
> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: natalia@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.
Re: Cloudtack manager UI returns Error 404
Posted by Natalia Costas Lago <na...@cesga.es>.
This is what I can see in localhost log:
cat /var/log/cloudstack/management/localhost.2018-05-21.log
[...]
SEVERE: Exception sending context initialized event to listener instance
of class
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener
java.lang.NullPointerException
at
com.cloud.vm.UserVmManagerImpl.loadVmDetailsInMapForExternalDhcpIp(UserVmManagerImpl.java:2019)
at
com.cloud.vm.UserVmManagerImpl.start(UserVmManagerImpl.java:1999)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$1.with(CloudStackExtendedLifeCycle.java:75)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:154)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.startBeans(CloudStackExtendedLifeCycle.java:72)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycleStart.run(CloudStackExtendedLifeCycleStart.java:46)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$1.with(DefaultModuleDefinitionSet.java:105)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.startContexts(DefaultModuleDefinitionSet.java:97)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:80)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
at
org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)
El 21/05/2018 a las 9:21, Natalia Costas Lago escribió:
>
> It has been working with 1.8 for long...
>
> I switched to java 1.7, but java errors persist.
>
> We recovered backups from a couple of days prior to the disaster, but
> we are still unlucky, it seems the problem is also in the backups copy.
>
> I installed a fresh cloudstack and copied a recent database and the
> manager still gives and Exception but to a later point in the execution:
>
>
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
> (logid:) Configuring CloudStack Components
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
> (logid:) Done Configuring CloudStack Components
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (localhost-startStop-1:null) (logid:) Loaded module context [ipmitool]
> in 171 ms
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (localhost-startStop-1:null) (logid:) Starting module [system]
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
> (logid:) Starting CloudStack Components
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
> (logid:) Done Starting CloudStack Components
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (localhost-startStop-1:null) (logid:) Starting module [core]
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
> (logid:) Starting CloudStack Components
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.h.h.m.HypervManagerImpl] (localhost-startStop-1:null) (logid:)
> Cleanup mounted mount points used in previous session
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
> Grabbing lock to check for database integrity.
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
> Performing database integrity check
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
> Starting Cluster manager, msid : 2199193518866
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.c.ClusterServiceServletContainer] (Thread-10:null) (logid:)
> Cluster service servlet container listening on port 9090
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
> Management server 2199193518866, runId 1526887063086 is being started
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
> Management server (host id : 2) is being started at 10.38.3.18:9090
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
> Cluster manager was started successfully
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:null) (logid:)
> Starting work
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:null) (logid:)
> Starting work
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:null) (logid:)
> Starting work
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:null) (logid:)
> Starting work
> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:null) (logid:)
> Starting work
> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN No
> appenders could be found for logger
> (com.cloud.utils.db.ConnectionConcierge).
> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN Please
> initialize the log4j system properly.
> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN See
> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> May 21 09:19:55 cldmanager.srv.cesga.es server: Exception in thread
> "Thread-10" java.lang.NoClassDefFoundError:
> org/apache/log4j/spi/ThrowableInformation
> May 21 09:19:55 cldmanager.srv.cesga.es server: at
> org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
> May 21 09:19:55 cldmanager.srv.cesga.es server: at
> org.apache.log4j.Category.forcedLog(Category.java:391)
> May 21 09:19:55 cldmanager.srv.cesga.es server: at
> org.apache.log4j.Category.error(Category.java:322)
> May 21 09:19:55 cldmanager.srv.cesga.es server: at
> com.cloud.cluster.ClusterServiceServletContainer$ListenerThread.run(ClusterServiceServletContainer.java:168)
> May 21 09:19:55 cldmanager.srv.cesga.es server: Caused by:
> java.lang.ClassNotFoundException:
> org.apache.log4j.spi.ThrowableInformation
> May 21 09:19:55 cldmanager.srv.cesga.es server: at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1892)
> May 21 09:19:55 cldmanager.srv.cesga.es server: at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1735)
> May 21 09:19:55 cldmanager.srv.cesga.es server: ... 4 more
>
>
>
> El 17/05/2018 a las 7:39, ilya musayev escribió:
>> You must be running java 1.7 - I assume Suresh was alluding to this.
>> We’ve seen this error with 1.8...
>>
>> On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
>> sureshkumar.anaparti@gmail.com> wrote:
>>
>>> Hi Natalia,
>>>
>>> Any changes in the java version or system configuration?
>>>
>>> -Suresh
>>>
>>> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo
>>> <Da...@shapeblue.com>
>>> wrote:
>>>
>>>> Hi Natalia,
>>>>
>>>> My guess is you have some sort of corruption on your original host.
>>>> Since
>>>> the management service itself is stateless it’s generally much quicker
>>> just
>>>> building a new one than trying to recover the old one.
>>>>
>>>> So my suggestions would be:
>>>> - Recover your DB to a dedicated DB host if you haven’t done so
>>>> already.
>>>> - Build a new management server and just point this to the original
>>>> DB as
>>>> described in http://docs.cloudstack.apache.org/projects/cloudstack-
>>>> installation/en/4.9/management-server/index.html#
>>>> additional-management-servers
>>>> NOTE: do not use the “–deploy-as” option as this will write a new
>>>> blank
>>>> database. If you used encryption keys during the original setup you
>>>> also
>>>> need to specify these.
>>>>
>>>> Once you have these two speaking you should in theory be back up and
>>>> running again.
>>>>
>>>> Regards,
>>>> Dag Sonstebo
>>>> Cloud Architect
>>>> ShapeBlue
>>>>
>>>>
>>>> Dag.Sonstebo@shapeblue.com
>>>> www.shapeblue.com
>>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>>>> @shapeblue
>>>>
>>>>
>>>>
>>>> On 16/05/2018, 16:44, "Natalia Costas Lago" <na...@cesga.es> wrote:
>>>>
>>>>
>>>> Dear all,
>>>>
>>>> For some reason our cloudstack manager died and now we are not
>>>> able
>>> to
>>>> make it work. Our cloudstack version is 4.9.0.
>>>>
>>>> We tried to do a clean install of the manager and recover the
>>> database
>>>> from the last backup, but we were not able to recover the
>>>> service.
>>>>
>>>> At present we can see in the logs that it almost boot, but at
>>>> the end
>>>> we
>>>> get this error:
>>>>
>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>> [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null)
>>>> (logid:)
>>>> Network Manager will run the NetworkGarbageCollector every '600'
>>>> seconds.
>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>> [c.c.a.ApiServer]
>>>> (Thread-12:null) (logid:) ApiServer listening on port 8096
>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>> [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (localhost-startStop-1:null)
>>>> (logid:) Start secondary storage vm manager
>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
>>>> appenders
>>>> could be found for logger
>>>> (com.cloud.utils.db.ConnectionConcierge).
>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN Please
>>>> initialize the log4j system properly.
>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
>>>> info.
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in
>>>> thread
>>>> "CapacityChecker" java.lang.NoClassDefFoundError:
>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>> ManagedContextTimerTask.java:27)
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>> java.util.TimerThread.run(Timer.java:505)
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
>>>> java.lang.ClassNotFoundException:
>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>> WebappClassLoaderBase.java:1892)
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>> WebappClassLoaderBase.java:1735)
>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in
>>>> thread
>>>> "Timer-1" java.lang.NoClassDefFoundError:
>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>> ManagedContextTimerTask.java:27)
>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>> java.util.TimerThread.run(Timer.java:505)
>>>>
>>>> Any help is appreciated..
>>>>
>>>> Kind regards,
>>>>
>>>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>>>> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>>>>
>>>> --
>>>> ====================================================
>>>> Natalia Costas Lago
>>>> Senior Communications Technician
>>>> Galicia Supercomputing Centre (CESGA)
>>>> (CESGA on Twitter | CESGA on Facebook)
>>>>
>>>> Avenida de Vigo, s/n (Campus Vida)
>>>> 15705 Santiago de Compostela - SPAIN
>>>>
>>>> E-mail: natalia@cesga.es
>>>> Cell: +34 981 56 98 10 (ext. 237)
>>>> Fax: +34 981 59 46 16
>>>> Web: https://www.cesga.es/
>>>> ====================================================
>>>> [IMPORTANTE] La información contenida en este mensaje y
>>>> sus posibles documentos adjuntos es privada y confidencial
>>>> y está dirigida únicamente a su destinatario/a. Si usted no
>>>> es el/la destinatario/a original de este mensaje, por favor
>>>> elimínelo. La distribución o copia de este mensaje no está
>>>> autorizada.
>>>>
>>>>
>>>>
>>>>
>
> NOTA: Para cualquier incidencia/consulta/petición que requiera
> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: natalia@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.
Re: Cloudtack manager UI returns Error 404
Posted by Natalia Costas Lago <na...@cesga.es>.
It has been working with 1.8 for long...
I switched to java 1.7, but java errors persist.
We recovered backups from a couple of days prior to the disaster, but we
are still unlucky, it seems the problem is also in the backups copy.
I installed a fresh cloudstack and copied a recent database and the
manager still gives and Exception but to a later point in the execution:
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Configuring CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Done Configuring CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
(logid:) Loaded module context [ipmitool] in 171 ms
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
(logid:) Starting module [system]
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Starting CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Done Starting CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
(logid:) Starting module [core]
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Starting CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.h.m.HypervManagerImpl] (localhost-startStop-1:null) (logid:)
Cleanup mounted mount points used in previous session
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
Grabbing lock to check for database integrity.
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
Performing database integrity check
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
Starting Cluster manager, msid : 2199193518866
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterServiceServletContainer] (Thread-10:null) (logid:) Cluster
service servlet container listening on port 9090
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
Management server 2199193518866, runId 1526887063086 is being started
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
Management server (host id : 2) is being started at 10.38.3.18:9090
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Cluster
manager was started successfully
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:null) (logid:) Starting
work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:null) (logid:) Starting
work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:null) (logid:) Starting
work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:null) (logid:) Starting
work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:null) (logid:) Starting
work
May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN No appenders
could be found for logger (com.cloud.utils.db.ConnectionConcierge).
May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN Please
initialize the log4j system properly.
May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
May 21 09:19:55 cldmanager.srv.cesga.es server: Exception in thread
"Thread-10" java.lang.NoClassDefFoundError:
org/apache/log4j/spi/ThrowableInformation
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.log4j.Category.forcedLog(Category.java:391)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.log4j.Category.error(Category.java:322)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
com.cloud.cluster.ClusterServiceServletContainer$ListenerThread.run(ClusterServiceServletContainer.java:168)
May 21 09:19:55 cldmanager.srv.cesga.es server: Caused by:
java.lang.ClassNotFoundException: org.apache.log4j.spi.ThrowableInformation
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1892)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1735)
May 21 09:19:55 cldmanager.srv.cesga.es server: ... 4 more
El 17/05/2018 a las 7:39, ilya musayev escribió:
> You must be running java 1.7 - I assume Suresh was alluding to this.
> We’ve seen this error with 1.8...
>
> On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
> sureshkumar.anaparti@gmail.com> wrote:
>
>> Hi Natalia,
>>
>> Any changes in the java version or system configuration?
>>
>> -Suresh
>>
>> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo <Da...@shapeblue.com>
>> wrote:
>>
>>> Hi Natalia,
>>>
>>> My guess is you have some sort of corruption on your original host. Since
>>> the management service itself is stateless it’s generally much quicker
>> just
>>> building a new one than trying to recover the old one.
>>>
>>> So my suggestions would be:
>>> - Recover your DB to a dedicated DB host if you haven’t done so already.
>>> - Build a new management server and just point this to the original DB as
>>> described in http://docs.cloudstack.apache.org/projects/cloudstack-
>>> installation/en/4.9/management-server/index.html#
>>> additional-management-servers
>>> NOTE: do not use the “–deploy-as” option as this will write a new blank
>>> database. If you used encryption keys during the original setup you also
>>> need to specify these.
>>>
>>> Once you have these two speaking you should in theory be back up and
>>> running again.
>>>
>>> Regards,
>>> Dag Sonstebo
>>> Cloud Architect
>>> ShapeBlue
>>>
>>>
>>> Dag.Sonstebo@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>> On 16/05/2018, 16:44, "Natalia Costas Lago" <na...@cesga.es> wrote:
>>>
>>>
>>> Dear all,
>>>
>>> For some reason our cloudstack manager died and now we are not able
>> to
>>> make it work. Our cloudstack version is 4.9.0.
>>>
>>> We tried to do a clean install of the manager and recover the
>> database
>>> from the last backup, but we were not able to recover the service.
>>>
>>> At present we can see in the logs that it almost boot, but at the end
>>> we
>>> get this error:
>>>
>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null) (logid:)
>>> Network Manager will run the NetworkGarbageCollector every '600'
>>> seconds.
>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>> [c.c.a.ApiServer]
>>> (Thread-12:null) (logid:) ApiServer listening on port 8096
>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.SecondaryStorageManagerImpl] (localhost-startStop-1:null)
>>> (logid:) Start secondary storage vm manager
>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
>>> appenders
>>> could be found for logger (com.cloud.utils.db.ConnectionConcierge).
>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN Please
>>> initialize the log4j system properly.
>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in thread
>>> "CapacityChecker" java.lang.NoClassDefFoundError:
>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>> ManagedContextTimerTask.java:27)
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>> java.util.TimerThread.run(Timer.java:505)
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
>>> java.lang.ClassNotFoundException:
>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>> WebappClassLoaderBase.java:1892)
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>> WebappClassLoaderBase.java:1735)
>>> May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
>>> May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in thread
>>> "Timer-1" java.lang.NoClassDefFoundError:
>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>> ManagedContextTimerTask.java:27)
>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>> java.util.TimerThread.mainLoop(Timer.java:555)
>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>> java.util.TimerThread.run(Timer.java:505)
>>>
>>> Any help is appreciated..
>>>
>>> Kind regards,
>>>
>>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>>> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>>>
>>> --
>>> ====================================================
>>> Natalia Costas Lago
>>> Senior Communications Technician
>>> Galicia Supercomputing Centre (CESGA)
>>> (CESGA on Twitter | CESGA on Facebook)
>>>
>>> Avenida de Vigo, s/n (Campus Vida)
>>> 15705 Santiago de Compostela - SPAIN
>>>
>>> E-mail: natalia@cesga.es
>>> Cell: +34 981 56 98 10 (ext. 237)
>>> Fax: +34 981 59 46 16
>>> Web: https://www.cesga.es/
>>> ====================================================
>>> [IMPORTANTE] La información contenida en este mensaje y
>>> sus posibles documentos adjuntos es privada y confidencial
>>> y está dirigida únicamente a su destinatario/a. Si usted no
>>> es el/la destinatario/a original de este mensaje, por favor
>>> elimínelo. La distribución o copia de este mensaje no está
>>> autorizada.
>>>
>>>
>>>
>>>
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: natalia@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.
Re: Cloudtack manager UI returns Error 404
Posted by ilya musayev <il...@gmail.com>.
You must be running java 1.7 - I assume Suresh was alluding to this.
We’ve seen this error with 1.8...
On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
sureshkumar.anaparti@gmail.com> wrote:
> Hi Natalia,
>
> Any changes in the java version or system configuration?
>
> -Suresh
>
> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo <Da...@shapeblue.com>
> wrote:
>
> > Hi Natalia,
> >
> > My guess is you have some sort of corruption on your original host. Since
> > the management service itself is stateless it’s generally much quicker
> just
> > building a new one than trying to recover the old one.
> >
> > So my suggestions would be:
> > - Recover your DB to a dedicated DB host if you haven’t done so already.
> > - Build a new management server and just point this to the original DB as
> > described in http://docs.cloudstack.apache.org/projects/cloudstack-
> > installation/en/4.9/management-server/index.html#
> > additional-management-servers
> > NOTE: do not use the “–deploy-as” option as this will write a new blank
> > database. If you used encryption keys during the original setup you also
> > need to specify these.
> >
> > Once you have these two speaking you should in theory be back up and
> > running again.
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >
> >
> > Dag.Sonstebo@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> > On 16/05/2018, 16:44, "Natalia Costas Lago" <na...@cesga.es> wrote:
> >
> >
> > Dear all,
> >
> > For some reason our cloudstack manager died and now we are not able
> to
> > make it work. Our cloudstack version is 4.9.0.
> >
> > We tried to do a clean install of the manager and recover the
> database
> > from the last backup, but we were not able to recover the service.
> >
> > At present we can see in the logs that it almost boot, but at the end
> > we
> > get this error:
> >
> > May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> > [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null) (logid:)
> > Network Manager will run the NetworkGarbageCollector every '600'
> > seconds.
> > May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> > [c.c.a.ApiServer]
> > (Thread-12:null) (logid:) ApiServer listening on port 8096
> > May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> > [o.a.c.s.SecondaryStorageManagerImpl] (localhost-startStop-1:null)
> > (logid:) Start secondary storage vm manager
> > May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
> > appenders
> > could be found for logger (com.cloud.utils.db.ConnectionConcierge).
> > May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN Please
> > initialize the log4j system properly.
> > May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
> > http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> > May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in thread
> > "CapacityChecker" java.lang.NoClassDefFoundError:
> > org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
> > ManagedContextTimerTask.java:27)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.mainLoop(Timer.java:555)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.run(Timer.java:505)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
> > java.lang.ClassNotFoundException:
> > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1892)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: at
> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1735)
> > May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
> > May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in thread
> > "Timer-1" java.lang.NoClassDefFoundError:
> > org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> > May 16 17:34:24 cldmanager.srv.cesga.es server: at
> > org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
> > ManagedContextTimerTask.java:27)
> > May 16 17:34:24 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.mainLoop(Timer.java:555)
> > May 16 17:34:24 cldmanager.srv.cesga.es server: at
> > java.util.TimerThread.run(Timer.java:505)
> >
> > Any help is appreciated..
> >
> > Kind regards,
> >
> > NOTA: Para cualquier incidencia/consulta/petición que requiera
> > seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
> >
> > --
> > ====================================================
> > Natalia Costas Lago
> > Senior Communications Technician
> > Galicia Supercomputing Centre (CESGA)
> > (CESGA on Twitter | CESGA on Facebook)
> >
> > Avenida de Vigo, s/n (Campus Vida)
> > 15705 Santiago de Compostela - SPAIN
> >
> > E-mail: natalia@cesga.es
> > Cell: +34 981 56 98 10 (ext. 237)
> > Fax: +34 981 59 46 16
> > Web: https://www.cesga.es/
> > ====================================================
> > [IMPORTANTE] La información contenida en este mensaje y
> > sus posibles documentos adjuntos es privada y confidencial
> > y está dirigida únicamente a su destinatario/a. Si usted no
> > es el/la destinatario/a original de este mensaje, por favor
> > elimínelo. La distribución o copia de este mensaje no está
> > autorizada.
> >
> >
> >
> >
>
Re: Cloudtack manager UI returns Error 404
Posted by Suresh Kumar Anaparti <su...@gmail.com>.
Hi Natalia,
Any changes in the java version or system configuration?
-Suresh
On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo <Da...@shapeblue.com>
wrote:
> Hi Natalia,
>
> My guess is you have some sort of corruption on your original host. Since
> the management service itself is stateless it’s generally much quicker just
> building a new one than trying to recover the old one.
>
> So my suggestions would be:
> - Recover your DB to a dedicated DB host if you haven’t done so already.
> - Build a new management server and just point this to the original DB as
> described in http://docs.cloudstack.apache.org/projects/cloudstack-
> installation/en/4.9/management-server/index.html#
> additional-management-servers
> NOTE: do not use the “–deploy-as” option as this will write a new blank
> database. If you used encryption keys during the original setup you also
> need to specify these.
>
> Once you have these two speaking you should in theory be back up and
> running again.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
> On 16/05/2018, 16:44, "Natalia Costas Lago" <na...@cesga.es> wrote:
>
>
> Dear all,
>
> For some reason our cloudstack manager died and now we are not able to
> make it work. Our cloudstack version is 4.9.0.
>
> We tried to do a clean install of the manager and recover the database
> from the last backup, but we were not able to recover the service.
>
> At present we can see in the logs that it almost boot, but at the end
> we
> get this error:
>
> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null) (logid:)
> Network Manager will run the NetworkGarbageCollector every '600'
> seconds.
> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> [c.c.a.ApiServer]
> (Thread-12:null) (logid:) ApiServer listening on port 8096
> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
> [o.a.c.s.SecondaryStorageManagerImpl] (localhost-startStop-1:null)
> (logid:) Start secondary storage vm manager
> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
> appenders
> could be found for logger (com.cloud.utils.db.ConnectionConcierge).
> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN Please
> initialize the log4j system properly.
> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in thread
> "CapacityChecker" java.lang.NoClassDefFoundError:
> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> May 16 17:34:20 cldmanager.srv.cesga.es server: at
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
> ManagedContextTimerTask.java:27)
> May 16 17:34:20 cldmanager.srv.cesga.es server: at
> java.util.TimerThread.mainLoop(Timer.java:555)
> May 16 17:34:20 cldmanager.srv.cesga.es server: at
> java.util.TimerThread.run(Timer.java:505)
> May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
> java.lang.ClassNotFoundException:
> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
> May 16 17:34:20 cldmanager.srv.cesga.es server: at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1892)
> May 16 17:34:20 cldmanager.srv.cesga.es server: at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1735)
> May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
> May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in thread
> "Timer-1" java.lang.NoClassDefFoundError:
> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> May 16 17:34:24 cldmanager.srv.cesga.es server: at
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
> ManagedContextTimerTask.java:27)
> May 16 17:34:24 cldmanager.srv.cesga.es server: at
> java.util.TimerThread.mainLoop(Timer.java:555)
> May 16 17:34:24 cldmanager.srv.cesga.es server: at
> java.util.TimerThread.run(Timer.java:505)
>
> Any help is appreciated..
>
> Kind regards,
>
> NOTA: Para cualquier incidencia/consulta/petición que requiera
> seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
>
> --
> ====================================================
> Natalia Costas Lago
> Senior Communications Technician
> Galicia Supercomputing Centre (CESGA)
> (CESGA on Twitter | CESGA on Facebook)
>
> Avenida de Vigo, s/n (Campus Vida)
> 15705 Santiago de Compostela - SPAIN
>
> E-mail: natalia@cesga.es
> Cell: +34 981 56 98 10 (ext. 237)
> Fax: +34 981 59 46 16
> Web: https://www.cesga.es/
> ====================================================
> [IMPORTANTE] La información contenida en este mensaje y
> sus posibles documentos adjuntos es privada y confidencial
> y está dirigida únicamente a su destinatario/a. Si usted no
> es el/la destinatario/a original de este mensaje, por favor
> elimínelo. La distribución o copia de este mensaje no está
> autorizada.
>
>
>
>
Re: Cloudtack manager UI returns Error 404
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Natalia,
My guess is you have some sort of corruption on your original host. Since the management service itself is stateless it’s generally much quicker just building a new one than trying to recover the old one.
So my suggestions would be:
- Recover your DB to a dedicated DB host if you haven’t done so already.
- Build a new management server and just point this to the original DB as described in http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/management-server/index.html#additional-management-servers
NOTE: do not use the “–deploy-as” option as this will write a new blank database. If you used encryption keys during the original setup you also need to specify these.
Once you have these two speaking you should in theory be back up and running again.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
On 16/05/2018, 16:44, "Natalia Costas Lago" <na...@cesga.es> wrote:
Dear all,
For some reason our cloudstack manager died and now we are not able to
make it work. Our cloudstack version is 4.9.0.
We tried to do a clean install of the manager and recover the database
from the last backup, but we were not able to recover the service.
At present we can see in the logs that it almost boot, but at the end we
get this error:
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
[o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null) (logid:)
Network Manager will run the NetworkGarbageCollector every '600' seconds.
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO [c.c.a.ApiServer]
(Thread-12:null) (logid:) ApiServer listening on port 8096
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.SecondaryStorageManagerImpl] (localhost-startStop-1:null)
(logid:) Start secondary storage vm manager
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No appenders
could be found for logger (com.cloud.utils.db.ConnectionConcierge).
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN Please
initialize the log4j system properly.
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in thread
"CapacityChecker" java.lang.NoClassDefFoundError:
org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
java.util.TimerThread.mainLoop(Timer.java:555)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
java.util.TimerThread.run(Timer.java:505)
May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
java.lang.ClassNotFoundException:
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1892)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1735)
May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in thread
"Timer-1" java.lang.NoClassDefFoundError:
org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
May 16 17:34:24 cldmanager.srv.cesga.es server: at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
May 16 17:34:24 cldmanager.srv.cesga.es server: at
java.util.TimerThread.mainLoop(Timer.java:555)
May 16 17:34:24 cldmanager.srv.cesga.es server: at
java.util.TimerThread.run(Timer.java:505)
Any help is appreciated..
Kind regards,
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicaciones@cesga.es
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: natalia@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.