You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@cloudstack.apache.org by "Stock, Alexander" <Al...@bitgroup.de> on 2017/03/02 09:12:01 UTC

Usage Server 4.9 Issue

Hi,

a week ago we discovered an issue with the usage server version (4.9).
After around two month from upgrading to 4.9.0 usage server stopped working and produced following error messages in /var/log/cloudstack/usage/usage.log:

ERROR [usage.dao.UsageDaoImpl] (Usage-Job-1:null) (logid:) error saving account to cloud_usage db
java.lang.NullPointerException
        at com.cloud.usage.dao.UsageDaoImpl.saveAccounts(UsageDaoImpl.java:116)
        at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.CGLIB$saveAccounts$15(<generated>)
        at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7_FastClassByCloudStack_80eceb8.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
        at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.saveAccounts(<generated>)
        at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:516)
        at com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:384)
        at com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:326)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:323)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Usage Manager error
com.cloud.utils.exception.CloudRuntimeException
        at com.cloud.usage.dao.UsageDaoImpl.saveAccounts(UsageDaoImpl.java:135)
        at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.CGLIB$saveAccounts$15(<generated>)
        at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7_FastClassByCloudStack_80eceb8.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
        at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.saveAccounts(<generated>)
        at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:516)
        at com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:384)
        at com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:326)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:323)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

This happened on our testing and prod environment on Ubuntu.
We also tried to fix it with the migrate-dynamicroles.py script, but that didn't fixed it.

The problem was gone after we did an Upgrade to 4.9.2.
So if someone has this problem too then upgrade to 4.9.2.

BR
Alexander

Re: Usage Server 4.9 Issue

Posted by Rene Moser <ma...@renemoser.net>.

Thanks for sharing!

Regards
Ren

On 03/02/2017 10:12 AM, Stock, Alexander wrote:
> Hi,
> 
> a week ago we discovered an issue with the usage server version (4.9).
> After around two month from upgrading to 4.9.0 usage server stopped working and produced following error messages in /var/log/cloudstack/usage/usage.log:
> 
> ERROR [usage.dao.UsageDaoImpl] (Usage-Job-1:null) (logid:) error saving account to cloud_usage db
> java.lang.NullPointerException
>         at com.cloud.usage.dao.UsageDaoImpl.saveAccounts(UsageDaoImpl.java:116)
>         at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.CGLIB$saveAccounts$15(<generated>)
>         at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7_FastClassByCloudStack_80eceb8.invoke(<generated>)
>         at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
>         at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.saveAccounts(<generated>)
>         at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:516)
>         at com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:384)
>         at com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:326)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:323)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> ERROR [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Usage Manager error
> com.cloud.utils.exception.CloudRuntimeException
>         at com.cloud.usage.dao.UsageDaoImpl.saveAccounts(UsageDaoImpl.java:135)
>         at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.CGLIB$saveAccounts$15(<generated>)
>         at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7_FastClassByCloudStack_80eceb8.invoke(<generated>)
>         at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
>         at com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_330c46f7.saveAccounts(<generated>)
>         at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:516)
>         at com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:384)
>         at com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:326)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:323)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 
> This happened on our testing and prod environment on Ubuntu.
> We also tried to fix it with the migrate-dynamicroles.py script, but that didn't fixed it.
> 
> The problem was gone after we did an Upgrade to 4.9.2.
> So if someone has this problem too then upgrade to 4.9.2.
> 
> BR
> Alexander
>