You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Sonali Jadhav <so...@servercentralen.se> on 2015/07/06 07:18:47 UTC
error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Hi,
After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
java.lang.NumberFormatException: For input string: "iptables"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.<init>(Long.java:702)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
Message: For input string: "iptables"
Stack: java.lang.NumberFormatException: For input string: "iptables"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.<init>(Long.java:702)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
what exactly is this error?
/sonali
Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Sonali,
Can you please check the MS logs for NetworkUsageCommand and answer for the errors on VR start after upgrade.
Thanks,
Jayapal
On 06-Jul-2015, at 1:44 PM, Sonali Jadhav <so...@servercentralen.se>
wrote:
> Hi,
>
> I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully executed cmd " nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &" to update VRs.
>
> I also cross checked from ACS, version of each VR. It was 4.5.
>
> But I was seeing these errors in management logs from few VRs. so like you said I observed in those VR, NETWORK_STATS chain was missing. I just rebooted them and then I was able to see that chain in iptables. So I rebooted all those VRs with error logs. And now it seems to be ok.
>
> /Sonali
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, July 6, 2015 12:45 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>
> Hi Sonali,
>
> Can you give me some context on how this issue is seen. I want to understand when these chains got removed in VR.
>
> Thanks,
> Jayapal
>
> On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>
>> Sorry, error log was not for all VRs but some.
>>
>> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs in ACS.
>> I'll keep watch if error comes again.
>>
>> Thanks for help.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Monday, July 6, 2015 11:00 AM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>>
>>
>> It seems VR does not have the usage related iptables chains.
>> Please check the 'iptables -L -nv' for NETWORK_STATS chains.
>> These rules supposed to be configured on VR start.
>>
>> Thanks,
>> Jayapal
>>
>>
>>
>> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
>> wrote:
>>
>> Hi,
>>
>> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
>>
>>
>> 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
>> 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
>> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
>> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
>> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
>> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
>> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
>> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
>> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
>> java.lang.NumberFormatException: For input string: "iptables"
>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:441)
>> at java.lang.Long.<init>(Long.java:702)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
>> Message: For input string: "iptables"
>> Stack: java.lang.NumberFormatException: For input string: "iptables"
>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:441)
>> at java.lang.Long.<init>(Long.java:702)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> what exactly is this error?
>>
>> /sonali
>>
>
Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Sonali,
Can you please tell me what kind of configuration is there in the VR.
How many public interfaces, number of firewall rules etc approximately.
In our lab we want to try with the similar VR config.
THanks,
Jayapal
On 20-Jul-2015, at 3:13 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Jaypal,
>
> I did not reboot VR outside of cloudstack, since I was in middle of upgrade. I gave cloudstack-sysvmadm cmd, this is the log,
>
> [root@CSMgmt01 ~]# cat sysvm.log
> nohup: ignoring input
> Stopping and starting 1 secondary storage vm(s)...
> Done stopping and starting secondary storage vm(s)
> Stopping and starting 1 console proxy vm(s)...
> Done stopping and starting console proxy vm(s) .
> Stopping and starting 11 running routing vm(s)...
>
> After that I cross checked all System VMs and VRs, all showing version 4.5 in CloudStack. But like I said before, after reboot of VRs it was gone.
>
>
> /Sonali
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, July 20, 2015 3:00 PM
> To: <us...@cloudstack.apache.org>; Sonali Jadhav
> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>
> Hi Sonali,
>
> In our lab setup this issue is not reproduced.
> I am trying to understand how NETWORK_STATS is missed in VR. This can be missed if the VR is rebooted outside the cloudstack. If this is not the case some thing might removed the rule.
> Can you please recollect to understand how this chain is removed.
>
> Thanks,
> Jayapal
>
>
>
>
> On 06-Jul-2015, at 1:44 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>
>> Hi,
>>
>> I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully executed cmd " nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &" to update VRs.
>>
>> I also cross checked from ACS, version of each VR. It was 4.5.
>>
>> But I was seeing these errors in management logs from few VRs. so like you said I observed in those VR, NETWORK_STATS chain was missing. I just rebooted them and then I was able to see that chain in iptables. So I rebooted all those VRs with error logs. And now it seems to be ok.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Monday, July 6, 2015 12:45 PM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>>
>> Hi Sonali,
>>
>> Can you give me some context on how this issue is seen. I want to understand when these chains got removed in VR.
>>
>> Thanks,
>> Jayapal
>>
>> On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>
>>> Sorry, error log was not for all VRs but some.
>>>
>>> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs in ACS.
>>> I'll keep watch if error comes again.
>>>
>>> Thanks for help.
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Monday, July 6, 2015 11:00 AM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>>>
>>>
>>> It seems VR does not have the usage related iptables chains.
>>> Please check the 'iptables -L -nv' for NETWORK_STATS chains.
>>> These rules supposed to be configured on VR start.
>>>
>>> Thanks,
>>> Jayapal
>>>
>>>
>>>
>>> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
>>> wrote:
>>>
>>> Hi,
>>>
>>> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
>>>
>>>
>>> 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
>>> 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
>>> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
>>> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
>>> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
>>> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
>>> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
>>> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
>>> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
>>> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
>>> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
>>> 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
>>> java.lang.NumberFormatException: For input string: "iptables"
>>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>>> at java.lang.Long.parseLong(Long.java:441)
>>> at java.lang.Long.<init>(Long.java:702)
>>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
>>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
>>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
>>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
>>> Message: For input string: "iptables"
>>> Stack: java.lang.NumberFormatException: For input string: "iptables"
>>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>>> at java.lang.Long.parseLong(Long.java:441)
>>> at java.lang.Long.<init>(Long.java:702)
>>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>> what exactly is this error?
>>>
>>> /sonali
>>>
>>
>
RE: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Sonali Jadhav <so...@servercentralen.se>.
Hi Jaypal,
I did not reboot VR outside of cloudstack, since I was in middle of upgrade. I gave cloudstack-sysvmadm cmd, this is the log,
[root@CSMgmt01 ~]# cat sysvm.log
nohup: ignoring input
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 11 running routing vm(s)...
After that I cross checked all System VMs and VRs, all showing version 4.5 in CloudStack. But like I said before, after reboot of VRs it was gone.
/Sonali
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, July 20, 2015 3:00 PM
To: <us...@cloudstack.apache.org>; Sonali Jadhav
Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Hi Sonali,
In our lab setup this issue is not reproduced.
I am trying to understand how NETWORK_STATS is missed in VR. This can be missed if the VR is rebooted outside the cloudstack. If this is not the case some thing might removed the rule.
Can you please recollect to understand how this chain is removed.
Thanks,
Jayapal
On 06-Jul-2015, at 1:44 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi,
>
> I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully executed cmd " nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &" to update VRs.
>
> I also cross checked from ACS, version of each VR. It was 4.5.
>
> But I was seeing these errors in management logs from few VRs. so like you said I observed in those VR, NETWORK_STATS chain was missing. I just rebooted them and then I was able to see that chain in iptables. So I rebooted all those VRs with error logs. And now it seems to be ok.
>
> /Sonali
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, July 6, 2015 12:45 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>
> Hi Sonali,
>
> Can you give me some context on how this issue is seen. I want to understand when these chains got removed in VR.
>
> Thanks,
> Jayapal
>
> On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>
>> Sorry, error log was not for all VRs but some.
>>
>> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs in ACS.
>> I'll keep watch if error comes again.
>>
>> Thanks for help.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Monday, July 6, 2015 11:00 AM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>>
>>
>> It seems VR does not have the usage related iptables chains.
>> Please check the 'iptables -L -nv' for NETWORK_STATS chains.
>> These rules supposed to be configured on VR start.
>>
>> Thanks,
>> Jayapal
>>
>>
>>
>> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
>> wrote:
>>
>> Hi,
>>
>> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
>>
>>
>> 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
>> 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
>> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
>> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
>> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
>> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
>> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
>> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
>> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
>> java.lang.NumberFormatException: For input string: "iptables"
>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:441)
>> at java.lang.Long.<init>(Long.java:702)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
>> Message: For input string: "iptables"
>> Stack: java.lang.NumberFormatException: For input string: "iptables"
>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:441)
>> at java.lang.Long.<init>(Long.java:702)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> what exactly is this error?
>>
>> /sonali
>>
>
Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Sonali,
In our lab setup this issue is not reproduced.
I am trying to understand how NETWORK_STATS is missed in VR. This can be missed if the VR is rebooted outside the cloudstack. If this is not the case some thing might removed the rule.
Can you please recollect to understand how this chain is removed.
Thanks,
Jayapal
On 06-Jul-2015, at 1:44 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi,
>
> I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully executed cmd " nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &" to update VRs.
>
> I also cross checked from ACS, version of each VR. It was 4.5.
>
> But I was seeing these errors in management logs from few VRs. so like you said I observed in those VR, NETWORK_STATS chain was missing. I just rebooted them and then I was able to see that chain in iptables. So I rebooted all those VRs with error logs. And now it seems to be ok.
>
> /Sonali
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, July 6, 2015 12:45 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>
> Hi Sonali,
>
> Can you give me some context on how this issue is seen. I want to understand when these chains got removed in VR.
>
> Thanks,
> Jayapal
>
> On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>
>> Sorry, error log was not for all VRs but some.
>>
>> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs in ACS.
>> I'll keep watch if error comes again.
>>
>> Thanks for help.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Monday, July 6, 2015 11:00 AM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>>
>>
>> It seems VR does not have the usage related iptables chains.
>> Please check the 'iptables -L -nv' for NETWORK_STATS chains.
>> These rules supposed to be configured on VR start.
>>
>> Thanks,
>> Jayapal
>>
>>
>>
>> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
>> wrote:
>>
>> Hi,
>>
>> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
>>
>>
>> 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
>> 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
>> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
>> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
>> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
>> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
>> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
>> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
>> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
>> java.lang.NumberFormatException: For input string: "iptables"
>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:441)
>> at java.lang.Long.<init>(Long.java:702)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
>> Message: For input string: "iptables"
>> Stack: java.lang.NumberFormatException: For input string: "iptables"
>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:441)
>> at java.lang.Long.<init>(Long.java:702)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> what exactly is this error?
>>
>> /sonali
>>
>
RE: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Sonali Jadhav <so...@servercentralen.se>.
Hi,
I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully executed cmd " nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &" to update VRs.
I also cross checked from ACS, version of each VR. It was 4.5.
But I was seeing these errors in management logs from few VRs. so like you said I observed in those VR, NETWORK_STATS chain was missing. I just rebooted them and then I was able to see that chain in iptables. So I rebooted all those VRs with error logs. And now it seems to be ok.
/Sonali
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, July 6, 2015 12:45 PM
To: <us...@cloudstack.apache.org>
Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Hi Sonali,
Can you give me some context on how this issue is seen. I want to understand when these chains got removed in VR.
Thanks,
Jayapal
On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Sorry, error log was not for all VRs but some.
>
> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs in ACS.
> I'll keep watch if error comes again.
>
> Thanks for help.
>
> /Sonali
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, July 6, 2015 11:00 AM
> To: <us...@cloudstack.apache.org>
> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>
>
> It seems VR does not have the usage related iptables chains.
> Please check the 'iptables -L -nv' for NETWORK_STATS chains.
> These rules supposed to be configured on VR start.
>
> Thanks,
> Jayapal
>
>
>
> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
> wrote:
>
> Hi,
>
> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
>
>
> 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
> 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
> 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
> java.lang.NumberFormatException: For input string: "iptables"
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:441)
> at java.lang.Long.<init>(Long.java:702)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
> Message: For input string: "iptables"
> Stack: java.lang.NumberFormatException: For input string: "iptables"
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:441)
> at java.lang.Long.<init>(Long.java:702)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> what exactly is this error?
>
> /sonali
>
Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Sonali,
Can you give me some context on how this issue is seen. I want to understand when these chains got removed in VR.
Thanks,
Jayapal
On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Sorry, error log was not for all VRs but some.
>
> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs in ACS.
> I'll keep watch if error comes again.
>
> Thanks for help.
>
> /Sonali
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, July 6, 2015 11:00 AM
> To: <us...@cloudstack.apache.org>
> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>
>
> It seems VR does not have the usage related iptables chains.
> Please check the 'iptables -L -nv' for NETWORK_STATS chains.
> These rules supposed to be configured on VR start.
>
> Thanks,
> Jayapal
>
>
>
> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
> wrote:
>
> Hi,
>
> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
>
>
> 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
> 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
> 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
> java.lang.NumberFormatException: For input string: "iptables"
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:441)
> at java.lang.Long.<init>(Long.java:702)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
> Message: For input string: "iptables"
> Stack: java.lang.NumberFormatException: For input string: "iptables"
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:441)
> at java.lang.Long.<init>(Long.java:702)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> what exactly is this error?
>
> /sonali
>
RE: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Sonali Jadhav <so...@servercentralen.se>.
Sorry, error log was not for all VRs but some.
I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs in ACS.
I'll keep watch if error comes again.
Thanks for help.
/Sonali
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, July 6, 2015 11:00 AM
To: <us...@cloudstack.apache.org>
Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
It seems VR does not have the usage related iptables chains.
Please check the 'iptables -L -nv' for NETWORK_STATS chains.
These rules supposed to be configured on VR start.
Thanks,
Jayapal
On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
wrote:
Hi,
After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
java.lang.NumberFormatException: For input string: "iptables"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.<init>(Long.java:702)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
Message: For input string: "iptables"
Stack: java.lang.NumberFormatException: For input string: "iptables"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.<init>(Long.java:702)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
what exactly is this error?
/sonali
Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
It seems VR does not have the usage related iptables chains.
Please check the 'iptables -L -nv' for NETWORK_STATS chains.
These rules supposed to be configured on VR start.
Thanks,
Jayapal
On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <so...@servercentralen.se>>
wrote:
Hi,
After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management server logs, for each and every VR,
2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs
2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs
2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is running...
2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers.
2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status.
2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status.
2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for external networking, so skipping usage check.
2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request
2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018) Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to
java.lang.NumberFormatException: For input string: "iptables"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.<init>(Long.java:702)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received:
2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage: For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string: \"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d) Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
Message: For input string: "iptables"
Stack: java.lang.NumberFormatException: For input string: "iptables"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.<init>(Long.java:702)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
what exactly is this error?
/sonali