You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Simon Weller <sw...@ena.com> on 2013/07/22 17:30:13 UTC
[ACS411] CLOUDSTACK-2188: patch request
Hi,
I'd like to request a patch for Cloudstack-2188 for ACS 4.1.1.
The master commit was e56d2a401c40b4208d062c0a0ce1ec01df73dd08, but it appears the code has been greatly re-factored since 4.1 was originally branched.
This NPE appears to be causing a memory leak in our production environment that consumes memory quickly due to the number of times this NPE is being triggered. We currently have tomcat max memory set to 2G, and individual management servers are running out of memory within 4 to 5 days.
MS NPE log for reference:
2013-07-22 09:40:47,404 DEBUG [agent.manager.AgentManagerImpl] (AgentConnectTaskPool-1066:null) Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2013-07-22 09:40:47,407 ERROR [agent.manager.AgentManagerImpl] (AgentConnectTaskPool-1066:null) Monitor DownloadListener says there is an error in the connect process for 28 due to null
java.lang.NullPointerException
at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:694)
at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:620)
at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1111)
at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:145)
at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.run(AgentManagerImpl.java:1186)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Thanks,
- Si
Re: [ACS411] CLOUDSTACK-2188: patch request
Posted by Simon Weller <sw...@ena.com>.
Issue filed as CLOUDSTACK-3716 .
----- Original Message -----
From: "Simon Weller" <sw...@ena.com>
To: dev@cloudstack.apache.org
Sent: Monday, July 22, 2013 11:52:08 AM
Subject: Re: [ACS411] CLOUDSTACK-2188: patch request
Ok, will do.
The reason I hadn't brought this up until now is that we only upgraded to 4.1 in production last week. That's when I started digging into this NPE and found the reported related issues.
----- Original Message -----
From: "Ilya Musayev" <im...@webmd.net>
To: dev@cloudstack.apache.org
Sent: Monday, July 22, 2013 11:32:50 AM
Subject: RE: [ACS411] CLOUDSTACK-2188: patch request
Simon,
Since this issue has been resolved, please open a separate issue for ACS 4.1 and link CLOUDSTACK-2177 and CLOUDSTACK-2188 to it.
Please mention your environment and steps to reproduce along with error trace.
Thanks
ilya
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, July 22, 2013 12:24 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [ACS411] CLOUDSTACK-2188: patch request
>
> Simon,
>
> I've looked at patch and yes - it's been refactored greatly.
>
> I wish this would have been brought up earlier :( While due to time
> constraint, we probably wont be able to make into 4.1.1, if we can get the
> patch, we will try to push it into next 4.1.2 update.
>
> Please update the ticket and ask if this patch can be back ported to 4.1. I'll do
> the same.
>
> Thanks
> ilya
>
> > -----Original Message-----
> > From: Simon Weller [mailto:sweller@ena.com]
> > Sent: Monday, July 22, 2013 11:30 AM
> > To: dev@cloudstack.apache.org
> > Subject: [ACS411] CLOUDSTACK-2188: patch request
> >
> > Hi,
> >
> > I'd like to request a patch for Cloudstack-2188 for ACS 4.1.1.
> >
> > The master commit was e56d2a401c40b4208d062c0a0ce1ec01df73dd08, but
> it
> > appears the code has been greatly re-factored since 4.1 was originally
> > branched.
> > This NPE appears to be causing a memory leak in our production
> > environment that consumes memory quickly due to the number of times
> > this NPE is being triggered. We currently have tomcat max memory set
> > to 2G, and individual management servers are running out of memory
> > within 4 to 5 days.
> >
> > MS NPE log for reference:
> >
> > 2013-07-22 09:40:47,404 DEBUG [agent.manager.AgentManagerImpl]
> > (AgentConnectTaskPool-1066:null) Details from executing class
> > com.cloud.agent.api.storage.ListVolumeCommand: success
> > 2013-07-22 09:40:47,407 ERROR [agent.manager.AgentManagerImpl]
> > (AgentConnectTaskPool-1066:null) Monitor DownloadListener says there
> > is an error in the connect process for 28 due to null
> > java.lang.NullPointerException at
> >
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(Do
> > wnloadMonitorImpl.java:694)
> > at
> >
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(Download
> > MonitorImpl.java:620)
> > at
> >
> com.cloud.storage.download.DownloadListener.processConnect(DownloadL
> > istener.java:385)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(
> > AgentManagerImpl.java:611)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(Agen
> > tManagerImpl.java:1111)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerIm
> > pl.java:145)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.ru
> > n(AgentManagerImpl.java:1186)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> > av
> > a:1146)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > ja
> > va:615)
> > at java.lang.Thread.run(Thread.java:679)
> >
> >
> > Thanks,
> >
> > - Si
Re: [ACS411] CLOUDSTACK-2188: patch request
Posted by Simon Weller <sw...@ena.com>.
Ok, will do.
The reason I hadn't brought this up until now is that we only upgraded to 4.1 in production last week. That's when I started digging into this NPE and found the reported related issues.
----- Original Message -----
From: "Ilya Musayev" <im...@webmd.net>
To: dev@cloudstack.apache.org
Sent: Monday, July 22, 2013 11:32:50 AM
Subject: RE: [ACS411] CLOUDSTACK-2188: patch request
Simon,
Since this issue has been resolved, please open a separate issue for ACS 4.1 and link CLOUDSTACK-2177 and CLOUDSTACK-2188 to it.
Please mention your environment and steps to reproduce along with error trace.
Thanks
ilya
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, July 22, 2013 12:24 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [ACS411] CLOUDSTACK-2188: patch request
>
> Simon,
>
> I've looked at patch and yes - it's been refactored greatly.
>
> I wish this would have been brought up earlier :( While due to time
> constraint, we probably wont be able to make into 4.1.1, if we can get the
> patch, we will try to push it into next 4.1.2 update.
>
> Please update the ticket and ask if this patch can be back ported to 4.1. I'll do
> the same.
>
> Thanks
> ilya
>
> > -----Original Message-----
> > From: Simon Weller [mailto:sweller@ena.com]
> > Sent: Monday, July 22, 2013 11:30 AM
> > To: dev@cloudstack.apache.org
> > Subject: [ACS411] CLOUDSTACK-2188: patch request
> >
> > Hi,
> >
> > I'd like to request a patch for Cloudstack-2188 for ACS 4.1.1.
> >
> > The master commit was e56d2a401c40b4208d062c0a0ce1ec01df73dd08, but
> it
> > appears the code has been greatly re-factored since 4.1 was originally
> > branched.
> > This NPE appears to be causing a memory leak in our production
> > environment that consumes memory quickly due to the number of times
> > this NPE is being triggered. We currently have tomcat max memory set
> > to 2G, and individual management servers are running out of memory
> > within 4 to 5 days.
> >
> > MS NPE log for reference:
> >
> > 2013-07-22 09:40:47,404 DEBUG [agent.manager.AgentManagerImpl]
> > (AgentConnectTaskPool-1066:null) Details from executing class
> > com.cloud.agent.api.storage.ListVolumeCommand: success
> > 2013-07-22 09:40:47,407 ERROR [agent.manager.AgentManagerImpl]
> > (AgentConnectTaskPool-1066:null) Monitor DownloadListener says there
> > is an error in the connect process for 28 due to null
> > java.lang.NullPointerException at
> >
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(Do
> > wnloadMonitorImpl.java:694)
> > at
> >
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(Download
> > MonitorImpl.java:620)
> > at
> >
> com.cloud.storage.download.DownloadListener.processConnect(DownloadL
> > istener.java:385)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(
> > AgentManagerImpl.java:611)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(Agen
> > tManagerImpl.java:1111)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerIm
> > pl.java:145)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.ru
> > n(AgentManagerImpl.java:1186)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> > av
> > a:1146)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > ja
> > va:615)
> > at java.lang.Thread.run(Thread.java:679)
> >
> >
> > Thanks,
> >
> > - Si
RE: [ACS411] CLOUDSTACK-2188: patch request
Posted by "Musayev, Ilya" <im...@webmd.net>.
Simon,
Since this issue has been resolved, please open a separate issue for ACS 4.1 and link CLOUDSTACK-2177 and CLOUDSTACK-2188 to it.
Please mention your environment and steps to reproduce along with error trace.
Thanks
ilya
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, July 22, 2013 12:24 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [ACS411] CLOUDSTACK-2188: patch request
>
> Simon,
>
> I've looked at patch and yes - it's been refactored greatly.
>
> I wish this would have been brought up earlier :( While due to time
> constraint, we probably wont be able to make into 4.1.1, if we can get the
> patch, we will try to push it into next 4.1.2 update.
>
> Please update the ticket and ask if this patch can be back ported to 4.1. I'll do
> the same.
>
> Thanks
> ilya
>
> > -----Original Message-----
> > From: Simon Weller [mailto:sweller@ena.com]
> > Sent: Monday, July 22, 2013 11:30 AM
> > To: dev@cloudstack.apache.org
> > Subject: [ACS411] CLOUDSTACK-2188: patch request
> >
> > Hi,
> >
> > I'd like to request a patch for Cloudstack-2188 for ACS 4.1.1.
> >
> > The master commit was e56d2a401c40b4208d062c0a0ce1ec01df73dd08, but
> it
> > appears the code has been greatly re-factored since 4.1 was originally
> > branched.
> > This NPE appears to be causing a memory leak in our production
> > environment that consumes memory quickly due to the number of times
> > this NPE is being triggered. We currently have tomcat max memory set
> > to 2G, and individual management servers are running out of memory
> > within 4 to 5 days.
> >
> > MS NPE log for reference:
> >
> > 2013-07-22 09:40:47,404 DEBUG [agent.manager.AgentManagerImpl]
> > (AgentConnectTaskPool-1066:null) Details from executing class
> > com.cloud.agent.api.storage.ListVolumeCommand: success
> > 2013-07-22 09:40:47,407 ERROR [agent.manager.AgentManagerImpl]
> > (AgentConnectTaskPool-1066:null) Monitor DownloadListener says there
> > is an error in the connect process for 28 due to null
> > java.lang.NullPointerException at
> >
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(Do
> > wnloadMonitorImpl.java:694)
> > at
> >
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(Download
> > MonitorImpl.java:620)
> > at
> >
> com.cloud.storage.download.DownloadListener.processConnect(DownloadL
> > istener.java:385)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(
> > AgentManagerImpl.java:611)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(Agen
> > tManagerImpl.java:1111)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerIm
> > pl.java:145)
> > at
> >
> com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.ru
> > n(AgentManagerImpl.java:1186)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> > av
> > a:1146)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > ja
> > va:615)
> > at java.lang.Thread.run(Thread.java:679)
> >
> >
> > Thanks,
> >
> > - Si
RE: [ACS411] CLOUDSTACK-2188: patch request
Posted by "Musayev, Ilya" <im...@webmd.net>.
Simon,
I've looked at patch and yes - it's been refactored greatly.
I wish this would have been brought up earlier :( While due to time constraint, we probably wont be able to make into 4.1.1, if we can get the patch, we will try to push it into next 4.1.2 update.
Please update the ticket and ask if this patch can be back ported to 4.1. I'll do the same.
Thanks
ilya
> -----Original Message-----
> From: Simon Weller [mailto:sweller@ena.com]
> Sent: Monday, July 22, 2013 11:30 AM
> To: dev@cloudstack.apache.org
> Subject: [ACS411] CLOUDSTACK-2188: patch request
>
> Hi,
>
> I'd like to request a patch for Cloudstack-2188 for ACS 4.1.1.
>
> The master commit was e56d2a401c40b4208d062c0a0ce1ec01df73dd08, but it
> appears the code has been greatly re-factored since 4.1 was originally
> branched.
> This NPE appears to be causing a memory leak in our production
> environment that consumes memory quickly due to the number of times this
> NPE is being triggered. We currently have tomcat max memory set to 2G, and
> individual management servers are running out of memory within 4 to 5
> days.
>
> MS NPE log for reference:
>
> 2013-07-22 09:40:47,404 DEBUG [agent.manager.AgentManagerImpl]
> (AgentConnectTaskPool-1066:null) Details from executing class
> com.cloud.agent.api.storage.ListVolumeCommand: success
> 2013-07-22 09:40:47,407 ERROR [agent.manager.AgentManagerImpl]
> (AgentConnectTaskPool-1066:null) Monitor DownloadListener says there is
> an error in the connect process for 28 due to null
> java.lang.NullPointerException at
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(Do
> wnloadMonitorImpl.java:694)
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(Download
> MonitorImpl.java:620)
> at
> com.cloud.storage.download.DownloadListener.processConnect(DownloadL
> istener.java:385)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(
> AgentManagerImpl.java:611)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(Agen
> tManagerImpl.java:1111)
> at
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerIm
> pl.java:145)
> at
> com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.ru
> n(AgentManagerImpl.java:1186)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:615)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Thanks,
>
> - Si