You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Subroto <ss...@datameer.com> on 2013/05/17 16:39:56 UTC
Memory Leak while using LocalJobRunner
Hi,
I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
Cheers,
Subroto Sanyal
Re: Memory Leak while using LocalJobRunner
Posted by Subroto <ss...@datameer.com>.
Hi Vinod,
Thanks Vinod…..
I have raised the issue.
https://issues.apache.org/jira/browse/MAPREDUCE-5258
Cheers,
Subroto Sanyal
On May 17, 2013, at 7:33 PM, Vinod Kumar Vavilapalli wrote:
>
> It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On May 17, 2013, at 7:39 AM, Subroto wrote:
>
>> Hi,
>>
>> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
>> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>>
>> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>>
>> Cheers,
>> Subroto Sanyal
>
Re: Memory Leak while using LocalJobRunner
Posted by Subroto <ss...@datameer.com>.
Hi Vinod,
Thanks Vinod…..
I have raised the issue.
https://issues.apache.org/jira/browse/MAPREDUCE-5258
Cheers,
Subroto Sanyal
On May 17, 2013, at 7:33 PM, Vinod Kumar Vavilapalli wrote:
>
> It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On May 17, 2013, at 7:39 AM, Subroto wrote:
>
>> Hi,
>>
>> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
>> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>>
>> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>>
>> Cheers,
>> Subroto Sanyal
>
Re: Memory Leak while using LocalJobRunner
Posted by Subroto <ss...@datameer.com>.
Hi Vinod,
Thanks Vinod…..
I have raised the issue.
https://issues.apache.org/jira/browse/MAPREDUCE-5258
Cheers,
Subroto Sanyal
On May 17, 2013, at 7:33 PM, Vinod Kumar Vavilapalli wrote:
>
> It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On May 17, 2013, at 7:39 AM, Subroto wrote:
>
>> Hi,
>>
>> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
>> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>>
>> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>>
>> Cheers,
>> Subroto Sanyal
>
Re: Memory Leak while using LocalJobRunner
Posted by Subroto <ss...@datameer.com>.
Hi Vinod,
Thanks Vinod…..
I have raised the issue.
https://issues.apache.org/jira/browse/MAPREDUCE-5258
Cheers,
Subroto Sanyal
On May 17, 2013, at 7:33 PM, Vinod Kumar Vavilapalli wrote:
>
> It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On May 17, 2013, at 7:39 AM, Subroto wrote:
>
>> Hi,
>>
>> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
>> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>>
>> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>>
>> Cheers,
>> Subroto Sanyal
>
Re: Memory Leak while using LocalJobRunner
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On May 17, 2013, at 7:39 AM, Subroto wrote:
> Hi,
>
> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>
> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>
> Cheers,
> Subroto Sanyal
Re: Memory Leak while using LocalJobRunner
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On May 17, 2013, at 7:39 AM, Subroto wrote:
> Hi,
>
> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>
> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>
> Cheers,
> Subroto Sanyal
Re: Memory Leak while using LocalJobRunner
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On May 17, 2013, at 7:39 AM, Subroto wrote:
> Hi,
>
> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>
> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>
> Cheers,
> Subroto Sanyal
Re: Memory Leak while using LocalJobRunner
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It's inconvenient, but you can try DefaultMetricsSystem.shutDown(). Please see if that works and file a ticket to make LocalJobRunner automatically do this on finish.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On May 17, 2013, at 7:39 AM, Subroto wrote:
> Hi,
>
> I have an application which uses LocalJobRunner to process MR Jobs. From the heap dump of my application I can see multiple object with Name $Proxy90; happens to be created by org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(Callback).
> Every-time a LocalJobRunner is launched it creates JobTrackerInstrumentation and QueueMetrics.
>
> While creating this MetricsSystem ; it registers and adds a Callback to ArrayList which keeps on growing as the DefaultMetricsSystem is Singleton. Is it a known bug? Is there any workaround to empty this list?
>
> Cheers,
> Subroto Sanyal