You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "shanyu zhao (JIRA)" <ji...@apache.org> on 2014/07/15 23:21:04 UTC
[jira] [Updated] (HADOOP-10840) Fix OutOfMemoryError caused by
metrics system in Azure File System
[ https://issues.apache.org/jira/browse/HADOOP-10840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
shanyu zhao updated HADOOP-10840:
---------------------------------
Attachment: HADOOP-10840.patch
Patch attached.
This patch added a finalize() method to NativeAzureFileSystem to force a call to close() which calls unregisterSoruce() on the metrics system impl. Note that close() can be called multiple times without negative impact so that in the future when we introduce close() to FileContext to let client manually call close() there shall be no changes in this area. I also removed registration code to DefaultMetricsSystem because we only need to register with one metrics system implementation which is AzureFileSystemMetricsSystem.
> Fix OutOfMemoryError caused by metrics system in Azure File System
> ------------------------------------------------------------------
>
> Key: HADOOP-10840
> URL: https://issues.apache.org/jira/browse/HADOOP-10840
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 2.4.1
> Reporter: shanyu zhao
> Assignee: shanyu zhao
> Attachments: HADOOP-10840.patch
>
>
> In Hadoop 2.x the Hadoop File System framework changed and no cache is implemented (refer to HADOOP-6356). This means for every WASB access, a new NativeAzureFileSystem is created, along which a Metrics source created and added to MetricsSystemImpl. Over time the sources accumulated, eating memory and causing Java OutOfMemoryError.
> The fix is to utilize the unregisterSource() method added to MetricsSystem in HADOOP-10839.
--
This message was sent by Atlassian JIRA
(v6.2#6252)