You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Dick King (JIRA)" <ji...@apache.org> on 2010/07/03 01:48:50 UTC
[jira] Created: (MAPREDUCE-1914) TrackerDistributedCacheManager
never cleans its input directories
TrackerDistributedCacheManager never cleans its input directories
-----------------------------------------------------------------
Key: MAPREDUCE-1914
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Dick King
Assignee: Dick King
When we localize a file into a node's cache, it's installed in a directory whose subroot is a random {{long}} . These {{long}} s all sit in a single flat directory [per disk, per cluster node]. When the cached file is no longer needed, its reference count becomes zero in a tracking data structure. The file then becomes eligible for deletion when the total amount of space occupied by cached files exceeds 10G [by default] or the total number of such files exceeds 10K.
However, when we delete a cached file, we don't delete the directory that contains it; this importantly includes the elements of the flat directory, which then accumulate until they reach a system limit, 32K in some cases, and then the node stops working.
We need to delete the flat directory when we delete the localized cache file it contains.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1914) TrackerDistributedCacheManager
never cleans its input directories
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1914:
---------------------------------
Status: Patch Available (was: Open)
> TrackerDistributedCacheManager never cleans its input directories
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1914
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Dick King
> Assignee: Dick King
> Attachments: MAPREDUCE-1914--2010-07-30--1336.patch
>
>
> When we localize a file into a node's cache, it's installed in a directory whose subroot is a random {{long}} . These {{long}} s all sit in a single flat directory [per disk, per cluster node]. When the cached file is no longer needed, its reference count becomes zero in a tracking data structure. The file then becomes eligible for deletion when the total amount of space occupied by cached files exceeds 10G [by default] or the total number of such files exceeds 10K.
> However, when we delete a cached file, we don't delete the directory that contains it; this importantly includes the elements of the flat directory, which then accumulate until they reach a system limit, 32K in some cases, and then the node stops working.
> We need to delete the flat directory when we delete the localized cache file it contains.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1914) TrackerDistributedCacheManager
never cleans its input directories
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894215#action_12894215 ]
Dick King commented on MAPREDUCE-1914:
--------------------------------------
Forgot to mention: this patch also guarantees uniqueness of the local cache directory names. It's the concatenation of the creation time of the cache manager, and an incremented {{AtomicLong}} .
> TrackerDistributedCacheManager never cleans its input directories
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1914
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Dick King
> Assignee: Dick King
> Attachments: MAPREDUCE-1914--2010-07-30--1336.patch
>
>
> When we localize a file into a node's cache, it's installed in a directory whose subroot is a random {{long}} . These {{long}} s all sit in a single flat directory [per disk, per cluster node]. When the cached file is no longer needed, its reference count becomes zero in a tracking data structure. The file then becomes eligible for deletion when the total amount of space occupied by cached files exceeds 10G [by default] or the total number of such files exceeds 10K.
> However, when we delete a cached file, we don't delete the directory that contains it; this importantly includes the elements of the flat directory, which then accumulate until they reach a system limit, 32K in some cases, and then the node stops working.
> We need to delete the flat directory when we delete the localized cache file it contains.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1914) TrackerDistributedCacheManager
never cleans its input directories
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1914:
---------------------------------
Attachment: MAPREDUCE-1914--2010-07-30--1336.patch
This patch adds two advantages to the code in MAPREDUCE-1538:
* it tests the deletion of the directories
* It improve that loop of potentially 10K iterations that takes place with a lock held [ see MAPREDUCE-1909 ]
* reordering the {{if}} clause in the loop
* using {{cachedArchives.values()}}
The full regression test is in progress. I don't expect any problems, since I've done the relevant unit tests.
> TrackerDistributedCacheManager never cleans its input directories
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1914
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Dick King
> Assignee: Dick King
> Attachments: MAPREDUCE-1914--2010-07-30--1336.patch
>
>
> When we localize a file into a node's cache, it's installed in a directory whose subroot is a random {{long}} . These {{long}} s all sit in a single flat directory [per disk, per cluster node]. When the cached file is no longer needed, its reference count becomes zero in a tracking data structure. The file then becomes eligible for deletion when the total amount of space occupied by cached files exceeds 10G [by default] or the total number of such files exceeds 10K.
> However, when we delete a cached file, we don't delete the directory that contains it; this importantly includes the elements of the flat directory, which then accumulate until they reach a system limit, 32K in some cases, and then the node stops working.
> We need to delete the flat directory when we delete the localized cache file it contains.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1914) TrackerDistributedCacheManager
never cleans its input directories
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894213#action_12894213 ]
Dick King commented on MAPREDUCE-1914:
--------------------------------------
The regression passes, except for the usual {{TestTaskTrackerLocalization}} .
> TrackerDistributedCacheManager never cleans its input directories
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1914
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Dick King
> Assignee: Dick King
> Attachments: MAPREDUCE-1914--2010-07-30--1336.patch
>
>
> When we localize a file into a node's cache, it's installed in a directory whose subroot is a random {{long}} . These {{long}} s all sit in a single flat directory [per disk, per cluster node]. When the cached file is no longer needed, its reference count becomes zero in a tracking data structure. The file then becomes eligible for deletion when the total amount of space occupied by cached files exceeds 10G [by default] or the total number of such files exceeds 10K.
> However, when we delete a cached file, we don't delete the directory that contains it; this importantly includes the elements of the flat directory, which then accumulate until they reach a system limit, 32K in some cases, and then the node stops working.
> We need to delete the flat directory when we delete the localized cache file it contains.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1914) TrackerDistributedCacheManager
never cleans its input directories
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885100#action_12885100 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1914:
----------------------------------------------------
I see that this has been fixed by MAPREDUCE-1538 with following code change :
{code}
@@ -268,18 +282,9 @@
for (CacheStatus lcacheStatus : deleteSet) {
synchronized (lcacheStatus) {
deleteLocalPath(asyncDiskService,
- FileSystem.getLocal(conf), lcacheStatus.localizedLoadPath);
- // decrement the size of the cache from baseDirSize
- synchronized (baseDirSize) {
- Long dirSize = baseDirSize.get(lcacheStatus.localizedBaseDir);
- if ( dirSize != null ) {
- dirSize -= lcacheStatus.size;
- baseDirSize.put(lcacheStatus.localizedBaseDir, dirSize);
- } else {
- LOG.warn("Cannot find record of the baseDir: " +
- lcacheStatus.localizedBaseDir + " during delete!");
- }
- }
+ FileSystem.getLocal(conf), lcacheStatus.getLocalizedUniqueDir());
+ // Update the maps baseDirSize and baseDirNumberSubDir
+ deleteCacheInfoUpdate(lcacheStatus);
{code}
The above code always deletes lcacheStatus.getLocalizedUniqueDir(), which is reported in this issue. Am I missing something?
> TrackerDistributedCacheManager never cleans its input directories
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1914
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Dick King
> Assignee: Dick King
>
> When we localize a file into a node's cache, it's installed in a directory whose subroot is a random {{long}} . These {{long}} s all sit in a single flat directory [per disk, per cluster node]. When the cached file is no longer needed, its reference count becomes zero in a tracking data structure. The file then becomes eligible for deletion when the total amount of space occupied by cached files exceeds 10G [by default] or the total number of such files exceeds 10K.
> However, when we delete a cached file, we don't delete the directory that contains it; this importantly includes the elements of the flat directory, which then accumulate until they reach a system limit, 32K in some cases, and then the node stops working.
> We need to delete the flat directory when we delete the localized cache file it contains.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.