You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Dick King (JIRA)" <ji...@apache.org> on 2010/07/01 23:50:53 UTC

[jira] Created: (MAPREDUCE-1909) TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times

TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times
--------------------------------------------------------------------------------------

                 Key: MAPREDUCE-1909
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1909
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
            Reporter: Dick King
            Assignee: Dick King


In {{TrackerDistributedCachaManager.java}} , the portion where the cache is cleaned up, the lock is taken on the main hash table and then all the entries are scanned to see if they can be deleted.  That's a long lockage.  The table is likely to have 10K entries.

I would like to reduce the longest lock duration by maintaining the set of {{CacheStatus}} es to delete incrementally.

1: Let there be a new {{HashSet}}, {{deleteSet}}, that's protected under {{synchronized(cachedArchives)}}

2: When {{refcount}} is decreased to 0, move the {{CacheStatus}} from {{cachedArchives}} to {{deleteSet}}

3: When seeking an existing {{CacheStatus}}, look in {{deleteSet}} if it isn't in {{cachedArchives}}

4: When {{refcount}} is increased from 0 to 1 in a pre-existing {{CacheStatus}} [see 3:, above] move the {{CacheStatus}} from {{deleteSet}} to {{cachedArchives}}

5: When we clean the cache, under {{synchronized(cachedArchives)}} , move {{deleteSet}} to a local variable and create a new empty {{HashSet}}.  This is constant time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1909) TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times

Posted by "Dick King (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894216#action_12894216 ] 

Dick King commented on MAPREDUCE-1909:
--------------------------------------

This patch may not be needed.

The loop still exists, but in another change I made it take about half as long, and it doesn't run [and freeze the cache localization system] very often.  In our busy clusters it'll run every week or two on each node.

I made the unique directory name change elsewhere.

> TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times
> --------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1909
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1909
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Dick King
>            Assignee: Dick King
>
> In {{TrackerDistributedCachaManager.java}} , the portion where the cache is cleaned up, the lock is taken on the main hash table and then all the entries are scanned to see if they can be deleted.  That's a long lockage.  The table is likely to have 10K entries.
> I would like to reduce the longest lock duration by maintaining the set of {{CacheStatus}} es to delete incrementally.
> 1: Let there be a new {{HashSet}}, {{deleteSet}}, that's protected under {{synchronized(cachedArchives)}}
> 2: When {{refcount}} is decreased to 0, move the {{CacheStatus}} from {{cachedArchives}} to {{deleteSet}}
> 3: When seeking an existing {{CacheStatus}}, look in {{deleteSet}} if it isn't in {{cachedArchives}}
> 4: When {{refcount}} is increased from 0 to 1 in a pre-existing {{CacheStatus}} [see 3:, above] move the {{CacheStatus}} from {{deleteSet}} to {{cachedArchives}}
> 5: When we clean the cache, under {{synchronized(cachedArchives)}} , move {{deleteSet}} to a local variable and create a new empty {{HashSet}}.  This is constant time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1909) TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times

Posted by "Dick King (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884485#action_12884485 ] 

Dick King commented on MAPREDUCE-1909:
--------------------------------------

Might as well make our unique directory names with 

{noformat}

     static private AtomicLong uniqueDirNumber = new AtomicLong();

  ...

     String uniqueString = "D" + uniqueDirNumber.getAndIncrement();

{noformat}

if we're in the area.

> TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times
> --------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1909
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1909
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Dick King
>            Assignee: Dick King
>
> In {{TrackerDistributedCachaManager.java}} , the portion where the cache is cleaned up, the lock is taken on the main hash table and then all the entries are scanned to see if they can be deleted.  That's a long lockage.  The table is likely to have 10K entries.
> I would like to reduce the longest lock duration by maintaining the set of {{CacheStatus}} es to delete incrementally.
> 1: Let there be a new {{HashSet}}, {{deleteSet}}, that's protected under {{synchronized(cachedArchives)}}
> 2: When {{refcount}} is decreased to 0, move the {{CacheStatus}} from {{cachedArchives}} to {{deleteSet}}
> 3: When seeking an existing {{CacheStatus}}, look in {{deleteSet}} if it isn't in {{cachedArchives}}
> 4: When {{refcount}} is increased from 0 to 1 in a pre-existing {{CacheStatus}} [see 3:, above] move the {{CacheStatus}} from {{deleteSet}} to {{cachedArchives}}
> 5: When we clean the cache, under {{synchronized(cachedArchives)}} , move {{deleteSet}} to a local variable and create a new empty {{HashSet}}.  This is constant time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1909) TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times

Posted by "Dick King (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dick King updated MAPREDUCE-1909:
---------------------------------

    Priority: Minor  (was: Major)

> TrackerDistributedCacheManager takes a blocking lock fo a loop that executes 10K times
> --------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1909
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1909
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Dick King
>            Assignee: Dick King
>            Priority: Minor
>
> In {{TrackerDistributedCachaManager.java}} , the portion where the cache is cleaned up, the lock is taken on the main hash table and then all the entries are scanned to see if they can be deleted.  That's a long lockage.  The table is likely to have 10K entries.
> I would like to reduce the longest lock duration by maintaining the set of {{CacheStatus}} es to delete incrementally.
> 1: Let there be a new {{HashSet}}, {{deleteSet}}, that's protected under {{synchronized(cachedArchives)}}
> 2: When {{refcount}} is decreased to 0, move the {{CacheStatus}} from {{cachedArchives}} to {{deleteSet}}
> 3: When seeking an existing {{CacheStatus}}, look in {{deleteSet}} if it isn't in {{cachedArchives}}
> 4: When {{refcount}} is increased from 0 to 1 in a pre-existing {{CacheStatus}} [see 3:, above] move the {{CacheStatus}} from {{deleteSet}} to {{cachedArchives}}
> 5: When we clean the cache, under {{synchronized(cachedArchives)}} , move {{deleteSet}} to a local variable and create a new empty {{HashSet}}.  This is constant time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.