You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Joseph Evans (JIRA)" <ji...@apache.org> on 2011/06/07 16:02:58 UTC
[jira] [Created] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Throttle the deletion of data from the distributed cache
--------------------------------------------------------
Key: MAPREDUCE-2572
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: distributed-cache
Affects Versions: 0.20.205.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans reopened MAPREDUCE-2572:
--------------------------------------------
OK Aaron you convinced me. I'll submit a patch to bump it up to keep something like 95% of the maximum around in between garbage collections.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans resolved MAPREDUCE-2572.
--------------------------------------------
Resolution: Won't Fix
I filed this and the more I think about it that setting the amount of the distributed cache to keep around between cleanings to a high number really seems like the best way to deal with this. Since it is just a configuration value there is no need to make any changes to code so I will just close this as Won't fix.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070704#comment-13070704 ]
Robert Joseph Evans commented on MAPREDUCE-2572:
------------------------------------------------
This is a simple change (The default value for a config) so I felt that no new tests would be needed. As for the failing tests and findbugs warning. they are not mine. I changed 1 line that does not impact either of the failed tests, or contrib. and the find bugs issue is in a completely different area.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2572-trunk-v1.patch, THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2572:
-------------------------------------------
Fix Version/s: 0.23.0
Affects Version/s: (was: 0.20.205.0)
0.23.0
Status: Patch Available (was: Reopened)
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2572-trunk-v1.patch, THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070529#comment-13070529 ]
Robert Joseph Evans commented on MAPREDUCE-2572:
------------------------------------------------
I have updated the patch in MAPREDUCE-2494 for 0.20.205 to have 0.95 as the default, so I will only submit a patch for trunk and 0.22.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046625#comment-13046625 ]
Robert Joseph Evans commented on MAPREDUCE-2572:
------------------------------------------------
Another thought I had was that if the high and low water marks are very close to one another then perhaps we don't really need to throttle at all. That way we would do much more frequent deletion of archives, but we would delete a lot less each time. Perhaps this JIRA should transform into change the default low water mark to be 95% of the high water mark or even higher.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2572:
-------------------------------------------
Attachment: THROTTLING-security-v1.patch
This patch includes a backport of MAPREDUCE-2494 LRU ordering of deletion and throttling. Currently we are throttling based off of a given number of bytes per second. There is a lot of work that still needs to go into this. The tests need to be improved and the sleep interval needs to take into account the amount of time spent actually deleting data.
It has also been suggested that perhaps we want to have the throttling be tied to the fill rate of the cache, so that the faster it fills the faster we clear it out. I would like some feedback on this.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045533#comment-13045533 ]
Robert Joseph Evans commented on MAPREDUCE-2572:
------------------------------------------------
The patch I posted is against 20.S I am working on a patch for trunk too.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070334#comment-13070334 ]
Aaron T. Myers commented on MAPREDUCE-2572:
-------------------------------------------
Hey Robert, even if no algorithmic changes are necessary to address this issue, perhaps a good change to make would be to change the default value of this config to a higher value?
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans resolved MAPREDUCE-2572.
--------------------------------------------
Resolution: Duplicate
This is not longer relevant because MRV1 is deprecated. MAPREDUCE-2969 will do the same work for MRV2.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2572-trunk-v1.patch, THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070695#comment-13070695 ]
Hadoop QA commented on MAPREDUCE-2572:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12487725/MR-2572-trunk-v1.patch
against trunk revision 1150533.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
org.apache.hadoop.fs.TestFileSystem
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/500//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/500//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/500//console
This message is automatically generated.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2572-trunk-v1.patch, THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2572:
-------------------------------------------
Status: Open (was: Patch Available)
Now that YARN is in trunk the patch really does not apply any more. I will take a look at the cache deletion policy and see if I should just close this JIRA or not.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2572-trunk-v1.patch, THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2572:
-------------------------------------------
Attachment: MR-2572-trunk-v1.patch
It looks like the LRU changes never made it into 0.22 so no patch will be submitted for that.
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-2572-trunk-v1.patch, THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Nathan Roberts (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046036#comment-13046036 ]
Nathan Roberts commented on MAPREDUCE-2572:
-------------------------------------------
One option might be to just do something very simple after we wakeup from every CheckPeriod. Rather than throttling logic that could be a little sophisticated and error prone. Maybe just something like:
- If below low water mark - do nothing
- If between low and high water marks - delete up-to a configurable number of archives (e.g. 1 is probably a nice default), or until LWM is reached
- If above high water mark, delete archives until back at low water mark
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
from the distributed cache
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045529#comment-13045529 ]
Todd Lipcon commented on MAPREDUCE-2572:
----------------------------------------
This should be against trunk, right?
> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-2572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distributed-cache
> Affects Versions: 0.20.205.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira