You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Mahadev konar (JIRA)" <ji...@apache.org> on 2010/07/14 23:13:56 UTC
[jira] Created: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
------------------------------------------------------------------------
Key: MAPREDUCE-1943
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.22.0
We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-0.20-yahoo.patch
this patch adds tests to the above featuers.
also, changed the limits to group = 50 and counters in each group = 70.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1521-0.20-yahoo.patch
this patch imposes some limits.
the following are the limits it imposes:
1) The number of counters per group is limited to 40. If the counters increase that amount they are dropped silently.
2) The number of counter groups is restricted to 40. Again if the groups are more than the limit they are dropped silently.
3) The string size of counter name is restricted to 64 characters.
4) the string size of group name is restricted to 128 characters.
5) The number of block locations returned by a split is restricted to 100, this can be changed with a configuration parameter.
6) limit the reporter.setstatus() string size to 512 characters.
I havent added tests yet. Will upload one shortly. Also, this patch is for yahoo 0.20 branch. I will upload one for the trunk shortly.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: (was: MAPREDUCE-1521-0.20-yahoo.patch)
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888693#action_12888693 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1943:
----------------------------------------------------
Limiting task diagnostic info and status are done in MAPREDUCE-1482.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
an updated patch with test cases and a limit of 80 on counters. This patch throws a runtimeexception if the limit on counters is exceeded. Also, the number of block locations has a hard limit of 100.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Fix Version/s: 0.22.0
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
the fix addendum with a test case.
I will be uploading a single patch for trunk soon.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
this patch is an addendum to the last patch. This fixes a bug wherein counters arent counted across tasks before the job completes. This patch updates the numer of counters on every heartbeat for the job and kill the job in case it exceeds the limit.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
fixes minor bug in my earlier patch with respect to configuration settings and calling getjobcounters without job being initialized.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888557#action_12888557 ]
Scott Chen commented on MAPREDUCE-1943:
---------------------------------------
+1 to the idea. We have seen the huge split-size kills JT. This will help.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
this patch is updated with some loopholes we found while testing. It prevents the exceeded exception from interrupting the flow of job expiration.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Fix Version/s: (was: 0.22.0)
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
the earlier patches on every heartbeat computed to see if the counters had exceeded the limit. I made a change in this patch to make it much lighter. Here the check is done only after job is done running its maps and reduce tasks.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
an updated patch that fixes findbugs warnings and also makes sure we check for counters return value always.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Liyin Liang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920079#action_12920079 ]
Liyin Liang commented on MAPREDUCE-1943:
----------------------------------------
your latest patch is based on your previous patch, why?
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-0.20-yahoo.patch
attached the wrong file.. :)
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: (was: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch)
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job
JobConf, Counters, StatusReport, Split-Sizes
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
sorry attached a wrong file.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.