You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Luke Lu (JIRA)" <ji...@apache.org> on 2010/08/21 03:33:16 UTC
[jira] Created: (MAPREDUCE-2025) Improve job counter wire protocol
with incremental schema and delta values
Improve job counter wire protocol with incremental schema and delta values
--------------------------------------------------------------------------
Key: MAPREDUCE-2025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2025
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, jobtracker, task
Affects Versions: 0.20.2
Reporter: Luke Lu
Assignee: Luke Lu
Fix For: 0.22.0
The current job counter protocol is quite verbose (string key and vlong value pairs) especially when keys are long. By using an incremental schema (key to key id mapping) and data (vint key id to vlong value delta) protocol, we can significantly lower the counter overhead and allow us to up (from the current 1 update/minute) the counter update frequency. This jira tracks the user counter (for apps or tools (pig etc.) built on top of mr framework) improvement. Framework (mr/hdfs etc) counter improvement is tracked by MAPREDUCE-901 (a more common case with more optimization opportunities, as the schema is fixed.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.