You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Christian Kunz (JIRA)" <ji...@apache.org> on 2007/06/30 05:47:04 UTC
[jira] Created: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Extensive logging of C++ application can slow down task by an order of magnitude
--------------------------------------------------------------------------------
Key: HADOOP-1553
URL: https://issues.apache.org/jira/browse/HADOOP-1553
Project: Hadoop
Issue Type: Bug
Components: pipes
Affects Versions: 0.13.0
Reporter: Christian Kunz
We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Status: Open (was: Patch Available)
It didn't work under windows.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch, new-log-3.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1553) Extensive logging of C++
application can slow down task by an order of magnitude
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12516880 ]
Hadoop QA commented on HADOOP-1553:
-----------------------------------
+0, new Findbugs warnings
http://issues.apache.org/jira/secure/attachment/12362939/new-log.patch
applied and successfully tested against trunk revision r561603,
but there appear to be new Findbugs warnings introduced by this patch.
New Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/496/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/496/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/496/console
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Attachment: (was: new-log.patch)
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1553) Extensive logging of C++
application can slow down task by an order of magnitude
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517018 ]
Hadoop QA commented on HADOOP-1553:
-----------------------------------
-1, build or testing failed
2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12362983/new-log-2.patch against trunk revision r561603.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/500/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/500/console
Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-1553) Extensive logging of C++
application can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-1553 started by Owen O'Malley.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Assignee: Owen O'Malley
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Fix For: 0.14.0
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1553) Extensive logging of C++
application can slow down task by an order of magnitude
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517133 ]
Hadoop QA commented on HADOOP-1553:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12363013/new-log-3.patch applied and successfully tested against trunk revision r561975.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/506/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/506/console
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch, new-log-3.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1553) Extensive logging of C++
application can slow down task by an order of magnitude
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510808 ]
Christian Kunz commented on HADOOP-1553:
----------------------------------------
Here some additional data:
A certain application with extensive logging for debugging turned on took 5.5 hours, with logging turned off just 3 minutes.
When logging was turned on, the throughput to userlogs was 6kBps, with about 90 log messages per sec.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Status: Patch Available (was: In Progress)
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Component/s: (was: pipes)
mapred
This applies not only to pipes, but to Java programs too. I'm investigating the cause.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Attachment: new-log-3.patch
This patch fixes the path names to work under windows.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch, new-log-3.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Attachment: new-log.patch
This patch fixes the performance problems with user task logging. Before the patch, running the word count example on a given input (Alice in Wonderland *smile*) would take 6 seconds normally and minutes if the program printed to stdout. After the patch, it takes 4 seconds with no stdout and 6 seconds with printing.
This patch includes several incompatible changes:
1. The user logs are no longer stored in segments, but rather complete files.
2. All tasks are launched via bash to get input redirection.
3. The cap on user logs has been turned off by default. It is still available, but makes the command used to launch tasks much more complicated.
4. The entire length of the user log cap is stored in memory now rather than disk. Thus, setting the cap to a large value may cause problems.
5. The task logger has fewer configuration knobs that have been removed from the log4j.properties.
6. The urls to access the task logs from the task tracker have changed. The new urls only have start and end offsets, but the offsets may be either positive from the start of the file or negative from the end of the file.
7. The jsp has been replaced by a servlet, so that the bytes don't need to be interpreted as a string.
8. The servlet does not buffer the entire log into memory before it sent to the user.
9. The TaskLog class is now public so that pipes can use it.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1553) Extensive logging of C++
application can slow down task by an order of magnitude
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517824 ]
Arun C Murthy commented on HADOOP-1553:
---------------------------------------
+1 (assuming the shell shenanigans work on Windows and cgywin+cygpath... I only tested the patch on Linux. *smile*)
Personally I'm glad we only have 3 files: ${HADOOP_LOG_DIR}/userlogs/${taskid}/[stdout|stderr|syslog] rather than subdirs... much easier to debug stuff.
Super-minor nit: Given that {{mapred.userlog.retain.hours}} defaults to 24hrs in hadoop-default.xml...
{noformat}
Index: src/java/org/apache/hadoop/mapred/TaskTracker.java
===================================================================
--- src/java/org/apache/hadoop/mapred/TaskTracker.java (revision 561588)
+++ src/java/org/apache/hadoop/mapred/TaskTracker.java (working copy)
@@ -1758,6 +1764,7 @@
Task task = umbilical.getTask(taskid);
JobConf job = new JobConf(task.getJobFile());
+ TaskLog.cleanup(job.getInt("mapred.userlog.retain.hours", 12));
task.setConf(job);
defaultConf.addFinalResource(new Path(task.getJobFile()));
{noformat}
should be:
{noformat}
+ TaskLog.cleanup(job.getInt("mapred.userlog.retain.hours", 24));
{noformat}
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch, new-log-3.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
This has been commited.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch, new-log-3.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1553) Extensive logging of C++
application can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514794 ]
Owen O'Malley commented on HADOOP-1553:
---------------------------------------
Ok, I tried moving the logging down into the Task and that fixed the performance problem, but doesn't help any code that writes directly to stdout or stderr. That doesn't help.
So I propose running all Tasks via /bin/sh, using the shell to do the redirect of stdin, stdout, and stderr. This will allow us to keep the system performant and log the user's output.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Status: Patch Available (was: Open)
Now it does.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch, new-log-3.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Attachment: new-log-2.patch
Fixed findbugs warnings.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Status: Open (was: Patch Available)
Because of find bugs warnings.
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Fix Version/s: 0.14.0
Priority: Blocker (was: Major)
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1553) Extensive logging of C++ application
can slow down task by an order of magnitude
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1553:
----------------------------------
Status: Patch Available (was: Open)
> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
> Key: HADOOP-1553
> URL: https://issues.apache.org/jira/browse/HADOOP-1553
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.13.0
> Reporter: Christian Kunz
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: new-log-2.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.