You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Lei Xu (JIRA)" <ji...@apache.org> on 2009/04/03 20:43:13 UTC
[jira] Created: (HADOOP-5625) Add I/O duration time in client trace
Add I/O duration time in client trace
--------------------------------------
Key: HADOOP-5625
URL: https://issues.apache.org/jira/browse/HADOOP-5625
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Affects Versions: 0.19.1
Reporter: Lei Xu
Assignee: Lei Xu
Priority: Minor
Fix For: 0.21.0
Add I/O duration information into client trace log for analyzing performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698051#action_12698051 ]
Hadoop QA commented on HADOOP-5625:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12405185/duration-3.patch
against trunk revision 764085.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/182/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/182/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/182/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/182/console
This message is automatically generated.
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-0.19.1.patch, duration-1.patch, duration-3.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697977#action_12697977 ]
Chris Douglas commented on HADOOP-5625:
---------------------------------------
+1
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-0.19.1.patch, duration-1.patch, duration-3.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5625) Add I/O duration time in client trace
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-5625:
---------------------------
Attachment: duration.patch
Put duration time into client trace
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696560#action_12696560 ]
Hong Tang commented on HADOOP-5625:
-----------------------------------
If we only care about lapsed time, maybe System.nanoTime a better choice? (offering better resolution and is less expensive than currentTimeMillis)
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-1.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695626#action_12695626 ]
Lei Xu commented on HADOOP-5625:
--------------------------------
It fails from
[junit] Running org.apache.hadoop.cli.TestCLI
[junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 34 sec
[junit] Test org.apache.hadoop.cli.TestCLI FAILED
which is not caused by my patch, since svn HEAD source fails too.
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695623#action_12695623 ]
Brian Bockelman commented on HADOOP-5625:
-----------------------------------------
Hi,
This patch is useful for us to do timing statistics on our cluster - without it, we can't tell how many MB/s we're doing via pure HDFS.
Compiles and works for me -- tests pass. Don't know what the Hudson failure is from.
Brian
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701095#action_12701095 ]
Chris Douglas commented on HADOOP-5625:
---------------------------------------
{noformat}
Test org.apache.hadoop.hdfs.server.namenode.TestReplicationPolicy FAILED
Test org.apache.hadoop.mapred.TestMRServerPorts FAILED
Test org.apache.hadoop.mapred.TestQueueCapacities FAILED
{noformat}
None of the test failures are related to this patch.
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-0.19.1.patch, duration-1.patch, duration-3.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701256#action_12701256 ]
Hudson commented on HADOOP-5625:
--------------------------------
Integrated in Hadoop-trunk #813 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/813/])
. Add operation duration to clienttrace. Contributed by Lei Xu
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-0.19.1.patch, duration-1.patch, duration-3.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5625) Add I/O duration time in client trace
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-5625:
---------------------------
Attachment: duration-0.19.1.patch
This patch works for 0.19.1. It has to be applied after HADOOP-5222.
Since there are some format issues between 0.19.1 and svn HEAD. So I submit this one for build a 0.19.1 release.
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-0.19.1.patch, duration-1.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5625) Add I/O duration time in client trace
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-5625:
---------------------------
Attachment: duration-1.patch
I put the endTime assignment just after receiver close(), You are right, this approach is more accurate .
And I add an entry into TaskTracker, however I am not familiar with TaskTracker, I am not so sure it is correct.
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-1.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5625) Add I/O duration time in client trace
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-5625:
---------------------------
Attachment: duration-3.patch
Use System.nanoTime() to measure the duration time for I/O operation.
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-0.19.1.patch, duration-1.patch, duration-3.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695949#action_12695949 ]
Chris Douglas commented on HADOOP-5625:
---------------------------------------
This will be useful. A couple nits:
* {{System.currentTimeMillis}} can be fairly expensive; would it make sense to guard the call with {{ClientTraceLog.isInfoEnabled}}? It should also be final, as in {code}final long starttime = ClientTraceLog.isInfoEnabled() ? System.currentTimeMillis() : 0;{code}
* Similarly, endtime should be final and assigned right after the operation completes. In BlockReceiver, it's not obvious to me whether it should follow the set of synchronized calls into the datanode that precede it (as in the current patch) or if the time of the transfer is more accurately recorded before those calls. If this will be used to calculate network metrics, then the latter should be preferred.
* This should also add an entry for duration to the shuffle metrics in the TT clientrace log, in TaskTracker.MapOutputServlet
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5625) Add I/O duration time in client trace
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5625:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this. Thanks, Lei
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration-0.19.1.patch, duration-1.patch, duration-3.patch, duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5625) Add I/O duration time in client trace
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-5625:
---------------------------
Release Note: Add duration time into client trace
Status: Patch Available (was: Open)
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5625) Add I/O duration time in client
trace
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695546#action_12695546 ]
Hadoop QA commented on HADOOP-5625:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12404577/duration.patch
against trunk revision 761632.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/112/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/112/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/112/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/112/console
This message is automatically generated.
> Add I/O duration time in client trace
> --------------------------------------
>
> Key: HADOOP-5625
> URL: https://issues.apache.org/jira/browse/HADOOP-5625
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.1
> Reporter: Lei Xu
> Assignee: Lei Xu
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: duration.patch
>
>
> Add I/O duration information into client trace log for analyzing performance.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.