[ https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111987#comment-16111987 ]
ASF GitHub Bot commented on MAPREDUCE-6931:
-------------------------------------------
GitHub user dennishuo opened a pull request:
https://github.com/apache/hadoop/pull/259
MAPREDUCE-6931. Fix TestDFSIO "Total Throughput" calculation.
Previously it failed to convert ms to seconds and thus reports aggregate
throughput as 1/1000x actual numbers. Also, make all the bytes-to-mb
and milliseconds-to-seconds conversions consistent in the reporting
messages to help avoid this type of error in the future.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dennishuo/hadoop trunk
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop/pull/259.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #259
----
commit 23a374d6e818004daa0b1cf16c4fedb439ab4943
Author: Dennis Huo <dh...@google.com>
Date: 2017-08-02T21:30:51Z
MAPREDUCE-6931. Fix TestDFSIO "Total Throughput" calculation.
Previously it failed to convert ms to seconds and thus reports aggregate
throughput as 1/1000x actual numbers. Also, make all the bytes-to-mb
and milliseconds-to-seconds conversions consistent in the reporting
messages to help avoid this type of error in the future.
----
> Fix TestDFSIO "Total Throughput" calculation
> --------------------------------------------
>
> Key: MAPREDUCE-6931
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks, test
> Affects Versions: 2.8.0
> Reporter: Dennis Huo
> Priority: Trivial
>
> The new "Total Throughput" line added in https://issues.apache.org/jira/browse/HDFS-9153 is currently calculated as {{toMB(size) / ((float)execTime)}} and claims to be in units of "MB/s", but {{execTime}} is in milliseconds; thus, the reported number is 1/1000x the actual value:
> {code:java}
> String resultLines[] = {
> "----- TestDFSIO ----- : " + testType,
> " Date & time: " + new Date(System.currentTimeMillis()),
> " Number of files: " + tasks,
> " Total MBytes processed: " + df.format(toMB(size)),
> " Throughput mb/sec: " + df.format(size * 1000.0 / (time * MEGA)),
> "Total Throughput mb/sec: " + df.format(toMB(size) / ((float)execTime)),
> " Average IO rate mb/sec: " + df.format(med),
> " IO rate std deviation: " + df.format(stdDev),
> " Test exec time sec: " + df.format((float)execTime / 1000),
> "" };
> {code}
> The different calculated fields can also use toMB and a shared milliseconds-to-seconds conversion to make it easier to keep units consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org