You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hong Tang (JIRA)" <ji...@apache.org> on 2010/08/20 11:25:29 UTC
[jira] Created: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
TestDFSIO read test may not read specified bytes.
-------------------------------------------------
Key: MAPREDUCE-2023
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks
Reporter: Hong Tang
TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926080#action_12926080 ]
Hudson commented on MAPREDUCE-2023:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.22.0
>
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, mr-2023-yahoo-hadoop-20.1xx.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang reassigned MAPREDUCE-2023:
------------------------------------
Assignee: Hong Tang
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-2023:
---------------------------------
Status: Open (was: Patch Available)
The earlier patch was not generated with --no-prefix. Will upload a new one.
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2023:
----------------------------------------------
Status: Open (was: Patch Available)
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900652#action_12900652 ]
Hong Tang commented on MAPREDUCE-2023:
--------------------------------------
The problem is due to the following code segments:
{code}
public static class ReadMapper extends IOStatMapper<Long> {
public ReadMapper() {
}
public Long doIO(Reporter reporter,
String name,
long totalSize // in bytes
) throws IOException {
// open file
DataInputStream in = fs.open(new Path(getDataDir(getConf()), name));
long actualSize = 0;
try {
for(int curSize = bufferSize;
curSize == bufferSize && actualSize < totalSize;) { // <-- HERE
curSize = in.read(buffer, 0, bufferSize);
if(curSize < 0) break;
actualSize += curSize;
reporter.setStatus("reading " + name + "@" +
actualSize + "/" + totalSize
+ " ::host = " + hostName);
}
} finally {
in.close();
}
return Long.valueOf(actualSize);
}
}
{code}
The problem is that the for-loop breaks out as soon as the previous read fails to fulfill the full buffer. The fix is pretty simple:
{code}
for(int curSize = bufferSize; actualSize < totalSize;) {
{code}
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-2023:
---------------------------------
Attachment: mr-2023-yahoo-hadoop-20.1xx.patch
patch for yahoo hadoop 20.1xx branch. Not to be committed.
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.22.0
>
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, mr-2023-yahoo-hadoop-20.1xx.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-2023:
---------------------------------
Attachment: mr-2023-20100826.patch
trivial patch. for-loop is replaced with a while-loop
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-2023:
---------------------------------
Status: Patch Available (was: Open)
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905759#action_12905759 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-2023:
---------------------------------------------------
I ran TestDFSIO with a similar patch earlier. Without the patch, the number of bytes read by TestDFSIO may be less than the one specified in the parameter. In such case, the reported Throughput and Average IO rate could be > 1 GB/second, which is obviously incorrectly.
I have committed this. Thanks, Hong!
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905745#action_12905745 ]
Hong Tang commented on MAPREDUCE-2023:
--------------------------------------
test-patch passed on my local machine:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2023:
----------------------------------------------
Status: Patch Available (was: Open)
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2023:
----------------------------------------------
Hadoop Flags: [Reviewed]
+1 patch looks good.
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-2023:
---------------------------------
Status: Patch Available (was: Open)
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2023:
----------------------------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.22.0
Resolution: Fixed
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.22.0
>
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900654#action_12900654 ]
Hong Tang commented on MAPREDUCE-2023:
--------------------------------------
To confirm that DFS indeed may return less than requested bytes even before reaching the end, I wrote a test program (attached), and the output of a sample run looks as follows:
{noformat}
hadoop dfs -ls /user/gridperf/gridmix3/part-m-00332/segment-0
Found 1 items
-rw-rw-rw- 3 gridperf hdfs 1073741824 2010-08-20 08:22 /user/gridperf/gridmix3/part-m-00332/segment-0
hadoop org.apache.hadoop.fs.TestFsRead /user/gridperf/gridmix3/part-m-00332/segment-0 1000000
10995954 bytes read
21199983 bytes read
32106261 bytes read
42209617 bytes read
52456131 bytes read
63551911 bytes read
73836262 bytes read
84369397 bytes read
95182878 bytes read
105047397 bytes read
115740295 bytes read
126323360 bytes read
137166764 bytes read
147066000 bytes read
157744477 bytes read
168319334 bytes read
178856592 bytes read
188884554 bytes read
199324045 bytes read
209995098 bytes read
220916802 bytes read
231218738 bytes read
241772291 bytes read
251883835 bytes read
262306687 bytes read
Fail to read a full buffer before reaching the end: pos=267640189, expected=994623, actual=795267.
272862612 bytes read
283737254 bytes read
293851212 bytes read
304525446 bytes read
314766024 bytes read
325604342 bytes read
335604768 bytes read
346475397 bytes read
357311830 bytes read
367574920 bytes read
377834612 bytes read
388029682 bytes read
398728223 bytes read
408966064 bytes read
419626247 bytes read
430260987 bytes read
440440647 bytes read
451030835 bytes read
461808645 bytes read
471996795 bytes read
482529325 bytes read
493106417 bytes read
503960340 bytes read
514155195 bytes read
524460261 bytes read
534955349 bytes read
Fail to read a full buffer before reaching the end: pos=536250423, expected=999458, actual=620489.
545734170 bytes read
556326582 bytes read
567046173 bytes read
577480068 bytes read
587338410 bytes read
598115745 bytes read
608759717 bytes read
619418792 bytes read
629597629 bytes read
639906390 bytes read
650264871 bytes read
661414262 bytes read
671205472 bytes read
681856772 bytes read
692394138 bytes read
702803762 bytes read
713182701 bytes read
723720128 bytes read
734531251 bytes read
745188960 bytes read
755814801 bytes read
765670009 bytes read
776047213 bytes read
786592324 bytes read
797786600 bytes read
Fail to read a full buffer before reaching the end: pos=804788073, expected=613320, actual=518295.
808158276 bytes read
818373817 bytes read
828549794 bytes read
838915719 bytes read
850189376 bytes read
860102547 bytes read
870902116 bytes read
881206170 bytes read
891441081 bytes read
902119052 bytes read
912394977 bytes read
923010497 bytes read
933330792 bytes read
944216276 bytes read
954226049 bytes read
965371734 bytes read
975663038 bytes read
986215681 bytes read
996274088 bytes read
1006954729 bytes read
1017375248 bytes read
1027801749 bytes read
1038384467 bytes read
1049383853 bytes read
1059662742 bytes read
1070106760 bytes read
{noformat}
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-2023:
---------------------------------
Attachment: mr-2023-20100902.patch
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-2023-20100826.patch, mr-2023-20100902.patch, TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2023) TestDFSIO read test may not read
specified bytes.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-2023:
---------------------------------
Attachment: TestFsRead.java
test program
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
> Attachments: TestFsRead.java
>
>
> TestDFSIO's read test may read less bytes than specified when reading large files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.