You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2007/07/04 19:38:04 UTC
[jira] Created: (HADOOP-1564) Write unit tests to detect CRC
corruption
Write unit tests to detect CRC corruption
-----------------------------------------
Key: HADOOP-1564
URL: https://issues.apache.org/jira/browse/HADOOP-1564
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur
Assignee: dhruba borthakur
The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510252 ]
dhruba borthakur commented on HADOOP-1564:
------------------------------------------
The above tests are run for variead values of io.bytes.per.checksum and dfs.block.size. It tests for the case when the meta file is multiple blocks.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.3
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Attachment: crctest.5
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.5
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Attachment: (was: crctest.3)
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12511516 ]
Raghu Angadi commented on HADOOP-1564:
--------------------------------------
Dhruba,
Yesterday we started enforcing the condition that block size should be a multiple of io.bytes.per.checksum for new files. This test fails since one of the configs does not match. Could you update the patch with the correction?
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510446 ]
dhruba borthakur commented on HADOOP-1564:
------------------------------------------
Of course the patch process failed. Because HADOOP-1134 is a pre-requisite for this one.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12511539 ]
Raghu Angadi commented on HADOOP-1564:
--------------------------------------
One more change in DFSTestUtil.java: read() used to try to read as much of data as possible. Now FSInputChecker reads at most bytesPerChecksum in each read. It also provides readFully() which reads requested number of bytes in a loop. DFSTestUtil should replace read() with readFully.
With the above two changes, these tests are passing.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510445 ]
Hadoop QA commented on HADOOP-1564:
-----------------------------------
-1, build or testing failed
2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12361213/crctest.4 against trunk revision r553080.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/363/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/363/console
Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Attachment: crctest.4
Added javadoc comments to the Unit test.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Attachment: (was: crctest.4)
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.5
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1564:
---------------------------------
Fix Version/s: 0.14.0
Status: Open (was: Patch Available)
This should be marked "Patch Available" again after HADOOP-1134 is committed.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Attachment: crctest.5
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.5
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-1564) Write unit tests to
detect CRC corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510231 ]
dhruba borthakur edited comment on HADOOP-1564 at 7/4/07 11:56 AM:
-------------------------------------------------------------------
This test creates a bunch of files/directories with replication factor of 2. Then verifies that a client can automatically accesses the remaining valid replica inspite of the following types of simulated errors:
1. Delete meta file on one replica
2. Truncates meta file on one replica
3. Corrupts the meta file header on one replica
4. Corrupts any random offset and portion of the meta file
5. Swaps two meta files, i.e the format of the meta files are valid but their CRCs do not match with their corresponding data blocks
Another portion of the test is commented out till HADOOP-1557 is addressed:
1. Create file with 2 replica, corrupt the meta file of replica, decrease replication factor from 2 to 1. Validate that the remaining replica is the good one.
2. Create file with 2 replica, corrupt the meta file of one replica, increase replication factor of file to 3. verify that the new replica was created from the non-corrupted replica.
was:
This test does the following creates a bunch of files/directories with replication factor of 2. Then verifies that a client can automatically accesses the remining valid replica inspite of the following tyoes of simulated errors:
1. Delete meta file on one replica
2. Truncates meta file on one replica
3. Corrupts the meta file header on one replica
4. Corrupts any random offset and portion of the meta file
5. Swaps two meta files, i.e the format of the meta files are valid but their CRCs do not match with their corresponding data blocks
Another portion of the test is being commented out till HADOOP-1557 is addressed:
1. Create file with 2 replica, corrupt the meta file of replica, decrease replication factor from 2 to 1. Validate that the remaining replica is the good one.
2. Create file with 2 replica, corrupt the meta file of one replica, increase replication factor of file to 3. verify that the new replica was created from the non-corrupted replica.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.3
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1564:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Dhruba!
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.5
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-1564) Write unit tests to
detect CRC corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510252 ]
dhruba borthakur edited comment on HADOOP-1564 at 7/4/07 2:57 PM:
------------------------------------------------------------------
The above tests are run for varied values of io.bytes.per.checksum and dfs.block.size. It tests for the case when the meta file is multiple blocks.
was:
The above tests are run for variead values of io.bytes.per.checksum and dfs.block.size. It tests for the case when the meta file is multiple blocks.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.3
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Attachment: (was: crctest.5)
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.5
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513342 ]
Hadoop QA commented on HADOOP-1564:
-----------------------------------
-1, build or testing failed
2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12361998/crctest.5 against trunk revision r556754.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/426/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/426/console
Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.5
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Status: Patch Available (was: Open)
Submitting patch so that it triggers findbugs and javadocs warnings (if any). This can be applied only after HADOOP-1134 is committed.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.4
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Attachment: crctest.3
This test does the following creates a bunch of files/directories with replication factor of 2. Then verifies that a client can automatically accesses the remining valid replica inspite of the following tyoes of simulated errors:
1. Delete meta file on one replica
2. Truncates meta file on one replica
3. Corrupts the meta file header on one replica
4. Corrupts any random offset and portion of the meta file
5. Swaps two meta files, i.e the format of the meta files are valid but their CRCs do not match with their corresponding data blocks
Another portion of the test is being commented out till HADOOP-1557 is addressed:
1. Create file with 2 replica, corrupt the meta file of replica, decrease replication factor from 2 to 1. Validate that the remaining replica is the good one.
2. Create file with 2 replica, corrupt the meta file of one replica, increase replication factor of file to 3. verify that the new replica was created from the non-corrupted replica.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: crctest.3
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1564) Write unit tests to detect CRC
corruption
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1564:
-------------------------------------
Status: Patch Available (was: Open)
Unit tests for block CRC.
> Write unit tests to detect CRC corruption
> -----------------------------------------
>
> Key: HADOOP-1564
> URL: https://issues.apache.org/jira/browse/HADOOP-1564
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.14.0
>
> Attachments: crctest.5
>
>
> The unit tests should have some way to test the case when CRC files are corrupted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.