You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Nigel Daley (JIRA)" <ji...@apache.org> on 2007/07/18 05:42:04 UTC
[jira] Created: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Block CRC Unit Tests: upgrade test
----------------------------------
Key: HADOOP-1629
URL: https://issues.apache.org/jira/browse/HADOOP-1629
Project: Hadoop
Issue Type: Test
Components: dfs
Affects Versions: 0.14.0
Reporter: Nigel Daley
Priority: Blocker
Fix For: 0.14.0
HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
- zero length files
- file with replication set higher than number of datanodes
- file with no .crc file
- file with corrupt .crc file
- file with multiple blocks (will need to set dfs.block.size to a small value)
- file with multiple checksum blocks
- empty directory
- all of the above again but with a different io.bytes.per.checksum setting
The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar reopened HADOOP-1629:
-----------------------------------
I am reopening this issue, since TestDFSUpgradeFromImage fails for hadoop-patch and hudson-nightly builds on hudson.
The error thrown is :
{noformat}
java.io.IOException: tar: z: unknown function modifier
at org.apache.hadoop.fs.Command.run(Command.java:33)
at org.apache.hadoop.fs.Command.execCommand(Command.java:89)
at org.apache.hadoop.dfs.TestDFSUpgradeFromImage.setUp(TestDFSUpgradeFromImage.java:75)
Standard Output
2007-08-15 13:22:38,601 INFO dfs.TestDFSUpgradeFromImage (TestDFSUpgradeFromImage.java:setUp(72)) - Unpacking the tar file /export/home/hudson/hudson/jobs/Hadoop-Patch/workspace/trunk/build/test/cache/hadoop-12-dfs-dir.tgz
{noformat}
It seems that gzip is not installed on the lucene.zones.apache.org . Can someone with the privileges check this out.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515445 ]
Raghu Angadi commented on HADOOP-1629:
--------------------------------------
Any suggestions on getting tar.gz functionality in java? Should we pack the directory tree with our own simple format?
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519494 ]
Raghu Angadi commented on HADOOP-1629:
--------------------------------------
This conflicts with HADOOP-1621 on the trunk. Will submit different patches for 14 and trunk.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1629:
---------------------------------
Attachment: HADOOP-1629.patch
Thanks to Nigel for helping through this.
Attached patch for new unit test "TestDFSUpgradeFromImage". This is an end-to-end test for upgrade from Hadoop-0.12 to current version. The initial image contain the various categories and error that Nigel mentioned in the Jira desription.
For now we are using tar-gzipped file. Hadoop anyway requires cygwin. Once HADOOP-1622 goes in we can change the format.
The patch does not actually contain the {{.tgz}} file. Will attach it. hadoop-12-dfs-dir.txt contains a description of the data and the file checksums that are verified during the unit test.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye reassigned HADOOP-1629:
----------------------------------------
Assignee: Raghu Angadi
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519489 ]
Hadoop QA commented on HADOOP-1629:
-----------------------------------
-1, build or testing failed
2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12363602/HADOOP-1629.patch against trunk revision r565434.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/545/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/545/console
Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-1629:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Raghu!
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1629:
---------------------------------
Attachment: HADOOP-1629-trunk.patch
Attached patch for the trunk. Please use HADOOP-1629.patch for branch 14. The difference between the patch is minor. (TestDFSUpgradeFromTrunk.java:136).
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley resolved HADOOP-1629.
---------------------------------
Resolution: Fixed
Enis, I opened a new Jira, HADOOP-1717, to track the failing of this test on Solaris.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-1629:
--------------------------------
Status: Patch Available (was: Open)
This patch is ready.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1629:
---------------------------------
Attachment: hadoop-12-dfs-dir.tgz
attaching hadoop-12-dfs-dir.tgz
Note commiters: Please place this file next to hadoop-12-dfs-dir.txt
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519523 ]
Raghu Angadi commented on HADOOP-1629:
--------------------------------------
The above failure is expected. "patch available" is valid.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515070 ]
Nigel Daley commented on HADOOP-1629:
-------------------------------------
I would suspect this could be far smaller than 500Kb once
compressed. Also, I think you want to use the java.io.Zip* stuff to
zip and unzip.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518875 ]
Nigel Daley commented on HADOOP-1629:
-------------------------------------
Looks good, just a few code review comments:
the test needs more comments, especially a reference to the ancillary files that it depends on
private void setupDFSImage()
should be
public void setup()
which Junit will call automatically before the test method.
while ( (line = reader.readLine()) != null ) {
perhaps should be
while ( (line = reader.readLine().trim()) != null ) {
some spacing is wrong
value variable should be better named
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515067 ]
Doug Cutting commented on HADOOP-1629:
--------------------------------------
> Any suggestions on how large the tar-gzip file should be?
1MB? We should be able to use small block and buffer sizes to get all of the desired sample files into a meg, no?
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519520 ]
Hadoop QA commented on HADOOP-1629:
-----------------------------------
-1, build or testing failed
2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12363716/HADOOP-1629-trunk.patch against trunk revision r565434.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/547/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/547/console
Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515867 ]
Doug Cutting commented on HADOOP-1629:
--------------------------------------
> Any suggestions on getting tar.gz functionality in java?
HADOOP-1622 adds tools to both pack and unpack jar files. (We currently have only a tool to unpack them in Hadoop, in util.RunJar.) Will those work? Jar files can have compression.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1629:
---------------------------------
Status: Patch Available (was: Open)
TestDFSUpgradeFromImage will fail in the build since it will not have the tar file.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1629:
---------------------------------
Attachment: HADOOP-1629.patch
Previous patch was missing a file. Now fixed.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1629:
---------------------------------
Attachment: HADOOP-1629.patch
Thanks for the feedback Nigel.
Attached the improved patch that includes all of the suggested changes.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515069 ]
Raghu Angadi commented on HADOOP-1629:
--------------------------------------
1MB sounds good. Will plan to keep it even smaller (< 500K) if possible.. so that this will be in the same order as some of the larger source files we have.
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515064 ]
Raghu Angadi commented on HADOOP-1629:
--------------------------------------
Any suggestions on how large the tar-gzip file should be? I can write simple text to blocks so that they compress very well. Since this is meant to be committed, what do you think is reasonable size?
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1629) Block CRC Unit Tests: upgrade test
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1629:
---------------------------------
Status: Open (was: Patch Available)
> Block CRC Unit Tests: upgrade test
> ----------------------------------
>
> Key: HADOOP-1629
> URL: https://issues.apache.org/jira/browse/HADOOP-1629
> Project: Hadoop
> Issue Type: Test
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: hadoop-12-dfs-dir.tgz, HADOOP-1629-trunk.patch, HADOOP-1629.patch, HADOOP-1629.patch, HADOOP-1629.patch
>
>
> HADOOP-1286 introduced a distributed upgrade framework. 1 or more unit tests should be developed that start with a zipped up Hadoop 0.12 file system (that is included in Hadoop's src/test directory under version controlled) and attempts to upgrade it to the current version of Hadoop (ie the version that the tests are running against). The zipped up file system should include some "interesting" files, such as:
> - zero length files
> - file with replication set higher than number of datanodes
> - file with no .crc file
> - file with corrupt .crc file
> - file with multiple blocks (will need to set dfs.block.size to a small value)
> - file with multiple checksum blocks
> - empty directory
> - all of the above again but with a different io.bytes.per.checksum setting
> The class that generates the zipped up file system should also be included in this patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.