You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2009/01/20 19:48:59 UTC
[jira] Created: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Copying a file to local with Crc throws an exception
----------------------------------------------------
Key: HADOOP-5085
URL: https://issues.apache.org/jira/browse/HADOOP-5085
Project: Hadoop Core
Issue Type: Bug
Components: fs
Reporter: Hairong Kuang
Fix For: 0.21.0
$ hadoop dfs -get -crc /user/aa/test.txt test.txt
get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
org.apache.hadoop.fs.ChecksumFileSystem
The problem seems to be caused by the line 251 in FsShell#copyToLocal:
{noformat}
250; if (copyCrc) {
251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
...
}
{noformat}
Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668691#action_12668691 ]
Hairong Kuang commented on HADOOP-5085:
---------------------------------------
My solution is consistent with how FsShell handles -crc option now. If the option is specified and stdout is the destination, the code also prints a warning and continue. I think it is no harm to turn off -crc option automatically.
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: crcChecksum.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668685#action_12668685 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5085:
------------------------------------------------
When -crc is specified in the command and the fs does not support crc, the patch prints a warning, turns off -crc automatically and then continues. I think we should throw an exception in such case. No?
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: crcChecksum.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-5085:
----------------------------------
Fix Version/s: (was: 0.21.0)
0.20.0
Status: Patch Available (was: Open)
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.20.0
>
> Attachments: crcChecksum.patch, crcChecksum1.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665546#action_12665546 ]
Raghu Angadi commented on HADOOP-5085:
--------------------------------------
>Copying crc files to local should not require the source file system to be ChecksumFileSystem.
"crc file" applies only to a ChecksumFileSystem. HDFS does not have a checksum file..
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Fix For: 0.21.0
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-5085:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this.
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.20.0
>
> Attachments: crcChecksum.patch, crcChecksum1.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673997#action_12673997 ]
Hudson commented on HADOOP-5085:
--------------------------------
Integrated in Hadoop-trunk #756 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/])
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.20.0
>
> Attachments: crcChecksum.patch, crcChecksum1.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669784#action_12669784 ]
Hadoop QA commented on HADOOP-5085:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12399284/crcChecksum1.patch
against trunk revision 740064.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3789/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3789/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3789/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3789/console
This message is automatically generated.
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.20.0
>
> Attachments: crcChecksum.patch, crcChecksum1.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-5085:
----------------------------------
Attachment: crcChecksum.patch
This patch gives a warning if a user specifies -crc option when source file system does not have .crc file.
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: crcChecksum.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang reassigned HADOOP-5085:
-------------------------------------
Assignee: Hairong Kuang
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: crcChecksum.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5085:
-------------------------------------------
Hadoop Flags: [Reviewed]
Ok, let's make it consistent with the stdout case.
+1
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: crcChecksum.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5085) Copying a file to local with Crc
throws an exception
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-5085:
----------------------------------
Attachment: crcChecksum1.patch
Thanks Nicholas for the review comments. This patch explicitly states that crc option is turned off automatically. Just to make the user to be better informed.
> Copying a file to local with Crc throws an exception
> ----------------------------------------------------
>
> Key: HADOOP-5085
> URL: https://issues.apache.org/jira/browse/HADOOP-5085
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.20.0
>
> Attachments: crcChecksum.patch, crcChecksum1.patch
>
>
> $ hadoop dfs -get -crc /user/aa/test.txt test.txt
> get: org.apache.hadoop.dfs.DistributedFileSystem cannot be cast to
> org.apache.hadoop.fs.ChecksumFileSystem
> The problem seems to be caused by the line 251 in FsShell#copyToLocal:
> {noformat}
> 250; if (copyCrc) {
> 251: ChecksumFileSystem csfs = (ChecksumFileSystem) srcFS;
> ...
> }
> {noformat}
> Copying crc files to local should not require the source file system to be ChecksumFileSystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.