[jira] [Commented] (HDFS-4732) TestDFSUpgradeFromImage fails on Windows due to failure to unpack old image tarball that contains hard links

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639217#comment-13639217 ] 

Chris Nauroth commented on HDFS-4732:
-------------------------------------

There are 2 components to this patch:

# I'm providing a new copy of hadoop-22-dfs-dir.tgz, attached to this issue.  It's equivalent to the old one, but does not contain links.  As a side effect, the file is bigger, because bytes are duplicated where previously they were shared via hard links.  Before: ~170K compressed, ~40 MB uncompressed.  After: ~310K compressed, ~80 MB uncompressed.  The compressed version is what's stored in the repo, so this doesn't really make the source tree much bigger.
# While investigating, I noticed a code path in {{TestDFSUpgradeFromImage}} where we could fail to shutdown a {{MiniDFSCluster}}.  In practice, this isn't an issue, but I'd like to clean it up.

To the committer: please download and commit hadoop-22-dfs-dir.tgz independent of the patch file.  It's a binary file, so I couldn't include it in the patch file.  It needs to go in hadoop-hdfs-project/hadoop-hdfs/src/test/resources and overwrite the existing file.

I tested this successfully on Mac (bsdtar), Ubuntu (GNU tar), and Windows (commons-compress).

                
> TestDFSUpgradeFromImage fails on Windows due to failure to unpack old image tarball that contains hard links
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4732
>                 URL: https://issues.apache.org/jira/browse/HDFS-4732
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 3.0.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: hadoop-22-dfs-dir.tgz
>
>
> On non-Windows, {{FileUtil#unTar}} is implemented using external Unix shell commands.  On Windows, {{FileUtil#unTar}} is implemented in Java using commons-compress.  {{TestDFSUpgradeFromImage}} uses a testing tarball image of an old HDFS layout version, hadoop-22-dfs-dir.tgz.  This file contains hard links.  It appears that commons-compress cannot handle the hard links correctly.  When it unpacks the file, each hard link ends up as a 0-length file.  This causes the test to fail during cluster startup, because the 0-length block files are considered corrupt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira