[jira] [Created] (MAPREDUCE-3276) hadoop dfs -copyToLocal/copyFromLocal called within Hadoop Streaming returns early

["Keith Stevens (Created) (JIRA)" <ji...@apache.org>]

hadoop dfs -copyToLocal/copyFromLocal called within Hadoop Streaming returns early
----------------------------------------------------------------------------------

                 Key: MAPREDUCE-3276
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3276
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: contrib/streaming
    Affects Versions: 0.20.2
         Environment: Linux RedHat Enterprise Linux 5.
31 node cluster with 1 as JobTracker and NameNode, and 30 as TaskTracker and DataNode.
            Reporter: Keith Stevens


I'm using the Cloudera hadoop realease 0.20.2.+737 to parallelize bash scripts with Hadoop Streaming.

Below is an example script that i've been running which simply copies a file from hdfs to a local node.
{code:title=SampleMapper.sh|borderStyle=solid}
 hadoop dfs -copyToLocal /path/to/some/large/file/myFile myFile
 # Spin until the file is fully copied.
 while [ ! -f myFile ]
 do 
  echo "spin"
  sleep 1 
 done
{code}

Surprisingly, the copy call returns before the file is copied, if the file is sufficiently large, and the while loop spins for several iterations.  I'm seeing similar behavior with copyFromLocal.

I've asked about this issue on other forms and no one else seems to have had the problem, although I don't know how many peoplpe are attempting to do this particular task.

Has this been fixed in more recent versions of hadoop?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira