[jira] [Resolved] (MAPREDUCE-3276) hadoop dfs -copyToLocal/copyFromLocal called within Hadoop Streaming returns early

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer resolved MAPREDUCE-3276.
-----------------------------------------
    Resolution: Later

This issue is pretty stale at this point.  Closing with later.  if it is still a problem, then please open a new jira.

> hadoop dfs -copyToLocal/copyFromLocal called within Hadoop Streaming returns early
> ----------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3276
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3276
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/streaming
>    Affects Versions: 0.20.2
>         Environment: Linux RedHat Enterprise Linux 5.
> 31 node cluster with 1 as JobTracker and NameNode, and 30 as TaskTracker and DataNode.
>            Reporter: Keith Stevens
>              Labels: hadoop, shell, streaming
>
> I'm using the Cloudera hadoop realease 0.20.2.+737 to parallelize bash scripts with Hadoop Streaming.
> Below is an example script that i've been running which simply copies a file from hdfs to a local node.
> {code:title=SampleMapper.sh|borderStyle=solid}
>  hadoop dfs -copyToLocal /path/to/some/large/file/myFile myFile
>  # Spin until the file is fully copied.
>  while [ ! -f myFile ]
>  do 
>   echo "spin"
>   sleep 1 
>  done
> {code}
> Surprisingly, the copy call returns before the file is copied, if the file is sufficiently large, and the while loop spins for several iterations.  I'm seeing similar behavior with copyFromLocal.
> I've asked about this issue on other forms and no one else seems to have had the problem, although I don't know how many peoplpe are attempting to do this particular task.
> Has this been fixed in more recent versions of hadoop?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)