[jira] [Created] (PIG-4412) Race condition in writing multiple outputs from STREAM op

["Josh Wills (JIRA)" <ji...@apache.org>]

Josh Wills created PIG-4412:
-------------------------------

             Summary: Race condition in writing multiple outputs from STREAM op
                 Key: PIG-4412
                 URL: https://issues.apache.org/jira/browse/PIG-4412
             Project: Pig
          Issue Type: Bug
          Components: impl
            Reporter: Josh Wills


Basically copying the issue described here:

http://stackoverflow.com/questions/28327044/pig-streaming-some-output-files-are-missing

Roughly, I believe the issue is that there is a race condition in the code in the HadoopExecutableManager that moves multiple output files from a script into HDFS and the MapReduce task that is shutting down after it writes the last bits from the "main" output of the STREAM task. Pig needs to make sure that the ExecutableManager is closed (and thus the files are moved from the local dir to HDFS) before it returns the end-of-stream tuple to signal that the stream is finished.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)