[jira] [Updated] (TEZ-3215) Support for MultipleOutputs

["Ming Ma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ming Ma updated TEZ-3215:
-------------------------
    Attachment: TEZ-3215.patch

Here is the draft patch. It provides multiple output files support as long as they share the same output format. We can add support for different output formats just like how MR MultipleOutputs later as a separate patch.

Another alternative way is to have MROutputs use MR MultipleOutputs. But given the extra API dependencies and the fact that MR MultipleOutputs' implementation is straightforward, the current approach appears to be simpler.



> Support for MultipleOutputs
> ---------------------------
>
>                 Key: TEZ-3215
>                 URL: https://issues.apache.org/jira/browse/TEZ-3215
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Ming Ma
>         Attachments: TEZ-3215.patch
>
>
> Here is the use case. A reducer might write its output to more than one file. The file name will be based on the mapper key. We don't know all possible keys ahead of time. In MR, MultipleOutputs provides such support. I couldn't find anything readily available in Tez.
> * Set up one DataSink per file ahead of time won't work as we don't know all possible keys.
> * Use MR MultipleOutputs directly from the Tez application processor. It isn't clear how to pass TaskInputOutputContext to MultipleOutputs.
> * Tez MROutput can create a DataSink based on the specified outputFormat. But it can't take MR MultipleOutputs.
> I end up modifying Tez MROutput with HashMap {{recordWriters}} to achieve this. If this is a solved problem, can anyone explain how to do it?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)