You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crunch.apache.org by "Suyash Agarwal (JIRA)" <ji...@apache.org> on 2019/07/03 03:15:00 UTC

[jira] [Commented] (CRUNCH-635) MultipleOutput Text Targets

    [ https://issues.apache.org/jira/browse/CRUNCH-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877448#comment-16877448 ] 

Suyash Agarwal commented on CRUNCH-635:
---------------------------------------

Attached the patch for the implementation of path per key target for Text.
I've successfully tried multiple nested sub-directories as the keys and compression also works.
I was getting AlreadyBeingCreatedException by another attempt while trying to append to an existing file (across task failures). So I've used committer to get the work path.
There were cases where a single key had too much data and reducer was taking a lot of time and was creating a single huge file so I tried writing per key output from mappers and it's working fine. I thought of maintaining a map of key and record writer under TextPathPerKeyRecordWriter to avoid creating a writer everytime a new key is encountered (especially in case of mappers) as done in [MultipleOutputs|[https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java#L383|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java#L383].]] because of the possibility of having many keys resulting in a huge map causing OOM. Not sure if MultipleOutputs will also run into the same issue. If the approach makes sense, I can implement that.

> MultipleOutput Text Targets
> ---------------------------
>
>                 Key: CRUNCH-635
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-635
>             Project: Crunch
>          Issue Type: Improvement
>          Components: IO
>            Reporter: Jim McStanton
>            Priority: Minor
>         Attachments: CRUNCH-635.patch
>
>
> CRUNCH-306 was logged to add the ability to write an output file per key in a PTable. The original issue covers just Avro output files, this Improvement would extend this by adding support for Text file targets. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)