Re: Muliple map writing into same hdfs file

[rab ra <ra...@gmail.com>]

Hello ,


I finally moved to per task write and reducers gather them all and write
them into the file

Thanks for the help


regards
rab


On Fri, Jul 11, 2014 at 10:50 AM, Bertrand Dechoux <de...@gmail.com>
wrote:

> And beside with a single file, if that were possible, how do you handle
> error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
> success... A per-task file has been a standard to easily solve that
> problem.
>
> Bertrand Dechoux
>
>
> On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> Current writes to a single file in HDFS is not possible today. You  may
>> want to write a per-task file and use that entire directory as your output.
>>
>> +Vinod
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>>
>>>
>>> hello
>>>
>>>
>>>
>>> I have one use-case that spans multiple map tasks in hadoop environment.
>>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>>> into a file stored in hdfs. This file is shared across all the map tasks.
>>> Though, they all computes thier output but some of them are missing in the
>>> output file.
>>>
>>>
>>>
>>> The output file is an excel file with 8 parameters(headings). Each map
>>> task is supposed to compute all these 8 values, and save it as soon as it
>>> is computed. This means, the programming logic of a map task opens the
>>> file, writes the value and close, 8 times.
>>>
>>>
>>>
>>> Can someone give me a hint on whats going wrong here?
>>>
>>>
>>>
>>> Is it possible to make more than one map task to write in a shared file
>>> in HDFS?
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>