You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by rab ra <ra...@gmail.com> on 2014/07/10 07:42:54 UTC
Muliple map writing into same hdfs file
hello
I have one use-case that spans multiple map tasks in hadoop environment. I
use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
into a file stored in hdfs. This file is shared across all the map tasks.
Though, they all computes thier output but some of them are missing in the
output file.
The output file is an excel file with 8 parameters(headings). Each map task
is supposed to compute all these 8 values, and save it as soon as it is
computed. This means, the programming logic of a map task opens the file,
writes the value and close, 8 times.
Can someone give me a hint on whats going wrong here?
Is it possible to make more than one map task to write in a shared file in
HDFS?
Re: Muliple map writing into same hdfs file
Posted by rab ra <ra...@gmail.com>.
Hello ,
I finally moved to per task write and reducers gather them all and write
them into the file
Thanks for the help
regards
rab
On Fri, Jul 11, 2014 at 10:50 AM, Bertrand Dechoux <de...@gmail.com>
wrote:
> And beside with a single file, if that were possible, how do you handle
> error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
> success... A per-task file has been a standard to easily solve that
> problem.
>
> Bertrand Dechoux
>
>
> On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> Current writes to a single file in HDFS is not possible today. You may
>> want to write a per-task file and use that entire directory as your output.
>>
>> +Vinod
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>>
>>>
>>> hello
>>>
>>>
>>>
>>> I have one use-case that spans multiple map tasks in hadoop environment.
>>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>>> into a file stored in hdfs. This file is shared across all the map tasks.
>>> Though, they all computes thier output but some of them are missing in the
>>> output file.
>>>
>>>
>>>
>>> The output file is an excel file with 8 parameters(headings). Each map
>>> task is supposed to compute all these 8 values, and save it as soon as it
>>> is computed. This means, the programming logic of a map task opens the
>>> file, writes the value and close, 8 times.
>>>
>>>
>>>
>>> Can someone give me a hint on whats going wrong here?
>>>
>>>
>>>
>>> Is it possible to make more than one map task to write in a shared file
>>> in HDFS?
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
Re: Muliple map writing into same hdfs file
Posted by rab ra <ra...@gmail.com>.
Hello ,
I finally moved to per task write and reducers gather them all and write
them into the file
Thanks for the help
regards
rab
On Fri, Jul 11, 2014 at 10:50 AM, Bertrand Dechoux <de...@gmail.com>
wrote:
> And beside with a single file, if that were possible, how do you handle
> error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
> success... A per-task file has been a standard to easily solve that
> problem.
>
> Bertrand Dechoux
>
>
> On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> Current writes to a single file in HDFS is not possible today. You may
>> want to write a per-task file and use that entire directory as your output.
>>
>> +Vinod
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>>
>>>
>>> hello
>>>
>>>
>>>
>>> I have one use-case that spans multiple map tasks in hadoop environment.
>>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>>> into a file stored in hdfs. This file is shared across all the map tasks.
>>> Though, they all computes thier output but some of them are missing in the
>>> output file.
>>>
>>>
>>>
>>> The output file is an excel file with 8 parameters(headings). Each map
>>> task is supposed to compute all these 8 values, and save it as soon as it
>>> is computed. This means, the programming logic of a map task opens the
>>> file, writes the value and close, 8 times.
>>>
>>>
>>>
>>> Can someone give me a hint on whats going wrong here?
>>>
>>>
>>>
>>> Is it possible to make more than one map task to write in a shared file
>>> in HDFS?
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
Re: Muliple map writing into same hdfs file
Posted by rab ra <ra...@gmail.com>.
Hello ,
I finally moved to per task write and reducers gather them all and write
them into the file
Thanks for the help
regards
rab
On Fri, Jul 11, 2014 at 10:50 AM, Bertrand Dechoux <de...@gmail.com>
wrote:
> And beside with a single file, if that were possible, how do you handle
> error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
> success... A per-task file has been a standard to easily solve that
> problem.
>
> Bertrand Dechoux
>
>
> On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> Current writes to a single file in HDFS is not possible today. You may
>> want to write a per-task file and use that entire directory as your output.
>>
>> +Vinod
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>>
>>>
>>> hello
>>>
>>>
>>>
>>> I have one use-case that spans multiple map tasks in hadoop environment.
>>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>>> into a file stored in hdfs. This file is shared across all the map tasks.
>>> Though, they all computes thier output but some of them are missing in the
>>> output file.
>>>
>>>
>>>
>>> The output file is an excel file with 8 parameters(headings). Each map
>>> task is supposed to compute all these 8 values, and save it as soon as it
>>> is computed. This means, the programming logic of a map task opens the
>>> file, writes the value and close, 8 times.
>>>
>>>
>>>
>>> Can someone give me a hint on whats going wrong here?
>>>
>>>
>>>
>>> Is it possible to make more than one map task to write in a shared file
>>> in HDFS?
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
Re: Muliple map writing into same hdfs file
Posted by rab ra <ra...@gmail.com>.
Hello ,
I finally moved to per task write and reducers gather them all and write
them into the file
Thanks for the help
regards
rab
On Fri, Jul 11, 2014 at 10:50 AM, Bertrand Dechoux <de...@gmail.com>
wrote:
> And beside with a single file, if that were possible, how do you handle
> error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
> success... A per-task file has been a standard to easily solve that
> problem.
>
> Bertrand Dechoux
>
>
> On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> Current writes to a single file in HDFS is not possible today. You may
>> want to write a per-task file and use that entire directory as your output.
>>
>> +Vinod
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>>
>>>
>>> hello
>>>
>>>
>>>
>>> I have one use-case that spans multiple map tasks in hadoop environment.
>>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>>> into a file stored in hdfs. This file is shared across all the map tasks.
>>> Though, they all computes thier output but some of them are missing in the
>>> output file.
>>>
>>>
>>>
>>> The output file is an excel file with 8 parameters(headings). Each map
>>> task is supposed to compute all these 8 values, and save it as soon as it
>>> is computed. This means, the programming logic of a map task opens the
>>> file, writes the value and close, 8 times.
>>>
>>>
>>>
>>> Can someone give me a hint on whats going wrong here?
>>>
>>>
>>>
>>> Is it possible to make more than one map task to write in a shared file
>>> in HDFS?
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
Re: Muliple map writing into same hdfs file
Posted by Bertrand Dechoux <de...@gmail.com>.
And beside with a single file, if that were possible, how do you handle
error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
success... A per-task file has been a standard to easily solve that problem.
Bertrand Dechoux
On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> Current writes to a single file in HDFS is not possible today. You may
> want to write a per-task file and use that entire directory as your output.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
>>
>> hello
>>
>>
>>
>> I have one use-case that spans multiple map tasks in hadoop environment.
>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>> into a file stored in hdfs. This file is shared across all the map tasks.
>> Though, they all computes thier output but some of them are missing in the
>> output file.
>>
>>
>>
>> The output file is an excel file with 8 parameters(headings). Each map
>> task is supposed to compute all these 8 values, and save it as soon as it
>> is computed. This means, the programming logic of a map task opens the
>> file, writes the value and close, 8 times.
>>
>>
>>
>> Can someone give me a hint on whats going wrong here?
>>
>>
>>
>> Is it possible to make more than one map task to write in a shared file
>> in HDFS?
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Muliple map writing into same hdfs file
Posted by Bertrand Dechoux <de...@gmail.com>.
And beside with a single file, if that were possible, how do you handle
error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
success... A per-task file has been a standard to easily solve that problem.
Bertrand Dechoux
On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> Current writes to a single file in HDFS is not possible today. You may
> want to write a per-task file and use that entire directory as your output.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
>>
>> hello
>>
>>
>>
>> I have one use-case that spans multiple map tasks in hadoop environment.
>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>> into a file stored in hdfs. This file is shared across all the map tasks.
>> Though, they all computes thier output but some of them are missing in the
>> output file.
>>
>>
>>
>> The output file is an excel file with 8 parameters(headings). Each map
>> task is supposed to compute all these 8 values, and save it as soon as it
>> is computed. This means, the programming logic of a map task opens the
>> file, writes the value and close, 8 times.
>>
>>
>>
>> Can someone give me a hint on whats going wrong here?
>>
>>
>>
>> Is it possible to make more than one map task to write in a shared file
>> in HDFS?
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Muliple map writing into same hdfs file
Posted by Bertrand Dechoux <de...@gmail.com>.
And beside with a single file, if that were possible, how do you handle
error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
success... A per-task file has been a standard to easily solve that problem.
Bertrand Dechoux
On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> Current writes to a single file in HDFS is not possible today. You may
> want to write a per-task file and use that entire directory as your output.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
>>
>> hello
>>
>>
>>
>> I have one use-case that spans multiple map tasks in hadoop environment.
>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>> into a file stored in hdfs. This file is shared across all the map tasks.
>> Though, they all computes thier output but some of them are missing in the
>> output file.
>>
>>
>>
>> The output file is an excel file with 8 parameters(headings). Each map
>> task is supposed to compute all these 8 values, and save it as soon as it
>> is computed. This means, the programming logic of a map task opens the
>> file, writes the value and close, 8 times.
>>
>>
>>
>> Can someone give me a hint on whats going wrong here?
>>
>>
>>
>> Is it possible to make more than one map task to write in a shared file
>> in HDFS?
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Muliple map writing into same hdfs file
Posted by Bertrand Dechoux <de...@gmail.com>.
And beside with a single file, if that were possible, how do you handle
error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
success... A per-task file has been a standard to easily solve that problem.
Bertrand Dechoux
On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> Current writes to a single file in HDFS is not possible today. You may
> want to write a per-task file and use that entire directory as your output.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
>>
>> hello
>>
>>
>>
>> I have one use-case that spans multiple map tasks in hadoop environment.
>> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
>> into a file stored in hdfs. This file is shared across all the map tasks.
>> Though, they all computes thier output but some of them are missing in the
>> output file.
>>
>>
>>
>> The output file is an excel file with 8 parameters(headings). Each map
>> task is supposed to compute all these 8 values, and save it as soon as it
>> is computed. This means, the programming logic of a map task opens the
>> file, writes the value and close, 8 times.
>>
>>
>>
>> Can someone give me a hint on whats going wrong here?
>>
>>
>>
>> Is it possible to make more than one map task to write in a shared file
>> in HDFS?
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Muliple map writing into same hdfs file
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Current writes to a single file in HDFS is not possible today. You may
want to write a per-task file and use that entire directory as your output.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
> hello
>
>
>
> I have one use-case that spans multiple map tasks in hadoop environment. I
> use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
> into a file stored in hdfs. This file is shared across all the map tasks.
> Though, they all computes thier output but some of them are missing in the
> output file.
>
>
>
> The output file is an excel file with 8 parameters(headings). Each map
> task is supposed to compute all these 8 values, and save it as soon as it
> is computed. This means, the programming logic of a map task opens the
> file, writes the value and close, 8 times.
>
>
>
> Can someone give me a hint on whats going wrong here?
>
>
>
> Is it possible to make more than one map task to write in a shared file in
> HDFS?
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Muliple map writing into same hdfs file
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Current writes to a single file in HDFS is not possible today. You may
want to write a per-task file and use that entire directory as your output.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
> hello
>
>
>
> I have one use-case that spans multiple map tasks in hadoop environment. I
> use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
> into a file stored in hdfs. This file is shared across all the map tasks.
> Though, they all computes thier output but some of them are missing in the
> output file.
>
>
>
> The output file is an excel file with 8 parameters(headings). Each map
> task is supposed to compute all these 8 values, and save it as soon as it
> is computed. This means, the programming logic of a map task opens the
> file, writes the value and close, 8 times.
>
>
>
> Can someone give me a hint on whats going wrong here?
>
>
>
> Is it possible to make more than one map task to write in a shared file in
> HDFS?
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Muliple map writing into same hdfs file
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Current writes to a single file in HDFS is not possible today. You may
want to write a per-task file and use that entire directory as your output.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
> hello
>
>
>
> I have one use-case that spans multiple map tasks in hadoop environment. I
> use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
> into a file stored in hdfs. This file is shared across all the map tasks.
> Though, they all computes thier output but some of them are missing in the
> output file.
>
>
>
> The output file is an excel file with 8 parameters(headings). Each map
> task is supposed to compute all these 8 values, and save it as soon as it
> is computed. This means, the programming logic of a map task opens the
> file, writes the value and close, 8 times.
>
>
>
> Can someone give me a hint on whats going wrong here?
>
>
>
> Is it possible to make more than one map task to write in a shared file in
> HDFS?
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Muliple map writing into same hdfs file
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Current writes to a single file in HDFS is not possible today. You may
want to write a per-task file and use that entire directory as your output.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Wed, Jul 9, 2014 at 10:42 PM, rab ra <ra...@gmail.com> wrote:
>
> hello
>
>
>
> I have one use-case that spans multiple map tasks in hadoop environment. I
> use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
> into a file stored in hdfs. This file is shared across all the map tasks.
> Though, they all computes thier output but some of them are missing in the
> output file.
>
>
>
> The output file is an excel file with 8 parameters(headings). Each map
> task is supposed to compute all these 8 values, and save it as soon as it
> is computed. This means, the programming logic of a map task opens the
> file, writes the value and close, 8 times.
>
>
>
> Can someone give me a hint on whats going wrong here?
>
>
>
> Is it possible to make more than one map task to write in a shared file in
> HDFS?
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.