You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by AnilKumar B <ak...@gmail.com> on 2012/11/21 15:20:26 UTC
When speculative execution is true, there is a data loss issue with multpleoutputs
Hi,
We have observed some data loss when we enabled speculative execution with
multipleoutputs. But when we disabled speculative execution,
multipleOutputs working fine.
I am also trying to find the issue, but meanwhile want to know, If anyone
faced similar issue and what could be the problem?
We are using very old version 0.20.203.
Thanks,
B Anil Kumar.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
Dne 21.11.2012 16:07, AnilKumar B napsal(a):
> Thanks Radim.
>
> Yes, as you said we are not writing into sub-directory of main job. I
> will try by making them as sub-directories of output dir.
>
> But one question, when I turn of speculative execution then it is
> working fine with same multiple output directory structure. May I
> know, how exactly it working in this case?
>
> When we change the speculative execution flag, why exactly there is a
> difference in output data?
because if you are not using multipleoutput then you are not writing to
real file, but to file with name generated from its task attempt in tmp
subdirectory. They do not overwrite each other. In HDFS you can have
only one writer per file.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
Dne 21.11.2012 16:07, AnilKumar B napsal(a):
> Thanks Radim.
>
> Yes, as you said we are not writing into sub-directory of main job. I
> will try by making them as sub-directories of output dir.
>
> But one question, when I turn of speculative execution then it is
> working fine with same multiple output directory structure. May I
> know, how exactly it working in this case?
>
> When we change the speculative execution flag, why exactly there is a
> difference in output data?
because if you are not using multipleoutput then you are not writing to
real file, but to file with name generated from its task attempt in tmp
subdirectory. They do not overwrite each other. In HDFS you can have
only one writer per file.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
Dne 21.11.2012 16:07, AnilKumar B napsal(a):
> Thanks Radim.
>
> Yes, as you said we are not writing into sub-directory of main job. I
> will try by making them as sub-directories of output dir.
>
> But one question, when I turn of speculative execution then it is
> working fine with same multiple output directory structure. May I
> know, how exactly it working in this case?
>
> When we change the speculative execution flag, why exactly there is a
> difference in output data?
because if you are not using multipleoutput then you are not writing to
real file, but to file with name generated from its task attempt in tmp
subdirectory. They do not overwrite each other. In HDFS you can have
only one writer per file.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
Dne 21.11.2012 16:07, AnilKumar B napsal(a):
> Thanks Radim.
>
> Yes, as you said we are not writing into sub-directory of main job. I
> will try by making them as sub-directories of output dir.
>
> But one question, when I turn of speculative execution then it is
> working fine with same multiple output directory structure. May I
> know, how exactly it working in this case?
>
> When we change the speculative execution flag, why exactly there is a
> difference in output data?
because if you are not using multipleoutput then you are not writing to
real file, but to file with name generated from its task attempt in tmp
subdirectory. They do not overwrite each other. In HDFS you can have
only one writer per file.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
this is another problem with fileoutputformat committer, its related to
your.
https://issues.apache.org/jira/browse/MAPREDUCE-3772
it works like this: if multipleoutput is relative to job output, then
there is a workaround to make it work with commiter and outputs from
multiple tasks do not clash with each other, problem mentioned in ticket
cheats that relative vs absolute output path detection and all output is
lost on task commit.
But if output is absolute path, then its written directly to output file
which fails because writers from multiple attempts crash together.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
this is another problem with fileoutputformat committer, its related to
your.
https://issues.apache.org/jira/browse/MAPREDUCE-3772
it works like this: if multipleoutput is relative to job output, then
there is a workaround to make it work with commiter and outputs from
multiple tasks do not clash with each other, problem mentioned in ticket
cheats that relative vs absolute output path detection and all output is
lost on task commit.
But if output is absolute path, then its written directly to output file
which fails because writers from multiple attempts crash together.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
this is another problem with fileoutputformat committer, its related to
your.
https://issues.apache.org/jira/browse/MAPREDUCE-3772
it works like this: if multipleoutput is relative to job output, then
there is a workaround to make it work with commiter and outputs from
multiple tasks do not clash with each other, problem mentioned in ticket
cheats that relative vs absolute output path detection and all output is
lost on task commit.
But if output is absolute path, then its written directly to output file
which fails because writers from multiple attempts crash together.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
this is another problem with fileoutputformat committer, its related to
your.
https://issues.apache.org/jira/browse/MAPREDUCE-3772
it works like this: if multipleoutput is relative to job output, then
there is a workaround to make it work with commiter and outputs from
multiple tasks do not clash with each other, problem mentioned in ticket
cheats that relative vs absolute output path detection and all output is
lost on task commit.
But if output is absolute path, then its written directly to output file
which fails because writers from multiple attempts crash together.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Radim.
Yes, as you said we are not writing into sub-directory of main job. I will
try by making them as sub-directories of output dir.
But one question, when I turn of speculative execution then it is working
fine with same multiple output directory structure. May I know, how exactly
it working in this case?
When we change the speculative execution flag, why exactly there is a
difference in output data?
Thanks,
B Anil Kumar.
On Wed, Nov 21, 2012 at 8:01 PM, Radim Kolar <hs...@filez.com> wrote:
> its not data loss, problem is caused that multipleoutputs do not work with
> standard committer if you do not write into subdirectory of main job output.
>
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Radim.
Yes, as you said we are not writing into sub-directory of main job. I will
try by making them as sub-directories of output dir.
But one question, when I turn of speculative execution then it is working
fine with same multiple output directory structure. May I know, how exactly
it working in this case?
When we change the speculative execution flag, why exactly there is a
difference in output data?
Thanks,
B Anil Kumar.
On Wed, Nov 21, 2012 at 8:01 PM, Radim Kolar <hs...@filez.com> wrote:
> its not data loss, problem is caused that multipleoutputs do not work with
> standard committer if you do not write into subdirectory of main job output.
>
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Radim.
Yes, as you said we are not writing into sub-directory of main job. I will
try by making them as sub-directories of output dir.
But one question, when I turn of speculative execution then it is working
fine with same multiple output directory structure. May I know, how exactly
it working in this case?
When we change the speculative execution flag, why exactly there is a
difference in output data?
Thanks,
B Anil Kumar.
On Wed, Nov 21, 2012 at 8:01 PM, Radim Kolar <hs...@filez.com> wrote:
> its not data loss, problem is caused that multipleoutputs do not work with
> standard committer if you do not write into subdirectory of main job output.
>
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Radim.
Yes, as you said we are not writing into sub-directory of main job. I will
try by making them as sub-directories of output dir.
But one question, when I turn of speculative execution then it is working
fine with same multiple output directory structure. May I know, how exactly
it working in this case?
When we change the speculative execution flag, why exactly there is a
difference in output data?
Thanks,
B Anil Kumar.
On Wed, Nov 21, 2012 at 8:01 PM, Radim Kolar <hs...@filez.com> wrote:
> its not data loss, problem is caused that multipleoutputs do not work with
> standard committer if you do not write into subdirectory of main job output.
>
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
its not data loss, problem is caused that multipleoutputs do not work
with standard committer if you do not write into subdirectory of main
job output.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
its not data loss, problem is caused that multipleoutputs do not work
with standard committer if you do not write into subdirectory of main
job output.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
its not data loss, problem is caused that multipleoutputs do not work
with standard committer if you do not write into subdirectory of main
job output.
Re: When speculative execution is true, there is a data loss issue
with multpleoutputs
Posted by Radim Kolar <hs...@filez.com>.
its not data loss, problem is caused that multipleoutputs do not work
with standard committer if you do not write into subdirectory of main
job output.