Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

[anil gupta <an...@buffalo.edu>]

+common-user@hadoop.apache.org

Hi Harsh,

Thanks for the information.
Is there any way to differentiate between a client side property and
server-side property?or a Document which enlists whether a property is
server or client-side? Many times i have to speculate over this and try out
test runs.

Thanks,
Anil

On Fri, Mar 30, 2012 at 9:54 PM, Harsh J <ha...@cloudera.com> wrote:

> Anil,
>
> You can also disable speculative execution on a per-job basis. See
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setMapSpeculativeExecution(boolean)
> (Which is why it is called a client-sided property - it applies
> per-job).
>
> If HBase strongly recommends turning it off, HBase should also, by
> default, turn it off for its own offered jobs?
>
> On Sat, Mar 31, 2012 at 4:02 AM, anil gupta <an...@buffalo.edu> wrote:
> > Hi Doug,
> >
> > Yes, that's why i had set that property as false in my mapred-site.xml.
> > But, to my surprise i didnt know that setting that property would be
> > useless for Hadoop jobs unless the mapred-site.xml is in classpath. The
> > idea of client side property is a little confusing to me at present since
> > there is no proper nomenclature for client side properties at present.
> > Thanks for your reply.
> >
> > ~Anil
> >
> > On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil <
> doug.meil@explorysmedical.com>wrote:
> >
> >>
> >> Speculative execution is on by default in Hadoop.  One of the
> Performance
> >> recommendations in the Hbase RefGuide is to turn it off.
> >>
> >>
> >>
> >>
> >>
> >> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
> >>
> >> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if
> >> >this is listed anywhere, maybe in the book.
> >> >
> >> >BTW usually HBase has a "client" somewhere in the same to indicate
> >> >it's client side.
> >> >
> >> >J-D
> >> >
> >> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <an...@buffalo.edu>
> wrote:
> >> >> Thanks for the quick reply, Jean. Is there any link where i can find
> the
> >> >> name of all client-side configuration for HBase?
> >> >>
> >> >> ~Anil
> >> >>
> >> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans
> >> >><jd...@apache.org>wrote:
> >> >>
> >> >>> This is a client-side configuration so if your mapred-site.xml is
> >> >>> _not_ on your classpath when you start the bulk load, it's not going
> >> >>> to pick it up. So either have that file on your classpath, or put it
> >> >>> in whatever other configuration file you have.
> >> >>>
> >> >>> J-D
> >> >>>
> >> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <an...@gmail.com>
> >> >>>wrote:
> >> >>> > Hi All,
> >> >>> >
> >> >>> > I am using cdh3u2. I ran HBase bulk loading with property
> >> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
> >> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading
> job
> >> >>>and
> >> >>> > after short analysis i realized that these jobs are killed because
> >> >>> another
> >> >>> > worker node completed the task, hence it means that speculative
> >> >>>execution
> >> >>> > is still on. Why the HBase Bulk loader is doing speculative
> execution
> >> >>> when
> >> >>> > i have set it to false in mapred-site.xml? Please let me know if
> i am
> >> >>> > missing something over here.
> >> >>> >
> >> >>> > --
> >> >>> > Thanks & Regards,
> >> >>> > Anil Gupta
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thanks & Regards,
> >> >> Anil Gupta
> >> >
> >>
> >>
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
>
> --
> Harsh J
>



-- 
Thanks & Regards,
Anil Gupta