You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by lu...@sohu.com on 2010/06/24 03:23:32 UTC

How can I skip error record in hive?

How can I skip error record in hive.
because when there is some error record in hive table,the hadoop job always fail!
Is there some configuration to avoid this?
Hope for your reply!

Re: How can I skip error record in hive?

Posted by Ted Xu <te...@gmail.com>.
Hi luocan,

Please try

set hive.groupby.*skewindata*=true;

2010/6/24 <lu...@sohu.com>

>
> thanks very much!
>
> it success!
>
> when I use distinct,for example "select count(distinct(commid)) from
> t_dw_comm2pos",
>
> the number of the readucers is only one.
>
> Is there some method to optimize it?
>
>  ----- 原文 -----
>
> *发件人:* Ted Xu
> *主 题:* Re: How can I skip error record in hive?
> *时 间:* 2010年6月24日 9:43:00
>
>
> Hi,
>
> Skipping bad record feature is provided by Hadoop, please refer to
> Hadoop tutorial
> http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Skipping+Bad+Records
> .
>
> Those settings with Java APIs are also configurable in Hive, like:
>
> set mapreduce.map.skip.maxrecords=10;
>
>
> 2010/6/24 <lu...@sohu.com>
>
> How can I skip error record in hive.
>
> because when there is some error record in hive table,the hadoop job always
> fail!
>
> Is there some configuration to avoid this?
>
> Hope for your reply!
>
>
>
>
> --
> Best Regards,
> Ted Xu
>
>


-- 
Best Regards,
Ted Xu

Re: How can I skip error record in hive?

Posted by lu...@sohu.com.
thanks very much!
it success!
when I use distinct,for example "select count(distinct(commid)) from t_dw_comm2pos",
the number of the readucers is only one.
Is there some method to optimize it?



----- 原文 ----- 发件人: Ted Xu 主 题: Re: How can I skip error record in hive?时 间: 2010年6月24日 9:43:00
Hi, 

Skipping bad record feature is provided by Hadoop, please refer to Hadoop&nbsp;tutorial&nbsp;http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Skipping+Bad+Records.

Those settings with Java APIs are also configurable in Hive, like:

set&nbsp;mapreduce.map.skip.maxrecords=10;


2010/6/24 &lt;luocan19826164@sohu.com&gt;

How can&nbsp;I skip error record in hive.
because when there is some error record in hive table,the hadoop job always fail!
Is there some configuration to avoid this?
Hope for your reply!-- Best Regards,Ted Xu

Re: How can I skip error record in hive?

Posted by Ted Xu <te...@gmail.com>.
Hi,

Skipping bad record feature is provided by Hadoop, please refer to
Hadoop tutorial
http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Skipping+Bad+Records
.

Those settings with Java APIs are also configurable in Hive, like:

set mapreduce.map.skip.maxrecords=10;


2010/6/24 <lu...@sohu.com>

> How can I skip error record in hive.
>
> because when there is some error record in hive table,the hadoop job always
> fail!
>
> Is there some configuration to avoid this?
>
> Hope for your reply!
>



-- 
Best Regards,
Ted Xu