How can i realize the “count(distinct )” function in hive ?

[1983 ddi <dd...@gmail.com>]

Hi all:
     I am trying to develop a function like "count ( distinct )" in hive ,
so here I  am trying to write a  UDAF which using a HashMap container to
store all the keys.

and at last ,It is expected to get the key size by calling map.size() in my
UDAF class , after which I will get the result like "count(distinct)" .

by I  am  confused about how can I write the UDAF class, is there anybody
who can give me a favor and thanks a lot if there is an example .

Re: How can i realize the “count(distinct )” function in hive ?

[Harsh J <qw...@gmail.com>]

You don't really need to store all incoming keys. If the input comes
sorted, you can rely on matching every two values and incrementing the
count accordingly (If you do it in the reduce side it comes sorted by
the key, so for non-distinct keys, you would have more than one value;
thus all you need to do is count all reduce calls as the grouping does
the rest). Just a suggestion to avoid possible memory issues. Correct
me if am wrong, please.

On Mon, Dec 13, 2010 at 5:36 PM, 1983 ddi <dd...@gmail.com> wrote:
> by I  am  confused about how can I write the UDAF class, is there anybody
> who can give me a favor and thanks a lot if there is an example .

About UDFs, read this developer article at Bizo that covers it well
enough: http://dev.bizo.com/2009/06/custom-udfs-and-hive.html

-- 
Harsh J
www.harshj.com