You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Koert Kuipers <ko...@tresata.com> on 2011/08/30 15:34:42 UTC
UDAF and group by
If i run my own UDAF with group by, can i be sure that a single UDAF
instance initialized once will process all members in a group? Or should i
code so as to take into account the situation where even within a group
multiple UDAFs could run, and i would have to deal with terminatePartial()
and merge() even within a group?
My problem is that my results within a group are not easily merged, but
between groups they are.
Re: UDAF and group by
Posted by Koert Kuipers <ko...@tresata.com>.
Thanks for your response.
My thinking was that by turning off hive.map.aggr hive would do the
following:
col3 becomes the key in mapping. All rows with same col3 go to same reducer.
In the reducer the values (=col1,col2) are sorted by key (=col3) and myUdf
iterates over the over the values, with terminate() and init() being called
when the key changes. If this is how it is implemented then what would be
the situation where merge() and terminatePartial() would be called?
I have run this with 200mm+ rows of data (but with all groups less than 10k
in size) so far without any calls to merge() or terminatePartial().
Best, Koert
On Sun, Sep 4, 2011 at 11:10 PM, Huan Li <yr...@gmail.com> wrote:
> Setting hive.map.aggr false will reduce the chance of terminatePartial()
> and merge() being called. Though I don't think it will eliminate the
> possibility. If your data is large, it's still possible that a group of data
> is processed by multiple reducers and those two methods are needed.
>
> If you need to process records in each group in a single method, you can
> first use collect_set to collect your group data and process them in a UDF.
>
>
> 2011/9/4 Koert Kuipers <ko...@tresata.com>
>
>> Hey, my question wasn't very clear. I have a UDAF that I apply per group.
>> The UDAF does not support terminatePartial() and merge(). So to do this i
>> run:
>>
>> set hive.map.aggr=false;
>> select myUdf(col1, col2) from table group by col3;
>>
>> Now this seems to work. But are my assumptions correct that this will
>> never call terminatePartial() or merge()?
>> Thanks Koert
>>
>>
>> On Thu, Sep 1, 2011 at 11:59 PM, Huan Li <yr...@gmail.com> wrote:
>>
>>> Koert, Not sure what you mean by "results can be merged between groups".
>>> UDAF should be used to aggregated records by group. Why need to merge
>>> between groups?
>>>
>>> Can you give some examples of what kind of query you'd like to run?
>>>
>>>
>>> 2011/8/30 Koert Kuipers <ko...@tresata.com>
>>>
>>>> If i run my own UDAF with group by, can i be sure that a single UDAF
>>>> instance initialized once will process all members in a group? Or should i
>>>> code so as to take into account the situation where even within a group
>>>> multiple UDAFs could run, and i would have to deal with terminatePartial()
>>>> and merge() even within a group?
>>>> My problem is that my results within a group are not easily merged, but
>>>> between groups they are.
>>>>
>>>
>>>
>>
>
Re: UDAF and group by
Posted by Huan Li <yr...@gmail.com>.
Setting hive.map.aggr false will reduce the chance of terminatePartial() and
merge() being called. Though I don't think it will eliminate the
possibility. If your data is large, it's still possible that a group of data
is processed by multiple reducers and those two methods are needed.
If you need to process records in each group in a single method, you can
first use collect_set to collect your group data and process them in a UDF.
2011/9/4 Koert Kuipers <ko...@tresata.com>
> Hey, my question wasn't very clear. I have a UDAF that I apply per group.
> The UDAF does not support terminatePartial() and merge(). So to do this i
> run:
>
> set hive.map.aggr=false;
> select myUdf(col1, col2) from table group by col3;
>
> Now this seems to work. But are my assumptions correct that this will never
> call terminatePartial() or merge()?
> Thanks Koert
>
>
> On Thu, Sep 1, 2011 at 11:59 PM, Huan Li <yr...@gmail.com> wrote:
>
>> Koert, Not sure what you mean by "results can be merged between groups".
>> UDAF should be used to aggregated records by group. Why need to merge
>> between groups?
>>
>> Can you give some examples of what kind of query you'd like to run?
>>
>>
>> 2011/8/30 Koert Kuipers <ko...@tresata.com>
>>
>>> If i run my own UDAF with group by, can i be sure that a single UDAF
>>> instance initialized once will process all members in a group? Or should i
>>> code so as to take into account the situation where even within a group
>>> multiple UDAFs could run, and i would have to deal with terminatePartial()
>>> and merge() even within a group?
>>> My problem is that my results within a group are not easily merged, but
>>> between groups they are.
>>>
>>
>>
>
Re: UDAF and group by
Posted by Koert Kuipers <ko...@tresata.com>.
Hey, my question wasn't very clear. I have a UDAF that I apply per group.
The UDAF does not support terminatePartial() and merge(). So to do this i
run:
set hive.map.aggr=false;
select myUdf(col1, col2) from table group by col3;
Now this seems to work. But are my assumptions correct that this will never
call terminatePartial() or merge()?
Thanks Koert
On Thu, Sep 1, 2011 at 11:59 PM, Huan Li <yr...@gmail.com> wrote:
> Koert, Not sure what you mean by "results can be merged between groups".
> UDAF should be used to aggregated records by group. Why need to merge
> between groups?
>
> Can you give some examples of what kind of query you'd like to run?
>
>
> 2011/8/30 Koert Kuipers <ko...@tresata.com>
>
>> If i run my own UDAF with group by, can i be sure that a single UDAF
>> instance initialized once will process all members in a group? Or should i
>> code so as to take into account the situation where even within a group
>> multiple UDAFs could run, and i would have to deal with terminatePartial()
>> and merge() even within a group?
>> My problem is that my results within a group are not easily merged, but
>> between groups they are.
>>
>
>
Re: UDAF and group by
Posted by Huan Li <yr...@gmail.com>.
Koert, Not sure what you mean by "results can be merged between groups".
UDAF should be used to aggregated records by group. Why need to merge
between groups?
Can you give some examples of what kind of query you'd like to run?
2011/8/30 Koert Kuipers <ko...@tresata.com>
> If i run my own UDAF with group by, can i be sure that a single UDAF
> instance initialized once will process all members in a group? Or should i
> code so as to take into account the situation where even within a group
> multiple UDAFs could run, and i would have to deal with terminatePartial()
> and merge() even within a group?
> My problem is that my results within a group are not easily merged, but
> between groups they are.
>