help with hive -UDAF function - AggregationBuffer overwritten

[Rathish A M <ra...@gmail.com>]

Hi Team,



I am trying to create a parent child hierarchy using hive UDAF function


My Mapper output is coming as per expected.

But in reducer Merge function my AggregationBuffer object is over
written while calling ((LazyBinaryMap) ).getMap() function.This is
happening intermittently

Please let me know is it a technical limitation with hive UDAF function.



highlighted the issue steps :- (When The getMap function executes the
value of ((Components) agg).buffer is overwritten)





public void merge(AggregationBuffer agg, Object partial)

                                                            throws
HiveException {

Map<Text, List>  map2 = ((Components) agg).buffer;

Map<Text, Object> map1 = getMap(partial);    // ((LazyBinaryMap) ).getMap()

Set<Text> set1 = map1.keySet();

for (Text key1 : set1) {

                              if (map2.containsKey(key1)) {


addDistinct(map2.get(key1), getList(map1, key1));

                              } else {

                                             map2.put(key1,
getList(map1, key1));

                              }

}

}



Steps to execute :



ADD JAR <JARNAME.jar>;

CREATE TEMPORARY FUNCTION pch as ‘pch_support';

select pch(empid, mgrid,empname,101) from employee;



Employee table:-



employee.empid  employee.empname        employee.mgrid  employee.designation

101     Browne  0       CEO

102     Steve   101     Development Manager

103     Nicole  101     QA Manager

104     Brian   101     Architect

105     John    102     SSE

106     Suzan   102     SE

107     Dave    102     Test Manager

108     Allen   103     Functional Tester

109     Marry   103     Performance Tester

110     Patty   103     Acceptance Tester

111     Caroline        107     Integration Tester

112     Peggy   107     Regression Tester

113     Albert  107     Stress tester



show create table employee;

OK

createtab_stmt

CREATE EXTERNAL TABLE `employee`(

  `empid` int,

  `empname` string,

  `mgrid` int,

  `designation` string)

ROW FORMAT DELIMITED

  FIELDS TERMINATED BY ','

STORED AS INPUTFORMAT

  'org.apache.hadoop.mapred.TextInputFormat'

OUTPUTFORMAT

  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

LOCATION

  'maprfs:/poc/employee'

TBLPROPERTIES (

  'transient_lastDdlTime'='1412051068')





Thanks & Regards,

Rathish A M