[jira] [Created] (HIVE-21618) Hive table query with a UDF call, shows multiple calls to the UDF.

["Rajveer Singh (JIRA)" <ji...@apache.org>]

Rajveer Singh created HIVE-21618:
------------------------------------

             Summary: Hive table query with a UDF call, shows multiple calls to the UDF.
                 Key: HIVE-21618
                 URL: https://issues.apache.org/jira/browse/HIVE-21618
             Project: Hive
          Issue Type: Bug
          Components: Tez
    Affects Versions: 2.3.4, 3.0.1
         Environment: *Hortonworks (HDP3)*
 * Hadoop 3.1.1.3.0.1.0-187
 * Hive 3.1.0
 * Tez 0.9.1

 

*EMR*
 * Hadoop 2.8.5-amzn-1
 * Hive 2.3.4
 * Tez 0.9.1
            Reporter: Rajveer Singh
         Attachments: TimestampUDF.java, emr_19_cluster_timestampUDF.log, hdp_3_cluster_timestampUDF.log

Hive table query with a UDF call, shows multiple calls to the UDF.

 

This behavior leads to performance issues as well as it also gives incorrect results while collecting the duration of any operation.

To show the problem TimestampUDF (sample code has been attached) has been created which returns three fields namely *start_time* (string), *end_time* (string) and *duration_milli* (long). To keep the things simple, fixed 2 second delay has been added. In the TimestampUDF duration_mill is calculated as (end_time - start_time).

The expectation is that *duration_milli* should be equal to *(end_time - start_time)*

 

I have tested following Hive Queries:
 # Select T.Candidate.startTime, T.Candidate.endTime, T.Candidate.duration_milli from (Select TimestampUDF() as Candidate) T;
 # Select T.Candidate.startTime, T.Candidate.endTime, T.Candidate.duration_milli from (Select TimestampUDF() as Candidate from *idtable*) T;

 

In Hive query (1)                              

I am getting correct values because TimestampUDF is being called once and if  we calculate the  *(end_time - start_time)* with returned *duration_milli,* they both are same.

 

However, in Hive query (2)          

*duration_milli* is not equal to *(end_time - start_time)* because the TimestampUDF is being called thrice (the number of fields in sub-select query) and also
 * start_time is from 1^st^  call of TimestampUDF
 * end_time is from 2^nd^ call of TimestampUDF
 * duration_milli is from 3^rd^ call of TimestampUDF

 

I got following results ** from hive query (2), same values can be validated in attached logs file or respective platform

 

*EMR* 

+---------------------------+---------------------------+-----------------+
| starttime | endtime | duration_milli |
+---------------------------+---------------------------+-----------------+
| 2019-04-16T11:53:13.595Z | 2019-04-16T11:53:17.619Z | 2000 |
+---------------------------+---------------------------+-----------------+

 

*HDP3*

+---------------------------+---------------------------+-----------------+
| starttime | endtime | duration_milli |
+---------------------------+---------------------------+-----------------+
| 2019-04-16T12:37:33.593Z | 2019-04-16T12:37:37.597Z | 2000 |
+---------------------------+---------------------------+-----------------+

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)