Change of behaviour on CBP for Hive 3 with null checks against complex types

[Eyal Ben Ivri <ey...@gmail.com>]

Hi there,

We are in the process of migrating to HDP 3.1.4, from an older HDP stack.

We have noticed strange behaviour regarding the CBP and how it treats null 
checks against complex types.

consider the following:



CREATE TABLE test_null_array (null_array ARRAY<STRING>);

INSERT OVERWRITE TABLE test_null_array SELECT EXPLODE(ARRAY(NULL, 
ARRAY('foobar')));
SELECT * FROM test_null_array; -- returns two rows (NULL, ["foobar"])

SET hive.cbo.enable=false;
SELECT * FROM test_null_array WHERE null_array IS NULL; -- returns one row 
(NULL) as expected
EXPLAIN SELECT * FROM test_null_array WHERE null_array IS NULL; -- line 9: 
predicate:null_array is null

SET hive.cbo.enable=true;
SELECT * FROM test_null_array WHERE null_array IS NULL; -- returns zero 
rows *unexpected*
EXPLAIN SELECT * FROM test_null_array WHERE null_array IS NULL; -- line 9: 
predicate:false

We have a suspicion that this is new behaviour for HIVE 3. Has anyone else 
noticed this behaviour, can direct me to documentation about this?




Thank you, Eyal Ben Ivri