[jira] [Created] (IMPALA-12620) Missing field ID in the eq-delete file could filter out rows with null values

["Gabor Kaszab (Jira)" <ji...@apache.org>]

Gabor Kaszab created IMPALA-12620:
-------------------------------------

             Summary: Missing field ID in the eq-delete file could filter out rows with null values
                 Key: IMPALA-12620
                 URL: https://issues.apache.org/jira/browse/IMPALA-12620
             Project: IMPALA
          Issue Type: Sub-task
            Reporter: Gabor Kaszab


If a malformed equality delete file doesn't have some of the equality field IDs then Parquet schema resolver will identify these ase missing fields but won't fail the query. Missing fields instead are filled with NULL values. But when some of the columns in the equality delete tuples are NULLs then when anti-joining them with the data rows, they will match the NULL values from the data rows. As a result a malformed equality delete file could cause the rows being ommitted from the result where the field ID of the data row contains NULL and the field ID in the equality delete file is missing.

 

E.g.
Test data is

i,     s
(1, "str1")
(NULL, "str2")

and equality field ID is 1 (corresponding to column i).

When an equality delete file doesn't have column i and doesn't have field ID 1 then it will make the second row missing from the result.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org