[jira] [Created] (IMPALA-15616) Iceberg V3: DEFAULT NULL on a nullable column is stored as the string 'NULL' or fails in catalogd

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-15616:
------------------------------------------

             Summary: Iceberg V3: DEFAULT NULL on a nullable column is stored as the string 'NULL' or fails in catalogd
                 Key: IMPALA-15616
                 URL: https://issues.apache.org/jira/browse/IMPALA-15616
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog, Frontend
            Reporter: Zoltán Borók-Nagy


On Iceberg V3 tables, DEFAULT NULL or DEFAULT CAST(NULL AS <type>) on a nullable column does not mean SQL NULL. Impala sends the default to catalogd as the text 'NULL', and catalogd parses that text as a value of the column type. This affects CREATE TABLE, ALTER TABLE ADD COLUMN(S) and ALTER TABLE ALTER COLUMN SET DEFAULT:
* STRING columns get the 4-character string 'NULL' as their default, with no error (wrong results).
* Other types pass analysis, then fail in catalogd with a raw parse error.

Repro on master, STRING column:
{code:sql}
create table t (i int, s string default null) stored by iceberg tblproperties('format-version'='3');
insert into t (i) values (1);
insert into t values (2, null);
select i, s, s is null, length(s) from t order by i;
{code}
{noformat}
1	NULL	false	4
2	NULL	true	NULL
{noformat}
Expected for row 1: NULL, true, NULL. The current-schema property in DESCRIBE FORMATTED t has the string as the default (quotes unescaped):
{noformat}
{"id":2,"name":"s","required":false,"type":"string","initial-default":"NULL","write-default":"NULL"}
{noformat}

ALTER TABLE gives the same result:
{code:sql}
create table u (i int) stored by iceberg tblproperties('format-version'='3');
insert into u values (1);
alter table u add column s string default null;
alter table u add columns (s2 string default cast(null as string));
alter table u add column s3 string default 'x';
alter table u alter column s3 set default null;
insert into u (i) values (2);
select i, s, s is null, s2, s2 is null, s3, s3 is null from u order by i;
{code}
{noformat}
1	NULL	false	NULL	false	x	false
2	NULL	false	NULL	false	NULL	false
{noformat}
Expected: s and s2 are NULL in both rows (IS NULL true), and s3 is 'x' in row 1 and NULL in row 2.

INT, BIGINT, DECIMAL, DOUBLE, BOOLEAN and DATE pass analysis but fail in catalogd. For example:
{code:sql}
create table t2 (i int default null) stored by iceberg tblproperties('format-version'='3');
{code}
{noformat}
ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
CAUSED BY: NumberFormatException: Character N is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
{noformat}
{code:sql}
alter table u add column b boolean default null;
{code}
{noformat}
ImpalaRuntimeException: Failed to parse default value 'NULL' for column 'b': Cannot parse default value 'NULL' as BOOLEAN: invalid BOOLEAN literal: NULL
{noformat}
{code:sql}
alter table u alter column i set default cast(null as int);
{code}
{noformat}
ImpalaRuntimeException: Failed to ALTER table 'u': Character N is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
{noformat}

Only a NULL default on a NOT NULL column is rejected in analysis:
{code:sql}
create table t5 (s string not null default null) stored by iceberg tblproperties('format-version'='3');
{code}
{noformat}
AnalysisException: Default value of NULL not allowed on non-nullable column: 's'
{noformat}

*Analysis*
ColumnDef.analyze() (in analyzeKuduOptions(), which also handles Iceberg defaults) keeps the NullLiteral as the analyzed default and rejects it only on NOT NULL or primary key columns. CreateTableStmt.toThrift(), AlterTableAddColsStmt.toThrift() and AlterTableAlterColStmt.toThrift() send ColumnDef.getIcebergDefaultValueString() to catalogd as iceberg_initial_default and iceberg_write_default. For a NullLiteral, this returns NullLiteral.getStringValue(), which is the text "NULL". In catalogd, IcebergSchemaConverter.parseStringToIcebergLiteral() parses that text with LiteralExpr.createFromUnescapedStr(). It is called from IcebergSchemaConverter.createIcebergField(), IcebergCatalogOpExecutor.addColumns() and IcebergCatalogOpExecutor.alterColumn(). For STRING the parse returns StringLiteral("NULL"). For the other types it throws. The text form cannot tell a NULL default from the string 'NULL'.

*Suggested fix*
Handle a NULL default during analysis, so that a NullLiteral is never turned into text. There are two options:
# (preferred) Treat DEFAULT NULL or CAST(NULL AS <type>) on a nullable Iceberg column as "no default". CREATE TABLE and ADD COLUMN(S) would then set neither iceberg_initial_default nor iceberg_write_default. ALTER COLUMN SET DEFAULT NULL would set iceberg_drop_write_default, so it does the same as DROP DEFAULT. Returning null from getIcebergDefaultValueString() is not enough: with iceberg_write_default unset, SET DEFAULT NULL would do nothing. This option matches Iceberg, where a field without initial-default or write-default has a null default. It also matches Kudu, which accepts DEFAULT NULL on nullable columns (kudu_create.test).
# Reject a NULL default with an AnalysisException in the Iceberg DDL paths, next to the IcebergSchemaConverter.validateIcebergDdlDefaultType() calls.

*Testing*
Add FE analysis tests and an EE test for CREATE TABLE, ADD COLUMN(S) and ALTER COLUMN SET DEFAULT with NULL and CAST(NULL AS <type>), on a STRING column and on a non-STRING column. Check IS NULL on rows written before the column was added and on INSERTs that omit the column.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)