[jira] [Commented] (BEAM-10379) Nulls are not passed to aggregation function in ZetaSQL.

["Kyle Weaver (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312698#comment-17312698 ] 

Kyle Weaver commented on BEAM-10379:
------------------------------------

Oh duh, the CombineFn wrapper just ignores nulls entirely for some reason.

https://github.com/apache/beam/blob/dcd8778cc10efbdf1df22b2a1e888f411d7e4429/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java#L55-L56

> Nulls are not passed to aggregation function in ZetaSQL.
> --------------------------------------------------------
>
>                 Key: BEAM-10379
>                 URL: https://issues.apache.org/jira/browse/BEAM-10379
>             Project: Beam
>          Issue Type: Task
>          Components: dsl-sql, dsl-sql-zetasql
>            Reporter: Yuwei Fu
>            Priority: P2
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> Performs a bitwise AND operation on expression and returns the result.
> Supported Argument Types: INT64
> Returned Data Types: INT64
> Examples
> {code:sql}
> SELECT BIT_AND(c) as bit_and FROM UNNEST([0xF001, 0x00A1]) as c;
> +---------+
> | bit_and |
> +---------+
> | 1       |
> +---------+
> {code}
>  
> What is expected: should include both Calcite and ZetaSQL dialects.
> How to test: unit tests
> Reference: [https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_and]
>  
> Problems: 
> After implementation, the current situation is:
>  1. When table is empty, the result is empty, correct.
>  2. When table contains only null values, the result is null, correct.
>  3.  When table contains only valid numerical values, the result is as expected, correct.
>  4. When table contains both valid numerical values and null values, the result should be null (BitAnd(null, val) = null). But it seems all null values have been directly ignored before doing the bit_and operation. Only numerical values are taken into operation, so the result is incorrect.
>  
> It turns out that on direct runner, NULL will not be passed to CombineFn thus all NULL inputs are ignored. And then if there is any non-null inputs, bit_and will be applied on them only, which leads to a non-null result, which is not correct.
> Before figuring out the root cause and proposing a fix, the previous implementation of BIT_AND is reverted.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)