You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/10/13 01:26:31 UTC

[I] [Bug] Flink bucket pruning misses compound keys containing placeholder literals [hudi]

cshuo opened a new issue, #20291:
URL: https://github.com/apache/hudi/issues/20291

   ### Bug Description
   
   **What happened:**
   
   Flink bucket pruning can silently omit existing rows when a compound record key contains the literal string `__empty__` or `__null__`.
   
   For example, with record key fields `uuid,name`, eight buckets, and `(uuid='__empty__', name='x')`, the writer assigns the row to bucket **1**, but the reader's equality-filter pruning selects bucket **4**. An unfiltered scan returns the row, while filtering on both key fields returns no rows. This was reproduced with both COPY_ON_WRITE and MERGE_ON_READ tables.
   
   **What you expected:**
   
   An equality query on a stored compound key should return the same matching row as an unfiltered scan.
   
   **Steps to reproduce:**
   
   1. Create a Flink Hudi table with a compound primary key and bucket index:
   
   ```sql
   CREATE TABLE t1 (
     uuid STRING,
     name STRING,
     age INT,
     ts TIMESTAMP(3),
     `partition` STRING,
     PRIMARY KEY (uuid, name) NOT ENFORCED
   ) PARTITIONED BY (`partition`) WITH (
     'connector' = 'hudi',
     'path' = '/tmp/hudi-compound-key-bucket-pruning',
     'table.type' = 'COPY_ON_WRITE',
     'hoodie.datasource.write.recordkey.field' = 'uuid,name',
     'ordering.fields' = 'ts',
     'index.type' = 'BUCKET',
     'hoodie.bucket.index.num.buckets' = '8'
   );
   
   INSERT INTO t1 VALUES
     ('__empty__', 'x', 23, TIMESTAMP '1970-01-01 00:00:01', 'par1');
   ```
   
   2. Run a batch scan and an equality query:
   
   ```sql
   SELECT * FROM t1;
   SELECT * FROM t1 WHERE name = 'x' AND uuid = '__empty__';
   ```
   
   3. The first query returns the row, but the second returns an empty result. The same issue occurs with the literal `__null__` and with MERGE_ON_READ tables.
   
   **Root cause:**
   
   The writer hashes values extracted from the encoded record key using `KeyGenUtils.extractRecordKeysByFields`. For compound keys, that extraction maps `__empty__` to an empty string and `__null__` to null. `PrimaryKeyPruners.getBucketIdFunc` instead hashes the filter literals directly, producing a different bucket ID. The mismatch also applies when only a subset of the compound key fields is used for bucket hashing. Single-field keys retain the literal placeholder strings and must preserve that behavior.
   
   ### Environment
   
   - Hudi: `1.3.0-SNAPSHOT`, reproduced against commit `a41f6eaf7df0`.
   - Query engine: Flink 1.20.1, batch SQL.
   - Java: 11.
   - Table types: COPY_ON_WRITE and MERGE_ON_READ.
   - Relevant configuration: compound record key `uuid,name`, bucket index, eight buckets.
   
   ### Logs and Stack Trace
   
   This is a silent incorrect-result issue, not a query exception. Regression assertions against the unpatched implementation report:
   
   ```text
   Bucket ID: expected: <1> but was: <4>
   SQL result: expected: <[+I[__empty__, x, 23, 1970-01-01T00:00:01, par1]]> but was: <[]>
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org