[I] [Bug] Z-order boolean FALSE encodes to the same bytes as NULL [paimon]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #9526:
URL: https://github.com/apache/paimon/issues/9526

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   ### Paimon version
   
   master, `9c7deebbd` (2.1-SNAPSHOT)
   
   ### Compute Engine
   
   Flink (`sort_compact` with `order_strategy = zorder`) and Spark (`sys.compact` with `order_strategy => 'zorder'`, and clustered writes). Each engine has its own copy of the encoding: `ZIndexer` in paimon-common, `SparkZOrderUDF` in paimon-spark-common.
   
   ### Minimal reproduce step
   
   Z-order encodes each order column into eight bytes and then interleaves the bits. `ZOrderByteUtils.NULL_BYTES` is eight zero bytes and stands for null. The boolean encoder writes only the first byte of its buffer:
   
   ```java
   // ZIndexer, BOOLEAN visitor
   ZOrderByteUtils.reuse(reuse, PRIMITIVE_BUFFER_SIZE);
   reuse.put(0, (byte) (row.getBoolean(fieldIndex) ? -127 : 0));
   return reuse.array();
   ```
   
   The remaining seven bytes of that per-column buffer are never written, so they stay zero for the life of the indexer. FALSE therefore encodes to eight zero bytes, byte for byte identical to `NULL_BYTES`, and a FALSE row and a NULL row get the same z-order key:
   
   ```sql
   CREATE TABLE T (a BOOLEAN, b INT) TBLPROPERTIES ('bucket' = '-1');
   INSERT INTO T VALUES (true, 1), (false, 2), (null, 3);
   CALL paimon.sys.compact(table => 'T', order_strategy => 'zorder', order_by => 'a,b');
   ```
   
   The compaction succeeds, and the rows with `a = false` and `a = null` are clustered as if that column held the same value.
   
   ### What doesn't meet your expectations?
   
   A boolean column has three states and each should get its own encoding, as it does for every other type in that class: TRUE is `0x81`, NULL is all zeros, and FALSE needs to be something else. Sharing the null encoding means the clustering silently does less than it claims: files that could be skipped for `a = false` are read anyway, and nothing reports a problem.
   
   ### Anything else?
   
   `SparkZOrderUDF.booleanToOrderedBytesUDF` has the same literal, so both engines are affected, and they have to stay in step: a column clustered by Spark and later compacted by Flink must land in the same order.
   
   While looking at this I also noticed that `NULL_BYTES` equals the encoding of `Long.MIN_VALUE` for a BIGINT or TIMESTAMP column, since `tinyintToOrderedBytes` and friends flip the sign bit. That one needs a different remedy and I am not proposing to change it here.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Z-order boolean FALSE encodes to the same bytes as NULL [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #9526: [Bug] Z-order boolean FALSE encodes to the same bytes as NULL
URL: https://github.com/apache/paimon/issues/9526


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org