You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "beryllw (via GitHub)" <gi...@apache.org> on 2025/07/28 12:43:25 UTC
[I] Prefix Lookup Fails in Paimon Lake Format: Key Encoding Mismatch in Fixed-Length Region for Fluss Primary Key Tables [fluss]
beryllw opened a new issue, #1412:
URL: https://github.com/apache/fluss/issues/1412
### Search before asking
- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
### Fluss version
0.7.0 (latest release)
### Please describe the bug 🐞
When using the Paimon lake format for Fluss primary key tables, keys are encoded using the Paimon format, which stores length and offset information for variable-length fields. While primary key point queries work correctly, prefix queries fail to return any data. This issue arises because the encoded fixed-length region of the stored key differs from that of the prefix query key.
**Root Cause:**
The PaimonBinaryRowWriter class (see com.alibaba.fluss.row.encode.paimon.PaimonBinaryRowWriter) encodes variable-length fields differently for primary key look up queries and prefix look up queries:
1. Primary Key Point Query:Encodes all key fields (e.g., two string fields)
2. Encodes only the prefix field (e.g., the first string field)
### Solution
Do we need to use Paimon's encoding for storing keys in primary key tables? Is it possible to use Fluss's default KeyEncoder instead?
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Prefix Lookup Fails in Paimon Lake Format: Key Encoding Mismatch in Fixed-Length Region for Fluss Primary Key Tables [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong closed issue #1412: Prefix Lookup Fails in Paimon Lake Format: Key Encoding Mismatch in Fixed-Length Region for Fluss Primary Key Tables
URL: https://github.com/apache/fluss/issues/1412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Prefix Lookup Fails in Paimon Lake Format: Key Encoding Mismatch in Fixed-Length Region for Fluss Primary Key Tables [fluss]
Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on issue #1412:
URL: https://github.com/apache/fluss/issues/1412#issuecomment-3130445163
When user initiates a write request containing a record with fields:
```
dt: '20250728',
user_id: 'prefix_look_up_paimon_test',
order_id: '1',
log_time: 1
```
The Flink Client encodes this record into a Paimon-formatted WriteRecord, structured as:
```
key: [byte 1 (Paimon Insert)][offset/len for 'prefix_look_up_paimon_test']['1' UTF-8 bytes]['prefix_look_up_paimon_test' UTF-8 bytes]
bucket_key:[byte 1 (Paimon Insert)][offset/len for 'prefix_look_up_paimon_test']['prefix_look_up_paimon_test' UTF-8 bytes]
row:[row UTF-8 bytes]
```
When user initiates a prefix lookup with a partial key matching:
```
dt: '20250728',
user_id: 'prefix_look_up_paimon_test'
```
The Flink Client calls prefixLookup(tableBucket, bucketKeyBytes), bucketKeyBytes mirrors the write path’s bucket_key structure:
```
bucketKeyBytes:[byte 1 (Paimon Insert)][offset/len for 'prefix_look_up_paimon_test']['prefix_look_up_paimon_test' UTF-8 bytes]
```
Key Difference: Write vs. Prefix Lookup
| Step | Key Structure (Simplified) | Purpose
|-- | -- | --|
|Write | [Paimon Insert][user_id meta][order_id UTF-8][user_id UTF-8] | Full record key for storage|
|Prefix Lookup | [Paimon Insert][user_id meta][user_id UTF-8] | Prefix scan for queries|
These prefixes do not match—RocksDB will never find the write key during a prefix scan with the prefix lookup key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Prefix Lookup Fails in Paimon Lake Format: Key Encoding Mismatch in Fixed-Length Region for Fluss Primary Key Tables [fluss]
Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on issue #1412:
URL: https://github.com/apache/fluss/issues/1412#issuecomment-3130282400
Steps to Reproduce:
1. Modify server.yml to set the data lake format:`table.datalake.format: paimon`
2. Start Local Cluster by: /usr/local/fluss/bin/local-cluster.sh start
```
coordinator-server:bind.listeners: FLUSS://{ip}:9123
tablet-server:bind.listeners: FLUSS://{ip}:9124
```
4. Create Primary Key Table```CREATE TABLE IF NOT EXISTS fluss_catalog.fluss.test_prefix_look_up (
`dt` STRING,
`user_id` STRING,
`order_id` STRING,
`log_time` BIGINT,
PRIMARY KEY (
`dt`,
`user_id`,
`order_id`
) NOT ENFORCED
) PARTITIONED BY (dt) WITH (
'bucket.key' = 'user_id',
'bucket.num' = '2',
'table.auto-partition.enabled' = 'true',
'table.auto-partition.time-unit' = 'DAY',
'table.auto-partition.num-precreate' = '1',
'table.auto-partition.num-retention' = '3',
'table.auto-partition.time-zone' = 'Asia/Shanghai'
);
```
4. Insert Test Data:
```
INSERT INTO fluss_catalog.fluss.test_prefix_look_up VALUES ('20250728', 'prefix_look_up_paimon_test', '1234567890123456789', 1);
```
5. Execute prefix look up query:
```java
LookupResult lookup = table.newLookup().lookupBy("dt","user_id").createLookuper().lookup(FlussAsyncLookUp.row("20250728","prefix_look_up_paimon_test")).get();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org