You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Flink Jira Bot (Jira)" <ji...@apache.org> on 2023/08/17 10:36:00 UTC

[jira] [Updated] (FLINK-11141) Key generation for RocksDBMapState can theoretically be ambiguous

     [ https://issues.apache.org/jira/browse/FLINK-11141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-11141:
-----------------------------------
    Labels: auto-deprioritized-critical auto-deprioritized-major auto-deprioritized-minor stale-minor  (was: auto-deprioritized-critical auto-deprioritized-major auto-deprioritized-minor)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.


> Key generation for RocksDBMapState can theoretically be ambiguous
> -----------------------------------------------------------------
>
>                 Key: FLINK-11141
>                 URL: https://issues.apache.org/jira/browse/FLINK-11141
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / State Backends
>    Affects Versions: 1.5.5, 1.6.2, 1.7.0
>            Reporter: Stefan Richter
>            Priority: Minor
>              Labels: auto-deprioritized-critical, auto-deprioritized-major, auto-deprioritized-minor, stale-minor
>
> RocksDBMap state stores values in RocksDB under a composite key from the serialized bytes of {{key-group-id|key|namespace|user-key}}. In this composition, key, namespace, and user-key can either have fixed sized or variable sized serialization formats. In cases of at least 2 variable formats, ambiguity can be possible, e.g.:
> abcd <-> efg
> abc <-> defg
> Our code takes care of this for all other states, where composite keys only consist of key and namespace by checking for 2x variable size and appending the serialized length to each byte sequence.
> However, for map state there is no inclusion of the user-key in the check for potential ambiguity, as well as for appending the size. This means that, in theory, some combinations can produce colliding composite keys in RocksDB. What is required is to include the user-key serializer in the check and append the length there as well.
> Please notice that this cannot be simply changed because it has implications for backwards compatibility and requires some form of migration for the state keys on restore.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)