[I] [Bug] PD TTL cleanup deletes the marker-shaped name instead of the expired key, so expired values stay in the store until read [hugegraph]

["bitflicker64 (via GitHub)" <gi...@apache.org>]

bitflicker64 opened a new issue, #3378:
URL: https://github.com/apache/hugegraph/issues/3378

   ### Bug Type (问题类型)
   
   logic (逻辑设计问题)
   
   ### Before submit
   
   - [x] I have confirmed and searched that there are no similar problems in the historical issue and documents
   
   ### Environment (环境信息)
   
   - Server Version: PD on `master` at `86c05ef9c` (1.8.0-SNAPSHOT); the code below is unchanged since `43cae1d2e` (#2478, 2024-04-19)
   - Backend: PD's own RocksDB metadata store (the KV service), checked without raft as the `pd-core-test` suite runs it
   - OS: macOS, JDK 17.0.20.1
   - Data Size: one key
   
   ### Expected & Actual behavior (期望与实际表现)
   
   A key written with a TTL (`KvService.put(key, value, ttl)`, the KV `putTTL` RPC, PD locks, the KV watch registrations, the balance-shard flag) gets a marker `K@T@<expireAt>@<key>` next to it. `TaskScheduleService` runs `KvService.clearTTLData()` every second on the leader, and it should delete the expired key and its marker.
   
   It deletes a different key. `clearTTLData` takes the key from the marker after `key.indexOf(KV_DELIMITER, 2)`, which is the `@` right after `T`, so it deletes `<expireAt>@<key>`, a key that never exists, and then removes the marker. `hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/KvService.java` at `86c05ef9c`, lines 300 to 306:
   
   ```java
   for (KV kv : kvList) {
       String key = new String(kv.getKey());
       int index = key.indexOf(KV_DELIMITER, 2);
       String delKey = key.substring(index + 1);
       delete(delKey);
       meta.remove(kv.getKey());
   }
   ```
   
   The same expression on a real marker key:
   
   ```text
   key    = "K@T@1729430000000@ttl-test/a"
   index  = 3
   delKey = "1729430000000@ttl-test/a"
   ```
   
   Measured 2026-10-20 on `master` `86c05ef9c` with JDK 17.0.20.1, by adding this test to `hg-pd-test`'s `core/KvServiceTest` and running `mvn test -pl hugegraph-pd/hg-pd-test -am -P pd-core-test -Dtest='org.apache.hugegraph.pd.core.KvServiceTest#testTtlCleanupDeletesTheExpiredKey'`:
   
   ```java
   KvService service = new KvService(getPdConfig());
   service.put("ttl-test/a", "old", 1L);
   byte[] storeKey = service.getStoreKey("ttl-test/a");
   Thread.sleep(20L);
   service.clearTTLData();
   Assert.assertNull(service.getMeta().getOne(storeKey));   // fails: the value is still stored
   ```
   
   ```text
   java.lang.AssertionError: expected null, but was:<[B...@1b18fa79>
   	at org.apache.hugegraph.pd.core.KvServiceTest.testTtlCleanupDeletesTheExpiredKey(KvServiceTest.java:75)
   ```
   
   What this does and does not break:
   
   1. Reads are not affected: `get` and `scanWithPrefix` go through `getValue`, which returns nothing for an expired value and removes it then. So a reader never sees an expired value.
   2. An expired value that nobody reads again stays in the store for good, with no marker left that would ever delete it.
   3. `keepAlive(key)` does not check the expiry, so it revives an expired lease that nobody has read yet, which the cleanup was meant to have deleted within a second.
   4. `getValue` (line 130) removes the marker under a wrong name too (`getTTLStoreKey(new String(keyBytes), ...)` passes the store key, `K@<key>`, where the key is expected), so its marker is left for the cleanup.
   
   The parse fix is small (start the search after `K@T@`), but the cleanup then deletes keys that writers may be renewing at that moment: a `keepAlive` or a `putTTL` of the same key that lands between the cleanup's read and its delete would be deleted. The delete should be conditional at the apply: only while the marker still exists and the key still holds the expired value it read.
   
   ### Vertex/Edge example (问题点 / 边数据举例)
   
   N/A
   
   ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
   
   N/A
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@hugegraph.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org