You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "gyang94 (via GitHub)" <gi...@apache.org> on 2026/10/22 16:12:50 UTC

[I] [kafka] Serve Metadata from the server cache with tablePropertyCache [fluss]

gyang94 opened a new issue, #4807:
URL: https://github.com/apache/fluss/issues/4807

   ### Description
   
   Follow-up to [review comment 1 on PR #4261](https://github.com/apache/fluss/pull/4261#discussion_r4345366418), under #4185. This work is deferred to a separate change.
   
   Kafka Metadata currently lists databases and tables in ZooKeeper for all-topics discovery and the internal topic-ID lookup path. Resolving individual tables can also reread table descriptors and schemas from ZooKeeper, even when routing metadata is cached. Frequent Metadata requests therefore increase ZooKeeper load as the catalog and client count grow.
   
   ### Proposed direction
   
   - Add a dedicated `tablePropertyCache` inside `TabletServerMetadataCache`, analogous to the existing schema cache. For now, cache only the parsed **boolean `table.kafka.enabled` value per table ID**. Do not retain property maps, property strings, full `TableInfo` objects, or schemas in this cache; keep its memory footprint small.
   - Populate and update the flag through the existing `UpdateMetadata` flow, which already carries table information. Handle table creation, property changes, deletion/recreation, and server restart without adding per-request ZooKeeper reads.
   - Keep `ServerMetadataSnapshot` unchanged. Construct a `MetadataView` directly from the routing snapshot and property cache under the metadata synchronization mechanism, so each request sees a coherent, stable view of both. An additional `MetadataCacheState` wrapper is unnecessary.
   - Use that view for named-topic lookup, all-topics discovery, and existing internal topic-ID resolution. Reuse the native Fluss metadata processing path for authorization, listener routing, and response construction, while preserving Kafka filtering and per-topic errors.
   - Verify startup and new-tablet-server cache population. Reuse existing coordinator notifications where sufficient; any coordinator change should address a demonstrated synchronization gap and include a focused test, rather than being assumed necessary for adding the boolean cache.
   
   ### Acceptance criteria
   
   - Steady-state Kafka Metadata processing performs no ZooKeeper catalog scans or per-table descriptor/schema reads.
   - Only the boolean flag is retained as the cached property value, and flag/routing updates cannot produce a mixed view within one request.
   - Tests cover enabled/disabled tables, property updates, deletion/recreation, concurrent metadata updates, and cache population after startup or restart.
   - Existing authorization, listener selection, aliases, and per-topic error behavior are preserved. This follow-up does not expand the currently supported Metadata protocol versions.
   
   The whole-request retry removal and per-topic error handling from review comment 2 have already been handled separately in PR #4261; its remaining requirement to enumerate and resolve tables from the same cache view belongs to this follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [kafka] Serve Metadata from the server cache with tablePropertyCache [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong closed issue #4807: [kafka] Serve Metadata from the server cache with tablePropertyCache
URL: https://github.com/apache/fluss/issues/4807


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [kafka] Serve Metadata from the server cache with tablePropertyCache [fluss]

Posted by "hutiefang76 (via GitHub)" <gi...@apache.org>.
hutiefang76 commented on issue #4807:
URL: https://github.com/apache/fluss/issues/4807#issuecomment-6352692841

   @gyang94, could I help with tests for `table.kafka.enabled` changes and cache population after a tablet-server restart? I see this follows your work in #4261. If you haven’t started that coverage, I can take those tests once we agree the scope and revision to use.
   
   I use Codex and DeepSeek assistance for investigation and test drafts; I’ll run the agreed tests before submitting.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org