[PR] [Subscription] Make column filters follow schema changes by default [iotdb]

["Caideyipi (via GitHub)" <gi...@apache.org>]

Caideyipi opened a new pull request, #18925:
URL: https://github.com/apache/iotdb/pull/18925

   ## Description
   
   Table subscription column filters currently bind to a schema snapshot when the topic matcher is initialized. A filter such as `column_name = "s2"` retains the original TAGs but omits TAGs added while the consumer is running. New matching FIELDs and tables also require rebinding.
   
   Make column filters follow schema changes by default. Selecting part of a table's FIELDs automatically retains all its TAGs, including later additions, without ALTER TOPIC or a consumer restart. Matching FIELDs and tables are discovered dynamically, and datatype predicates use the current committed table schema. Preserve view aliases, ATTRIBUTE exclusion, per-table TIME selection, and filters that match no columns.
   
   The expression is parsed once. Each Tablet resolves a cached table plan through the local committed schema cache; the plan is rebuilt only when that table's schema instance changes. Retain runtime TAGs in the existing pruning loop and replace per-column table lookups with direct set membership. The local schema accessor preserves metadata lease fencing and does not contact ConfigNode.
   
   Document the behavior in `iotdb-client/subscription/README.md`. There is no fixed column-schema snapshot option. The topic's `mode = snapshot` continues to control the historical data cutoff independently of column filtering.
   
   ## Validation
   
   - 43 unit tests passed: `ColumnFilterBinderTest`, `ColumnFilterParserTest`, `TabletColumnPrunerTest`, `SubscriptionTopicAgentTest`, `DataNodeTableCacheTest`, and `DataNodeTableCacheLeaseTest`.
   - Five changed/new methods in `IoTDBSubscriptionColumnFilterIT` passed: new tables, new FIELDs, new TAGs with selected FIELDs, new TAG values in TsFile payloads, and live datatype changes. Each case warms the matcher before the schema change.
   - `IoTDBConsensusSubscriptionColumnFilterClusterIT#testStreamingColumnFilterIncludesNewTagsAndFieldsOnThreeDataNodes` passed with 1 ConfigNode, 3 DataNodes, and data replication factor 2.
   - DataNode and integration-test compilation, Checkstyle, Spotless, and `git diff --check` passed.
   
   Integration-test condition: master currently compiles `CommonConfig.SUBSCRIPTION_ENABLED` as `false`, so stock test artifacts reject topic creation. The six integration tests ran with that flag enabled only in temporary generated test artifacts, recompiling `DataNodeMemoryConfig` to account for the inlined constant. The original generated jars were restored afterward; the production feature gate is unchanged by this PR.
   
   ## Performance
   
   A local pruning microbenchmark compared the original snapshot matcher/pruner from the base commit with the streaming implementation and a real `DataNodeTableCache`. After warmup, nine alternating samples were measured per case; the table reports medians in ns/Tablet.
   
   | Columns | Rows/Tablet | Snapshot | Streaming |
   | --- | --- | --- | --- |
   | 8 | 1 | 1119.8 | 1159.1 |
   | 8 | 1024 | 586.5 | 399.7 |
   | 32 | 1 | 1760.7 | 1073.5 |
   | 32 | 1024 | 1727.3 | 1057.8 |
   | 128 | 1 | 7134.6 | 4037.6 |
   | 128 | 1024 | 7335.9 | 4231.9 |
   
   The narrow single-row case adds about 39 ns (3.5%); the other measured cases spend less time pruning. These results cover pruning with unchanged schema, not end-to-end throughput. Given the small local lookup cost and reuse of table plans, no legacy schema-snapshot configuration is retained.
   
   This PR has:
   
   - [x] been self-reviewed, including concurrent schema reads and table-plan replacement.
   - [x] added documentation for modified behavior.
   - [x] added comments and Javadocs for the table-plan cache and local schema accessor.
   - [x] added or updated unit tests and integration tests.
   - [x] been tested in IoTDB test clusters under the integration-test condition above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [Subscription] Make column filters follow schema changes by default [iotdb]

["jt2594838 (via GitHub)" <gi...@apache.org>]

jt2594838 merged PR #18925:
URL: https://github.com/apache/iotdb/pull/18925


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org