[PR] [rust] Support writing records to multiple tables [fluss]

["damokelis (via GitHub)" <gi...@apache.org>]

damokelis opened a new pull request, #3962:
URL: https://github.com/apache/fluss/pull/3962

   Generated-by: Codex following AGENTS.md
   
   ### Purpose
   
   Linked issue: close #3961
   
   The Rust client currently requires CDC consumers to manage one table writer per target table. Add a connection-level multi-table writer that routes append, upsert, and delete records while preserving the schema id carried by each record.
   
   ### Brief change log
   
   - Add `MultiTableWriter`, `MultiTableWriteRecord`, and `MultiTableWriteOperation` to the Rust client API.
   - Cache existing append/upsert writer state by table path and schema id.
   - Resolve historical schemas and validate table/operation/schema combinations.
   - Build write batches from each record's table metadata and split Arrow/KV batches on schema changes.
   - Document the new Rust API.
   
   ### Tests
   
   - `cargo build --workspace --all-targets --exclude fluss_python --exclude fluss-cpp --exclude fluss_nif`
   - `cargo test --all-targets --workspace --exclude fluss_python --exclude fluss-cpp --exclude fluss_nif -- --skip client::metadata::tests::parse_bootstrap_preserves_later_entries_when_one_entry_is_unresolvable` (618 passed; one unrelated DNS-dependent test skipped locally)
   - `cargo test -p fluss-rs --features integration_tests writes_cdc_records_to_multiple_tables --test test_fluss -- --nocapture`
   - `cargo fmt --all -- --check`
   - `cargo clippy --all-targets --workspace -- -D warnings`
   - `RUSTDOCFLAGS='-D warnings' cargo doc --workspace --no-deps --exclude fluss_python`
   - `cargo deny check licenses`
   - Apache SkyWalking Eyes license header check
   - `npm run build` in `fluss-rust/website`
   
   ### API and Format
   
   Adds public Rust client APIs. No wire protocol or storage format changes.
   
   ### Documentation
   
   Updates the Rust API reference with the multi-table CDC writer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [rust] Support writing records to multiple tables [fluss]

["damokelis (via GitHub)" <gi...@apache.org>]

damokelis commented on PR #3962:
URL: https://github.com/apache/fluss/pull/3962#issuecomment-5267047802

   Downstream end-to-end validation completed against a real TKE test cluster using this PR's Rust revision .\n\nPipeline tested:\n\n\n\nCoverage and observed results:\n\n- Two primary-key Fluss tables were written concurrently through one .\n- Verified insert, non-key update, primary-key update, and delete on both tables.\n- Fluss streaming reads from  returned the expected changelog:\n  - non-key update: \n  - primary-key update: old-key  plus new-key \n  - delete: \n- Checkpoint recovery was tested by scaling the DTS process to zero, applying five source changes while it was stopped, and starting it again.\n  - persisted checkpoint before stop: , GTID \n  - recovery log resumed from exactly that checkpoint\n  - all five offline changes appeared in Fluss and the checkpoint advanced\n- Both Fluss tables had  and .\n- The existing tiering job generated and committed Paimon snapshot 2 for both tables after the recovered writes:\n  - \n  - \n- No Fluss write errors were observed. Tempo
 rary tables and the isolated namespace were removed after evidence collection.\n\nThe downstream test used serial event batching because its existing  mode depends on relational target metadata; that is a downstream parallelizer limitation, not a limitation of this PR's multi-table writer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org