You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "naivedogger (via GitHub)" <gi...@apache.org> on 2026/10/15 03:00:25 UTC

[I] [rust] Test cluster remote directory mismatch causes flaky scans [fluss]

naivedogger opened a new issue, #4651:
URL: https://github.com/apache/fluss/issues/4651

   ### Search before asking
   
   - [x] I searched the existing issues and did not find a matching report.
   
   ### Fluss version
   
   main (development), after #2782
   
   ### Bug description
   
   The Rust `integration::log_table::table_test::test_poll_batches` integration test intermittently fails when resubscribing from offset 3:
   
   ```text
   assertion `left == right` failed
     left: []
    right: [4, 5, 6, 7, 8]
   ```
   
   Examples:
   - [Run 38658861361, Rust integration job](https://github.com/apache/fluss/actions/runs/38658861361/job/116177430081): 92 passed, 1 failed.
   - [Run 38562011879](https://github.com/apache/fluss/actions/runs/38562011879): the same assertion on an unrelated PR.
   - [Run 38110580132](https://github.com/apache/fluss/actions/runs/38110580132): the same failure while #2782 was under development.
   
   `FlussTestingClusterBuilder::with_remote_data_dir()` configures and mounts a host directory for the tablet servers, but does not apply `remote.data.dir` to the coordinator. The coordinator therefore assigns tables the image's default `/tmp/fluss-remote-data` directory. Since #2782, tablet servers use that table-level directory for remote log storage. Those files are inside the container, outside the host directory mounted for the Rust tests.
   
   The shared Rust test cluster uses 120-byte log segments and a one-second remote log task interval. Reads can succeed through the local path, but a later read from offset 3 can require a tiered segment. The host-side Rust downloader then gets `NotFound` for `/tmp/fluss-remote-data/log/...` and retries until the test's polling deadline expires. This timing-dependent path selection explains why reruns sometimes pass.
   
   ### Reproduction and verification
   
   Using the original server-image artifact from run 38658861361 and its source revision `8664d3e7e249d6079015238bd9651298a449dcfd`:
   
   1. Add a three-second diagnostic delay immediately before creating `trunc_scanner` in `test_poll_batches`, allowing log tiering to run.
   2. Run the test against that server image. The offset-3 request uses remote logs, downloads fail with `NotFound`, and the assertion above fails.
   3. Configure the coordinator with the same host-accessible remote directory. With the same diagnostic delay and server image, the test passes and downloads use the shared directory.
   4. Remove the diagnostic delay and logging: the original Rust integration suite passes all 93 tests with the configuration fix.
   
   The delay was used only to reproduce the path selection; it is not proposed as a fix.
   
   ### Proposed solution
   
   Apply the explicit remote directory to the common test-cluster configuration so the coordinator and tablet servers agree on the directory. Preserve existing configuration when `with_remote_data_dir()` is not used, and add regression coverage for the generated server properties.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [rust] Test cluster remote directory mismatch causes flaky scans [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia closed issue #4651: [rust] Test cluster remote directory mismatch causes flaky scans
URL: https://github.com/apache/fluss/issues/4651


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org