You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "hongzhi-gao (via GitHub)" <gi...@apache.org> on 2026/06/24 07:52:04 UTC
[PR] Reduce IoTConsensus snapshot transmission memory and log overhead [iotdb]
hongzhi-gao opened a new pull request, #18019:
URL: https://github.com/apache/iotdb/pull/18019
# Optimize Snapshot Transfer Performance and Logging
## Description
### 1. Buffer Reuse on the Snapshot Send Path
- `SnapshotFragmentReader` previously allocated a fresh 10MB `ByteBuffer` per file.
- A snapshot may contain hundreds of thousands of tiny files, leading to significant GC pressure and high allocation costs.
- **Improvement**: The buffer is now caller-supplied and reused across all files (safe because each fragment is serialized synchronously before the next read clears the buffer).
### 2. Lower Per-File Logging Cost
- The full file-name list is now built and emitted only at `debug` level.
- Per-file progress logging is throttled to at most once per configured interval, instead of emitting one `INFO` line per file.
### 3. Make the Progress-Log Interval Configurable
- Added `data_region_iot_snapshot_transmission_progress_log_interval_ms` (default: `5000`, `<= 0` logs every file, hot-reloadable).
- Plumbed through `IoTConsensusConfig.Replication`.
---
## PR Checklist
- [x] Been self-reviewed
- [x] Added comments explaining the "why" and intent of the code wherever it would not be obvious to an unfamiliar reader
- [x] Added unit tests or modified existing tests to cover new code paths
---
## Key Changed/Added Classes (or Packages)
- `SnapshotFragmentReader` / `IoTConsensusServerImpl` (`consensus.iot.snapshot`)
- `IoTConsensusConfig.Replication`
- `IoTDBConfig` / `IoTDBDescriptor` / `DataRegionConsensusImpl`
- `SnapshotFragmentReaderTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce IoTConsensus snapshot transmission memory and log overhead [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #18019:
URL: https://github.com/apache/iotdb/pull/18019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org