You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "mmodzelewski (via GitHub)" <gi...@apache.org> on 2026/07/06 09:05:26 UTC
[PR] feat(server): respect allowed CPU set and add pin_cores toggle [iggy]
mmodzelewski opened a new pull request, #3615:
URL: https://github.com/apache/iggy/pull/3615
Shard threads pinned to absolute cores 0..n, silently violating any
affinity/cpuset restriction from the parent environment (systemd
AllowedCPUs=, container cpusets, taskset): sched_setaffinity to a
core outside the allowed set fails with EINVAL. On shared hosts it
also piled every process onto the same low-numbered cores while the
rest sat idle.
Draw pinned cores from the process's allowed CPU set instead, and
fail fast when an explicit range names a core outside it. Add
pin_cores (default true); set false on multi-tenant hosts to drop
pinning entirely so the kernel scheduler places shard threads
freely. With a NUMA-aware allocation, false also drops the memory-
node bindings and warns, since NUMA placement without pinning is
meaningless.
Also report cgroup memory limits in shard stats when tighter than
the host, so a quota-restricted server reports its real ceiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): respect CPU affinity and cgroup limits in shards and stats [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3615:
URL: https://github.com/apache/iggy/pull/3615#discussion_r3542885611
##########
core/configs/src/server_config/sharding.rs:
##########
@@ -101,6 +101,10 @@ const fn default_inbox_capacity() -> usize {
DEFAULT_INBOX_CAPACITY
}
+const fn default_pin_cores() -> bool {
+ true
+}
Review Comment:
nvm, i'll do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): respect CPU affinity and cgroup limits in shards and stats [iggy]
Posted by "mmodzelewski (via GitHub)" <gi...@apache.org>.
mmodzelewski merged PR #3615:
URL: https://github.com/apache/iggy/pull/3615
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): respect CPU affinity and cgroup limits in shards and stats [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3615:
URL: https://github.com/apache/iggy/pull/3615#discussion_r3542813098
##########
core/configs/src/server_config/sharding.rs:
##########
@@ -101,6 +101,10 @@ const fn default_inbox_capacity() -> usize {
DEFAULT_INBOX_CAPACITY
}
+const fn default_pin_cores() -> bool {
+ true
+}
Review Comment:
dont do it like so, default value should be read from toml, see how other config fields default values are assumed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): respect allowed CPU set and add pin_cores toggle [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3615:
URL: https://github.com/apache/iggy/pull/3615#issuecomment-4891444778
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/3615?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 72.29%. Comparing base ([`e32d296`](https://app.codecov.io/gh/apache/iggy/commit/e32d296a99576adbaa5cbcc5385bbf388a0d9245?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb9859f`](https://app.codecov.io/gh/apache/iggy/commit/cb9859fadf18fbeef365acb8c125ba33118f52ae?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3615 +/- ##
============================================
- Coverage 74.00% 72.29% -1.71%
Complexity 937 937
============================================
Files 1263 1200 -63
Lines 132810 128105 -4705
Branches 108447 108340 -107
============================================
- Hits 98282 92619 -5663
- Misses 31430 32476 +1046
+ Partials 3098 3010 -88
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.55% <ø> (+0.01%)` | :arrow_up: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.44% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.10% <ø> (ø)` | |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `91.65% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.29% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.27% <ø> (-4.08%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/3615/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.66% <ø> (-24.48%)` | :arrow_down: |
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/3615?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core/common/src/types/stats/mod.rs](https://app.codecov.io/gh/apache/iggy/pull/3615?src=pr&el=tree&filepath=core%2Fcommon%2Fsrc%2Ftypes%2Fstats%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb21tb24vc3JjL3R5cGVzL3N0YXRzL21vZC5ycw==) | `69.86% <ø> (ø)` | |
| [core/configs/src/server\_config/sharding.rs](https://app.codecov.io/gh/apache/iggy/pull/3615?src=pr&el=tree&filepath=core%2Fconfigs%2Fsrc%2Fserver_config%2Fsharding.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25maWdzL3NyYy9zZXJ2ZXJfY29uZmlnL3NoYXJkaW5nLnJz) | `85.44% <ø> (ø)` | |
| [core/configs/src/server\_config/validators.rs](https://app.codecov.io/gh/apache/iggy/pull/3615?src=pr&el=tree&filepath=core%2Fconfigs%2Fsrc%2Fserver_config%2Fvalidators.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25maWdzL3NyYy9zZXJ2ZXJfY29uZmlnL3ZhbGlkYXRvcnMucnM=) | `78.36% <ø> (-0.04%)` | :arrow_down: |
| [core/server-ng/src/bootstrap.rs](https://app.codecov.io/gh/apache/iggy/pull/3615?src=pr&el=tree&filepath=core%2Fserver-ng%2Fsrc%2Fbootstrap.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXItbmcvc3JjL2Jvb3RzdHJhcC5ycw==) | `10.93% <ø> (+<0.01%)` | :arrow_up: |
| [core/server/src/main.rs](https://app.codecov.io/gh/apache/iggy/pull/3615?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fmain.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL21haW4ucnM=) | `62.68% <ø> (+0.18%)` | :arrow_up: |
| [core/server/src/shard/system/stats.rs](https://app.codecov.io/gh/apache/iggy/pull/3615?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fshard%2Fsystem%2Fstats.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL3NoYXJkL3N5c3RlbS9zdGF0cy5ycw==) | `91.20% <ø> (ø)` | |
| [core/server/src/shard\_allocator.rs](https://app.codecov.io/gh/apache/iggy/pull/3615?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fshard_allocator.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL3NoYXJkX2FsbG9jYXRvci5ycw==) | `16.14% <ø> (+0.68%)` | :arrow_up: |
... and [123 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/3615/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): respect allowed CPU set and add pin_cores toggle [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3615:
URL: https://github.com/apache/iggy/pull/3615#discussion_r3535835633
##########
core/shard_allocator/src/lib.rs:
##########
@@ -287,39 +335,66 @@ impl ShardAllocator {
pub fn to_shard_assignments(&self) -> Result<Vec<ShardInfo>, ShardingError> {
match &self.allocation {
CpuAllocation::All => {
+ // `available_parallelism` already accounts for both the
+ // affinity mask and any cgroup CPU quota, so a
+ // quota-restricted process gets proportionally fewer shards.
let available_cpus = available_parallelism()
.map_err(|err| ShardingError::Other {
msg: format!("Failed to get available_parallelism: {:?}", err),
})?
.get();
- let shard_assignments: Vec<_> = (0..available_cpus)
- .map(|cpu_id| ShardInfo {
- cpu_set: HashSet::from([cpu_id]),
- numa_node: None,
- })
- .collect();
+ if !self.pin_cores {
+ info!("Using all available CPU cores ({available_cpus} shards, unpinned)");
+ return Ok(unpinned(available_cpus));
+ }
+
+ let allowed = allowed_cpus();
+ let shard_assignments =
+ pinned_from_allowed(&allowed, available_cpus.min(allowed.len()))?;
info!(
- "Using all available CPU cores ({} shards with affinity)",
- shard_assignments.len()
+ "Using all available CPU cores ({} shards pinned within allowed set {:?})",
+ shard_assignments.len(),
+ allowed
);
Ok(shard_assignments)
}
CpuAllocation::Count(count) => {
- let shard_assignments = (0..*count)
- .map(|cpu_id| ShardInfo {
- cpu_set: HashSet::from([cpu_id]),
- numa_node: None,
- })
+ if !self.pin_cores {
+ info!("Using {count} shard(s), unpinned");
+ return Ok(unpinned(*count));
+ }
+
+ let allowed = allowed_cpus();
+ let shard_assignments = pinned_from_allowed(&allowed, *count)?;
+ let cores: Vec<usize> = shard_assignments
+ .iter()
+ .flat_map(|shard| shard.cpu_set.iter().copied())
.collect();
- info!("Using {count} shards with affinity to cores 0..{count}");
+ info!("Using {count} shard(s) with affinity to cores {cores:?}");
Ok(shard_assignments)
}
CpuAllocation::Range(start, end) => {
+ if !self.pin_cores {
+ info!(
+ "Using {} shard(s) for range {start}..{end}, unpinned",
+ end - start
+ );
+ return Ok(unpinned(end - start));
Review Comment:
`unpinned(end - start)` underflows when `start > end` - debug build panics on the subtract, release wraps to a huge value and `unpinned()` then tries to allocate a vec of ~`usize::MAX` `ShardInfo` and aborts. not reachable in production today since both boot paths run `ShardingConfig::validate` first and it rejects `start >= end`, but `to_shard_assignments` is a pub fn in an internal crate and the server-ng `validate_sharding_runtime_knobs` net doesn't cover cpu_allocation, so a direct/test/embedded caller hits it. cheap fix - reject `start >= end` in this arm so the allocator self-guards instead of trusting the caller.
##########
core/server/src/shard/system/stats.rs:
##########
@@ -88,6 +88,14 @@ impl IggyShard {
stats.written_bytes = disk_usage.total_written_bytes.into();
stats.threads_count = process.tasks().map(|t| t.len() as u32).unwrap_or(0);
+
+ if let Some(limits) = process
+ .cgroup_limits()
+ .filter(|limits| limits.total_memory < sys.total_memory())
+ {
+ stats.total_memory = limits.total_memory.into();
+ stats.available_memory = limits.free_memory.into();
Review Comment:
under a memory-capped cgroup this sets `available_memory` to `memory.max - memory.current` (or lower if a parent cgroup is tighter), and `memory.current` charges reclaimable page cache as used. iggy is cache-heavy, so on a warm instance `memory.current` climbs toward the cap and `available_memory` trends toward ~0 even though the kernel reclaims that cache long before OOM - it under-reports real headroom, which is the opposite of what a per-instance stats endpoint wants. for an honest number you'd want a cgroup MemAvailable-equivalent: `memory.max - (memory.current - reclaimable file cache)`, where reclaimable is `inactive_file + active_file` from `memory.stat` - sysinfo's `free_memory` doesn't do that, so it needs reading `memory.stat` directly. also this override only fires when a memory limit is set (the `total < host total` filter just above), so an instance confined by cpuset alone still reports host-wide memory - multi-instance setups need a memory cap too. re-scoping `total
_memory` to the cap is correct.
##########
core/configs/src/server_config/validators.rs:
##########
@@ -488,9 +489,27 @@ impl Validatable<ConfigurationError> for ShardingConfig {
);
return Err(ConfigurationError::InvalidConfigurationValue);
}
- if *end > available_cpus {
+ if !self.pin_cores {
+ if *end - *start > available_cpus {
+ eprintln!(
+ "Invalid sharding configuration: cpu_allocation range {start}..{end} yields {} shards, exceeding available CPU cores {available_cpus}",
+ *end - *start
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ return Ok(());
+ }
+ if *end - *start > available_cpus {
Review Comment:
this `*end - *start > available_cpus` check and its `eprintln!` are identical to the one in the `!self.pin_cores` branch above. hoist it above the `if !self.pin_cores` split so it runs once, and leave only the allowed-set membership check gated behind pinning. the `start >= end` guard earlier still protects the subtraction. saves ~7 lines.
##########
core/shard_allocator/src/lib.rs:
##########
@@ -287,39 +335,66 @@ impl ShardAllocator {
pub fn to_shard_assignments(&self) -> Result<Vec<ShardInfo>, ShardingError> {
match &self.allocation {
CpuAllocation::All => {
+ // `available_parallelism` already accounts for both the
+ // affinity mask and any cgroup CPU quota, so a
+ // quota-restricted process gets proportionally fewer shards.
let available_cpus = available_parallelism()
.map_err(|err| ShardingError::Other {
msg: format!("Failed to get available_parallelism: {:?}", err),
})?
.get();
- let shard_assignments: Vec<_> = (0..available_cpus)
- .map(|cpu_id| ShardInfo {
- cpu_set: HashSet::from([cpu_id]),
- numa_node: None,
- })
- .collect();
+ if !self.pin_cores {
+ info!("Using all available CPU cores ({available_cpus} shards, unpinned)");
+ return Ok(unpinned(available_cpus));
+ }
+
+ let allowed = allowed_cpus();
+ let shard_assignments =
+ pinned_from_allowed(&allowed, available_cpus.min(allowed.len()))?;
info!(
- "Using all available CPU cores ({} shards with affinity)",
- shard_assignments.len()
+ "Using all available CPU cores ({} shards pinned within allowed set {:?})",
+ shard_assignments.len(),
+ allowed
);
Ok(shard_assignments)
}
CpuAllocation::Count(count) => {
- let shard_assignments = (0..*count)
- .map(|cpu_id| ShardInfo {
- cpu_set: HashSet::from([cpu_id]),
- numa_node: None,
- })
+ if !self.pin_cores {
+ info!("Using {count} shard(s), unpinned");
+ return Ok(unpinned(*count));
+ }
+
+ let allowed = allowed_cpus();
+ let shard_assignments = pinned_from_allowed(&allowed, *count)?;
+ let cores: Vec<usize> = shard_assignments
Review Comment:
this `cores` vec is only built to log it, and since `pinned_from_allowed` takes `allowed.iter().take(count)` in order it's just `&allowed[..*count]`. log the slice directly and drop the vec + `flat_map`.
##########
core/shard_allocator/src/lib.rs:
##########
@@ -287,39 +335,66 @@ impl ShardAllocator {
pub fn to_shard_assignments(&self) -> Result<Vec<ShardInfo>, ShardingError> {
match &self.allocation {
CpuAllocation::All => {
+ // `available_parallelism` already accounts for both the
+ // affinity mask and any cgroup CPU quota, so a
+ // quota-restricted process gets proportionally fewer shards.
let available_cpus = available_parallelism()
.map_err(|err| ShardingError::Other {
msg: format!("Failed to get available_parallelism: {:?}", err),
})?
.get();
- let shard_assignments: Vec<_> = (0..available_cpus)
- .map(|cpu_id| ShardInfo {
- cpu_set: HashSet::from([cpu_id]),
- numa_node: None,
- })
- .collect();
+ if !self.pin_cores {
+ info!("Using all available CPU cores ({available_cpus} shards, unpinned)");
+ return Ok(unpinned(available_cpus));
+ }
+
+ let allowed = allowed_cpus();
+ let shard_assignments =
+ pinned_from_allowed(&allowed, available_cpus.min(allowed.len()))?;
info!(
- "Using all available CPU cores ({} shards with affinity)",
- shard_assignments.len()
+ "Using all available CPU cores ({} shards pinned within allowed set {:?})",
+ shard_assignments.len(),
+ allowed
);
Ok(shard_assignments)
}
CpuAllocation::Count(count) => {
- let shard_assignments = (0..*count)
- .map(|cpu_id| ShardInfo {
- cpu_set: HashSet::from([cpu_id]),
- numa_node: None,
- })
+ if !self.pin_cores {
+ info!("Using {count} shard(s), unpinned");
+ return Ok(unpinned(*count));
+ }
+
+ let allowed = allowed_cpus();
+ let shard_assignments = pinned_from_allowed(&allowed, *count)?;
+ let cores: Vec<usize> = shard_assignments
+ .iter()
+ .flat_map(|shard| shard.cpu_set.iter().copied())
.collect();
- info!("Using {count} shards with affinity to cores 0..{count}");
+ info!("Using {count} shard(s) with affinity to cores {cores:?}");
Ok(shard_assignments)
}
CpuAllocation::Range(start, end) => {
+ if !self.pin_cores {
+ info!(
+ "Using {} shard(s) for range {start}..{end}, unpinned",
+ end - start
+ );
+ return Ok(unpinned(end - start));
+ }
+
+ // An explicit range is operator intent for exact cores; honor
Review Comment:
the comment says the range is honored as-is, but the validator already caps a pinned range by `available_parallelism()` before this runs, so a range wider than the quota never reaches here. minor, but the comment overstates what this arm actually sees.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org