You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tooling.apache.org by "sbp (via GitHub)" <gi...@apache.org> on 2025/12/09 14:22:37 UTC
[I] Task resource limits do not work consistently between architectures (tooling-trusted-releases)
sbp opened a new issue, #411:
URL: https://github.com/apache/tooling-trusted-releases/issues/411
It appears that memory limits in `atr/worker.py` apply on Linux but do not apply on macOS. @alitheg discovered this because the SBOM generator has too little memory to launch, but only from a task, which points to the task memory limit as being the cause. We need to investigate this further, fix the SBOM generation bug, and then try to make the behaviour consistent between platforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For additional commands, e-mail: dev-help@tooling.apache.org
Re: [I] Task resource limits do not work consistently between architectures (tooling-trusted-releases)
Posted by "dave2wave (via GitHub)" <gi...@apache.org>.
dave2wave commented on issue #411:
URL: https://github.com/apache/tooling-trusted-releases/issues/411#issuecomment-3781528729
rlimits for memory simply do not work on `macOS`.
ChatGPT wrote a test program for me.
```
import sys
import os
import psutil
import resource
import platform
import threading
import time
def set_memory_limit(bytes_limit):
system = platform.system()
if system == "Linux":
# RLIMIT_AS works reliably on Linux
soft, hard = bytes_limit, bytes_limit
resource.setrlimit(resource.RLIMIT_AS, (soft, hard))
print(f"[Linux] RLIMIT_AS set to {bytes_limit} bytes")
elif system == "Darwin":
# macOS: RLIMIT_AS exists but ignored; use psutil monitoring
print(f"[macOS] RLIMIT_AS is unreliable; using manual monitoring")
def monitor_memory():
process = psutil.Process(os.getpid())
while True:
mem = process.memory_info().rss
if mem > bytes_limit:
print(f"[macOS] Memory limit exceeded: {mem} > {bytes_limit}")
os._exit(1) # forcefully kill the process
time.sleep(0.1) # check every 100ms
t = threading.Thread(target=monitor_memory, daemon=True)
t.start()
else:
print(f"[{system}] Unsupported OS for memory limiting")
# Example usage
MEMORY_LIMIT = 2 * 1024**3 # 2 GB
set_memory_limit(MEMORY_LIMIT)
# Example allocation loop
lst = []
try:
while True:
lst.append(' ' * 50_000_000) # allocate 50MB chunks
print(f"Allocated {len(lst)*50} MB")
except MemoryError:
print("Stopped due to memory limit")
```
Let's tune for alpine / ubuntu and our containers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For additional commands, e-mail: dev-help@tooling.apache.org
Re: [I] Task resource limits do not work consistently between architectures (tooling-trusted-releases)
Posted by "alitheg (via GitHub)" <gi...@apache.org>.
alitheg commented on issue #411:
URL: https://github.com/apache/tooling-trusted-releases/issues/411#issuecomment-3810119843
This seems to be working as it can now - we get a warning in the logs if running on macOS, but otherwise the limits are set and all the tasks are working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For additional commands, e-mail: dev-help@tooling.apache.org
Re: [I] Task resource limits do not work consistently between architectures (tooling-trusted-releases)
Posted by "sbp (via GitHub)" <gi...@apache.org>.
sbp commented on issue #411:
URL: https://github.com/apache/tooling-trusted-releases/issues/411#issuecomment-3632603075
Further testing by @alitheg revealed that this also caused an issue with Java starting. GCed languages pre-allocate lots of virtual memory, e.g. "[golang malloc.go allocates 512GB heap for 64bit system](https://groups.google.com/g/golang-nuts/c/FCMPvaBMaMg)", and our resource limits in `worker.py` are for virtual memory rather than resident memory. Therefore we'd have to find an alternative strategy to constrain the resources of our workers if we want to keep such limits. We're considering using containers for the worker tasks, but we'd have to either use Docker in Docker (which currently requires an alternative, experimental runtime), or allow the container to orchestrate tasks on the host. Another possibility is that we could just move the orchestration out of the server entirely. It may be possible to make the tasks themselves self-orchestrating. It doesn't seem advisable to just drop resource limits.
Alternatively, of course, we are considering pushing some tasks to GitHub. As long as we can keep our remaining tasks under control this could be fine, but it does mean that bugs could starve the system of CPU or RAM. We should also probably think about inode limits etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For additional commands, e-mail: dev-help@tooling.apache.org
Re: [I] Task resource limits do not work consistently between architectures (tooling-trusted-releases)
Posted by "sbp (via GitHub)" <gi...@apache.org>.
sbp commented on issue #411:
URL: https://github.com/apache/tooling-trusted-releases/issues/411#issuecomment-3632672846
As pointed out by @alitheg, we don't necessarily have to run a container per task: we could run a pool of tasks. This has the benefit that one greedy task can take more resources than its fair share as long as the other tasks aren't being greedy. It also means we only have to run one extra container. It would only work if we can also probe for the offending task to terminate it; the process that manages this can be in the task pool container. On the other hand we would incur all of the extra conceptual overhead of running a separate container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For additional commands, e-mail: dev-help@tooling.apache.org
Re: [I] Task resource limits do not work consistently between architectures (tooling-trusted-releases)
Posted by "alitheg (via GitHub)" <gi...@apache.org>.
alitheg closed issue #411: Task resource limits do not work consistently between architectures
URL: https://github.com/apache/tooling-trusted-releases/issues/411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For additional commands, e-mail: dev-help@tooling.apache.org
Re: [I] Task resource limits do not work consistently between architectures (tooling-trusted-releases)
Posted by "dave2wave (via GitHub)" <gi...@apache.org>.
dave2wave commented on issue #411:
URL: https://github.com/apache/tooling-trusted-releases/issues/411#issuecomment-3781529587
See #550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tooling.apache.org
For additional commands, e-mail: dev-help@tooling.apache.org