You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "weiqingy (via GitHub)" <gi...@apache.org> on 2026/08/30 02:41:27 UTC
[I] [Bug][security][skills] Bound download and extraction size for URL skill sources [flink-agents]
weiqingy opened a new issue, #1072:
URL: https://github.com/apache/flink-agents/issues/1072
### Description
`Skills.from_url(...)` / `Skills.fromUrl(...)` download a remote zip and extract it without bounding how many bytes get written, in either runtime. A URL skill source can therefore fill the TaskManager's temp filesystem. That is a resource exhaustion property rather than a transport or integrity one, so it is tracked apart from #1003.
The download streams the response straight to a temp file with no cap, and without consulting `Content-Length`: `shutil.copyfileobj(resp, out)` at `python/flink_agents/runtime/skill/repository/_materialize.py:162`, and `Files.copy(in, tmpZip, StandardCopyOption.REPLACE_EXISTING)` at `runtime/src/main/java/org/apache/flink/agents/runtime/skill/repository/SkillMaterializer.java:263`. A source that streams without ending, or simply serves a very large body, runs until the disk is full.
Extraction is unbounded too. `extract_zip_safely` validates every entry against zip-slip and checks nothing else, then extracts: `zf.extractall(extract_dir)` at `_materialize.py:133`, and the per-entry `Files.copy(in, target)` at `SkillMaterializer.java:142`. There is no per-entry size cap, no cumulative uncompressed cap and no entry count cap, so a small archive can expand well past its transfer size.
Digest pinning does not close this. In the change proposed on #1005 the SHA-256 is computed after the download helper returns, so the file is already fully on disk before any operator expectation is checked. The Java side has the same ordering.
This came out of reviewing #1005 and was raised on #1003 first: https://github.com/apache/flink-agents/issues/1003#issuecomment-5383724087
### Expected behavior
- Bound the bytes written during download, failing closed with an error that names the limit.
- Bound extraction on cumulative uncompressed size, per-entry size and entry count, checked before writing past the limit rather than after.
- Where the server declares a `Content-Length` above the cap, reject before streaming. That is a cheap early exit, not a substitute for counting bytes as they arrive, since the header can be absent or wrong.
- Keep the existing partial-output cleanup on the failure path, so a rejected download or extraction leaves nothing behind.
- Keep the limits and any configuration surface aligned across Java and Python.
- Add focused Java and Python tests for a download past the cap, an archive that expands past the cap, and unchanged behavior for an archive inside the limits.
### How to reproduce
1. Serve a response at an HTTPS URL that streams without ending, or whose body is far larger than any real skill archive.
2. Configure it through `Skills.from_url(...)` or `Skills.fromUrl(...)`.
3. Observe that the download runs until the source stops or the filesystem fills. Pinning a digest does not change this, because the digest is verified after the file is written.
4. Separately, serve a small valid zip whose entries expand to many gigabytes.
5. Observe that extraction proceeds entry by entry until the filesystem fills.
### Version and environment
Present on `main` as of 2026-08-29 (`0417dad3`) and in 0.3.1, in both the Java and Python implementations. The line numbers above are from `main`. On the 0.3.1 tag the Java lines are `SkillMaterializer.java:264` and `:143`; the Python lines are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][security][skills] Bound download and extraction size for URL skill sources [flink-agents]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #1072:
URL: https://github.com/apache/flink-agents/issues/1072#issuecomment-5471912640
Thanks for offering @pranavshuklaa, it is open and yours if you want it. Your read of the direction looks right, and the Expected behavior list in the description is the bar I had in mind.
One thing before you branch. #1005 is open and touches the same code, including the download helper and the URL repository on both the Java and Python sides. It is still active, so you may want to wait for it to land or base your branch on it. Otherwise you are likely in for a rebase.
On the limits, pick numbers you can justify and propose them in the PR. Easier to discuss a concrete value in review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][security][skills] Bound download and extraction size for URL skill sources [flink-agents]
Posted by "pranavshuklaa (via GitHub)" <gi...@apache.org>.
pranavshuklaa commented on issue #1072:
URL: https://github.com/apache/flink-agents/issues/1072#issuecomment-5469381599
Hi @weiqingy I would like to work on this if it is open.
My initial read is to bound both download_to_tempfile and extract_zip_safely in _materialize.py reject an over-cap Content-Length up front, independently track bytes as they stream in so a missing or incorrect header can't bypass the check, and enforce per-entry size, cumulative size, and entry-count limits while extracting rather than after. I could then mirror the same logic in skillmaterializer.java on the java side.
I want to confirm the direction with you first, particularly around the limits and whether you’d prefer the Java and Python changes together or separately.
Also Do you have specific limits in mind for the download/extraction caps?
Happy to take this up and work on the implementation once I have your direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org