You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@magpie.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/10/24 20:34:29 UTC

[PR] feat(security): import email-delivered scans; add submit-to-oss-scan [magpie]

potiuk opened a new pull request, #1786:
URL: https://github.com/apache/magpie/pull/1786

   ## Summary
   
   - **Import a scan that arrives as one email.** The Anthropic OSS Scanner discloses a whole scan in a single multi-MB plain-text mail (an INDEX, one `ANT-YYYY-…` block per finding, a "NOT SECURITY" list, a GPG-encrypted archive). `security-issue-import-from-scan` only knew ASVS report folders and GitHub issues. This adds an `anthropic-oss-scanner` scan-format adapter: `security-management issue-import-from-scan parse-email` turns the `.eml` (or a saved Gmail-connector result) into a scan folder (`index.json` + one `evidence/<id>.md` per finding) and refuses to call a scan complete unless the preamble totals, every severity banner, the INDEX, the parsed blocks and the closing signature all agree. Attachments are saved byte-exact (sha256, OpenPGP framing checked, never decrypted); every file is written owner-only.
   - **Read the whole message.** New read-only `oauth-draft-fetch-message` saves a Gmail message's raw RFC-5322 source to a file. The Gmail connector inlines bodies into context and its text rendering drops control characters (a NUL-byte password reads as `""`), so connector-derived input is marked `lossy`. Reports built from a private mail never go to a gist.
   - **`security-submit-to-oss-scan`, a new skill.** Prepares the enrolment (or update) PR against `anthropics/oss-scanner` — `project.yaml`, an offline-capable `Dockerfile` derived from the project's own CI, and a `threat_model.md` **derived from the project's security model** — and walks the maintainer through `tools/validate.py` / `tools/check` before anything is opened. The summary is wired into the model lifecycle: `security-model-prepare` hands off to it, `security-model-update` Step 12 refreshes stale copies (tracked by model commit in a new *Scanner threat-model summaries* table), and `issue-import-from-scan` routes repeat by-design clusters back to `security-model-update`.
   
   ## Type of change
   
   - [x] Skill change — new `security-submit-to-oss-scan`; `issue-import-from-scan`, `security-model-update`, `security-model-prepare` amended; eval suite added
   - [x] Tool / bridge contract (`tools/scan-format/README.md` — email-delivered scans)
   - [x] Python package (`tools/security-management`, `tools/gmail/oauth-draft`)
   - [x] Project template (`security-model.md` — scanner summaries table)
   - [x] Documentation
   
   ## Test plan
   
   - [x] `prek run` on every changed file passes
   - [x] `uv run pytest` for `tools/security-management` and `tools/gmail/oauth-draft` (new: `test_issue_import_from_scan_oss_scanner.py`, `test_fetch_message.py`)
   - [x] End to end on a real 115-finding disclosure (3.27 MB `.eml`): 115 findings + 16 not-security bugs, `checks.complete: true`, identical parse from the `.eml` and the connector JSON except where the connector dropped control characters; the 1.5 MB encrypted archive saved intact
   - [x] New eval suite `tools/skill-evals/evals/security-submit-to-oss-scan/` (threat-model derivation: supported claim vs model gap)
   - [x] Nine rounds of `roborev` review, every finding fixed with a regression test
   
   ## RFC-AI-0004 compliance
   
   - [x] **HITL** — the import never auto-imports; the enrolment PR, its checklist boxes and the CLA signature stay the maintainer's
   - [x] **Sandbox** — `tools/check` is never run by the agent; nothing executes the scanner's reproducers
   - [x] **Vendor neutrality** — placeholders throughout; the scan-format contract stays adapter-based
   - [x] **Write-access discipline** — drafts only; `gh pr create --web` for the enrolment PR
   - [x] **Privacy LLM** — private disclosure mail stays local (no gist), files are 0600
   
   ## Notes for reviewers
   
   - The parser was exercised against the real layout, which differs from the Gmail rendering (padded header fields, an RFC 3676 signature delimiter); the fixture mirrors the raw form.
   - The scanner asks fixers to cite the finding id in the commit; the adapter doc notes that for a pre-advisory fix that line would disclose the vulnerability, so the id belongs in the tracker.
   
   ---
   
   Gen-AI disclosure: drafted with Claude Code (Opus 5); reviewed by the submitter.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(security): import email-delivered scans; add submit-to-oss-scan [magpie]

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.

github-advanced-security[bot] commented on code in PR #1786:
URL: https://github.com/apache/magpie/pull/1786#discussion_r4368439167


##########
tools/security-management/src/security_management/issue_import_from_scan/oss_scanner.py:
##########
@@ -0,0 +1,490 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""The `anthropic-oss-scanner` scan-format adapter: one disclosure email, many findings.
+
+The Anthropic OSS Scanner discloses a whole scan in a single plain-text
+email (`[Anthropic OSS Scanner] Disclosing N security vulnerabilities in
+<project>`): a preamble naming the target and commit, an INDEX, one
+`====`-fenced block per finding grouped under `####` severity banners, and
+a trailing `NOT SECURITY: N ORDINARY BUGS` list. An encrypted archive with
+reproducers and candidate patches is attached; this module never decrypts
+it and never runs anything from it.
+
+- `load_body` takes the email as an `.eml` file, as the plain-text body, or
+  as the JSON a Gmail connector saved (`plaintextBody`), and returns the
+  text with RFC 3676 `format=flowed` undone. Gmail's text rendering re-wraps
+  long lines, so every parse below is whitespace-tolerant.
+- `parse` returns the scan metadata, every finding normalised to the
+  `tools/scan-format/` schema with its verbatim block as evidence, the
+  not-security list, and completeness `checks`: the declared total, the
+  INDEX and the finding blocks must all agree, or the email was truncated.
+- `materialise` writes a scan folder (`index.json` + `evidence/<id>.md`)
+  that the skill reads one finding at a time.
+- `save_attachments` writes the attachments out of an `.eml` byte-exact
+  (sanitised names, sha256) and, for an encrypted archive, checks the
+  OpenPGP packet framing so a truncated download shows — without
+  decrypting it.
+"""
+
+from __future__ import annotations
+
+import email
+import email.policy
+import hashlib
+import json
+import os
+import re
+import tempfile
+from collections import Counter
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+FORMAT = "anthropic-oss-scanner"
+INDEX_FILE = "index.json"
+SEVERITIES = ("critical", "high", "medium", "low")
+
+_ID = r"ANT-\d{4}-[A-Z0-9]{6,}"
+_LOC = r"[\w./-]+:\d+"
+_BANNER = re.compile(r"^#{20,}\n(.+?)\n#{20,}$", re.M)
+_FENCE = re.compile(r"^={20,}$", re.M)

Review Comment:
   ## CodeQL / Unused global variable
   
   The global variable '_FENCE' is not used.
   
   [Show more details](https://github.com/apache/magpie/security/code-scanning/156)



##########
tools/security-management/src/security_management/issue_import_from_scan/oss_scanner.py:
##########
@@ -0,0 +1,490 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""The `anthropic-oss-scanner` scan-format adapter: one disclosure email, many findings.
+
+The Anthropic OSS Scanner discloses a whole scan in a single plain-text
+email (`[Anthropic OSS Scanner] Disclosing N security vulnerabilities in
+<project>`): a preamble naming the target and commit, an INDEX, one
+`====`-fenced block per finding grouped under `####` severity banners, and
+a trailing `NOT SECURITY: N ORDINARY BUGS` list. An encrypted archive with
+reproducers and candidate patches is attached; this module never decrypts
+it and never runs anything from it.
+
+- `load_body` takes the email as an `.eml` file, as the plain-text body, or
+  as the JSON a Gmail connector saved (`plaintextBody`), and returns the
+  text with RFC 3676 `format=flowed` undone. Gmail's text rendering re-wraps
+  long lines, so every parse below is whitespace-tolerant.
+- `parse` returns the scan metadata, every finding normalised to the
+  `tools/scan-format/` schema with its verbatim block as evidence, the
+  not-security list, and completeness `checks`: the declared total, the
+  INDEX and the finding blocks must all agree, or the email was truncated.
+- `materialise` writes a scan folder (`index.json` + `evidence/<id>.md`)
+  that the skill reads one finding at a time.
+- `save_attachments` writes the attachments out of an `.eml` byte-exact
+  (sanitised names, sha256) and, for an encrypted archive, checks the
+  OpenPGP packet framing so a truncated download shows — without
+  decrypting it.
+"""
+
+from __future__ import annotations
+
+import email
+import email.policy
+import hashlib
+import json
+import os
+import re
+import tempfile
+from collections import Counter
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+FORMAT = "anthropic-oss-scanner"
+INDEX_FILE = "index.json"
+SEVERITIES = ("critical", "high", "medium", "low")
+
+_ID = r"ANT-\d{4}-[A-Z0-9]{6,}"
+_LOC = r"[\w./-]+:\d+"

Review Comment:
   ## CodeQL / Unused global variable
   
   The global variable '_LOC' is not used.
   
   [Show more details](https://github.com/apache/magpie/security/code-scanning/155)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(security): import email-delivered scans; add submit-to-oss-scan [magpie]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk merged PR #1786:
URL: https://github.com/apache/magpie/pull/1786


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org