You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "wutongyuonce (via GitHub)" <gi...@apache.org> on 2026/09/22 15:30:27 UTC
[PR] feat(runtime): resolve relative image sources in WebFetch Markdown output [maka]
wutongyuonce opened a new pull request, #5598:
URL: https://github.com/apache/maka/pull/5598
## Summary
Fixes #5596
WebFetch converts fetched HTML to Markdown for the model. `<a href>` values were rewritten to absolute URLs against the document base, but `<img src>` was not, so a page without a `<base>` element produced Markdown whose images the model cannot resolve:
```markdown
See [the setup notes](https://docs.example.com/guide/intro/setup) before wiring this up.
 <- meaningless outside the page it came from
```
Rewrite `img[src]` through the same base-URL path as `[href]`, so both Markdown link forms carry absolute URLs.
## Verification
Ran from `apache/maka@30c406c9e` plus this change:
- `npm --workspace @maka/runtime run typecheck` — passed
- `npm --workspace @maka/runtime run build` — passed
- `node --test .../dist/__tests__/local-web-fetch.test.js` — 14 passed, 0 failed
- `node --test .../dist/__tests__/web-fetch-tool.test.js` — 19 passed, 0 failed
- `npx biome check` on both changed files — clean
Measured against the built executor, page `https://docs.example.com/guide/v2/install`:
| input | before | after |
|---|---|---|
| `<img src="../images/wiring.png">`, no `<base>` | `../images/wiring.png` | `https://docs.example.com/guide/images/wiring.png` |
| `<img src="logo.png">` with `<base href="https://cdn.example/assets/">` | already absolute | unchanged |
| `<a href="../intro/setup">`, no `<base>` | already absolute | unchanged |
One new test fails on the unmodified base (`resolves relative image sources against the response URL`). The other new test passes both before and after by design — it guards against the rewrite mangling already-absolute and inline `data:` sources rather than pinning new behavior. I dropped a `<base>`-relative image test after measuring that Readability already resolves those URIs, which would have made it unable to fail.
Not run: no live site was fetched; the tests drive the executor with synthetic responses.
## AI use
Select exactly one:
- [ ] No generative tool made a substantive contribution
- [x] Generative tooling made a substantive contribution
Tool(s) and scope: Pi (pi-coding-agent) traced the extraction path, wrote the change and tests, and ran the verification above. The commit carries a `Generated-by: Pi (pi-coding-agent)` trailer.
## Checklist
- [x] Tests cover the change and fail without it
- [x] Lint, format, typecheck and the affected suites pass locally
Does this PR entail a change in behavior?
- [x] Yes — described under Summary above
- [ ] No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime): resolve relative image sources in WebFetch Markdown output [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han merged PR #5598:
URL: https://github.com/apache/maka/pull/5598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime): resolve relative image sources in WebFetch Markdown output [maka]
Posted by "wutongyuonce (via GitHub)" <gi...@apache.org>.
wutongyuonce commented on code in PR #5598:
URL: https://github.com/apache/maka/pull/5598#discussion_r4082495005
##########
packages/runtime/src/local-web-fetch.ts:
##########
@@ -160,21 +160,29 @@ function htmlToMarkdown(html: string, pageUrl: string): string {
assertSafeHtmlNesting(html);
const { document } = parseHTML(html);
const baseHref = document.querySelector('base[href]')?.getAttribute('href');
- let linkBaseUrl = pageUrl;
+ let baseUrl = pageUrl;
if (baseHref) {
try {
- linkBaseUrl = new URL(baseHref, pageUrl).toString();
+ baseUrl = new URL(baseHref, pageUrl).toString();
} catch {
// Fall back to the response URL for a malformed page-authored base URL.
}
}
- for (const element of document.querySelectorAll<HTMLElement>('[href]')) {
- const href = element.getAttribute('href');
- if (!href) continue;
- try {
- element.setAttribute('href', new URL(href, linkBaseUrl).toString());
- } catch {
- // Keep malformed page-authored links as-is.
+ // Turndown renders both `<a href>` and `<img src>` as Markdown links, so each
+ // has to carry an absolute URL: a relative path stops meaning anything once
+ // the model reads the Markdown apart from the page it was extracted from.
+ for (const [selector, attribute] of [
+ ['[href]', 'href'],
+ ['img[src]', 'src'],
+ ] as const) {
+ for (const element of document.querySelectorAll<HTMLElement>(selector)) {
+ const value = element.getAttribute(attribute);
+ if (!value) continue;
+ try {
+ element.setAttribute(attribute, new URL(value, baseUrl).toString());
+ } catch {
+ // Keep malformed page-authored values as-is.
+ }
}
}
const article = new Readability(document as unknown as Document).parse();
Review Comment:
@Astro-Han Fixed in a14fb5379. Before Readability parses the page, htmlToMarkdown now installs the resolved document base URL, including a page-authored `<base>` value. Readability can then resolve lazy-loaded image URLs it synthesizes after the explicit rewrite pass. Added tests for `data-src` with and without an authored base; all 16 local WebFetch tests pass and Biome is clean. Runtime build is blocked by four TS7006 errors in unrelated model-factory and OpenAI test files in this checkout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime): resolve relative image sources in WebFetch Markdown output [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #5598:
URL: https://github.com/apache/maka/pull/5598#discussion_r4082289667
##########
packages/runtime/src/local-web-fetch.ts:
##########
@@ -160,21 +160,29 @@ function htmlToMarkdown(html: string, pageUrl: string): string {
assertSafeHtmlNesting(html);
const { document } = parseHTML(html);
const baseHref = document.querySelector('base[href]')?.getAttribute('href');
- let linkBaseUrl = pageUrl;
+ let baseUrl = pageUrl;
if (baseHref) {
try {
- linkBaseUrl = new URL(baseHref, pageUrl).toString();
+ baseUrl = new URL(baseHref, pageUrl).toString();
} catch {
// Fall back to the response URL for a malformed page-authored base URL.
}
}
- for (const element of document.querySelectorAll<HTMLElement>('[href]')) {
- const href = element.getAttribute('href');
- if (!href) continue;
- try {
- element.setAttribute('href', new URL(href, linkBaseUrl).toString());
- } catch {
- // Keep malformed page-authored links as-is.
+ // Turndown renders both `<a href>` and `<img src>` as Markdown links, so each
+ // has to carry an absolute URL: a relative path stops meaning anything once
+ // the model reads the Markdown apart from the page it was extracted from.
+ for (const [selector, attribute] of [
+ ['[href]', 'href'],
+ ['img[src]', 'src'],
+ ] as const) {
+ for (const element of document.querySelectorAll<HTMLElement>(selector)) {
+ const value = element.getAttribute(attribute);
+ if (!value) continue;
+ try {
+ element.setAttribute(attribute, new URL(value, baseUrl).toString());
+ } catch {
+ // Keep malformed page-authored values as-is.
+ }
}
}
const article = new Readability(document as unknown as Document).parse();
Review Comment:
**P1 — trigger ① (normal user path): relative sources synthesized by Readability's `_fixLazyImages` still reach the Markdown unresolved.** The rewrite loop only sees the pre-Readability DOM. During `parse()`, `_fixLazyImages` (`Readability.js:~2332`) copies `data-src`/`data-original`/`data-srcset`/`figure[data-src]` values into `src`/`srcset` *after* this loop ran, and `_fixRelativeUris` then cannot resolve them because linkedom's `document.baseURI` is `null` on pages without a `<base>` element (`interface/node.js` — it returns the raw `<base>` href or `location.href`, which is absent). Lazy-loading is the common shape for this exact defect, so a page whose images are all lazy-loaded still produces `` — the PR delivers nothing there. Verified end-to-end on the head pipeline: `<img data-src="/r.png">`, `<img src="/ph.png" data-src="/r.png">`, `<figure data-src>`, and protocol-relative values through that path all leak.
Smallest fix (verified): give the document an absolute `baseURI` so Readability's own `_fixRelativeUris` does the work — e.g. when no `<base>` exists, `document.head?.insertAdjacentHTML('afterbegin', \`<base href="${baseUrl}">\`)` before `parse()`. That resolves the lazy-synthesized `src`/`srcset`/`poster` for free, and keeps the explicit loop for everything else. A test that would have failed on base: `<img data-src="/real.png">` under a page with no `<base>`.
<details>
<summary>中文</summary>
改写循环只能看到 Readability 之前的 DOM;`_fixLazyImages` 在 `parse()` 内把 `data-src` 等值合成 `src`/`srcset` 发生在循环之后,而无 `<base>` 页面 linkedom 的 `baseURI` 为 `null`、`_fixRelativeUris` 解析失败 → 懒加载图片相对路径原样漏出(已端到端验证:data-src、占位 src+data-src、figure[data-src]、协议相对全漏)。最小修法:无 `<base>` 时注入 `<base href="${baseUrl}">`,让 `_fixRelativeUris` 像浏览器一样工作,srcset/poster 顺带解决;补一个 base 上会失败的回归(`<img data-src="/real.png">` + 无 `<base>`)。
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime): resolve relative image sources in WebFetch Markdown output [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #5598:
URL: https://github.com/apache/maka/pull/5598#discussion_r4082289667
##########
packages/runtime/src/local-web-fetch.ts:
##########
@@ -160,21 +160,29 @@ function htmlToMarkdown(html: string, pageUrl: string): string {
assertSafeHtmlNesting(html);
const { document } = parseHTML(html);
const baseHref = document.querySelector('base[href]')?.getAttribute('href');
- let linkBaseUrl = pageUrl;
+ let baseUrl = pageUrl;
if (baseHref) {
try {
- linkBaseUrl = new URL(baseHref, pageUrl).toString();
+ baseUrl = new URL(baseHref, pageUrl).toString();
} catch {
// Fall back to the response URL for a malformed page-authored base URL.
}
}
- for (const element of document.querySelectorAll<HTMLElement>('[href]')) {
- const href = element.getAttribute('href');
- if (!href) continue;
- try {
- element.setAttribute('href', new URL(href, linkBaseUrl).toString());
- } catch {
- // Keep malformed page-authored links as-is.
+ // Turndown renders both `<a href>` and `<img src>` as Markdown links, so each
+ // has to carry an absolute URL: a relative path stops meaning anything once
+ // the model reads the Markdown apart from the page it was extracted from.
+ for (const [selector, attribute] of [
+ ['[href]', 'href'],
+ ['img[src]', 'src'],
+ ] as const) {
+ for (const element of document.querySelectorAll<HTMLElement>(selector)) {
+ const value = element.getAttribute(attribute);
+ if (!value) continue;
+ try {
+ element.setAttribute(attribute, new URL(value, baseUrl).toString());
+ } catch {
+ // Keep malformed page-authored values as-is.
+ }
}
}
const article = new Readability(document as unknown as Document).parse();
Review Comment:
**P1 — trigger ① (normal user path): relative sources synthesized by Readability's `_fixLazyImages` still reach the Markdown unresolved.** The rewrite loop only sees the pre-Readability DOM. During `parse()`, `_fixLazyImages` (`Readability.js:~2332`) copies `data-src`/`data-original`/`data-srcset`/`figure[data-src]` values into `src`/`srcset` *after* this loop ran, and `_fixRelativeUris` then cannot resolve them because linkedom's `document.baseURI` is `null` on pages without a `<base>` element (`interface/node.js` — it returns the raw `<base>` href or `location.href`, which is absent). Lazy-loading is the common shape for this exact defect, so a page whose images are all lazy-loaded still produces `` — the PR delivers nothing there. Verified end-to-end on the head pipeline: `<img data-src="/r.png">`, `<img src="/ph.png" data-src="/r.png">`, `<figure data-src>`, and protocol-relative values through that path all leak.
Smallest fix (verified): give the document an absolute `baseURI` so Readability's own `_fixRelativeUris` does the work — e.g. when no `<base>` exists, `document.head?.insertAdjacentHTML('afterbegin', `<base href="${baseUrl}">`)` before `parse()`. That resolves the lazy-synthesized `src`/`srcset`/`poster` for free, and keeps the explicit loop for everything else. A test that would have failed on base: `<img data-src="/real.png">` under a page with no `<base>`.
<details>
<summary>中文</summary>
改写循环只能看到 Readability 之前的 DOM;`_fixLazyImages` 在 `parse()` 内把 `data-src` 等值合成 `src`/`srcset` 发生在循环之后,而无 `<base>` 页面 linkedom 的 `baseURI` 为 `null`、`_fixRelativeUris` 解析失败 → 懒加载图片相对路径原样漏出(已端到端验证:data-src、占位 src+data-src、figure[data-src]、协议相对全漏)。最小修法:无 `<base>` 时注入 `<base href="${baseUrl}">`,让 `_fixRelativeUris` 像浏览器一样工作,srcset/poster 顺带解决;补一个 base 上会失败的回归(`<img data-src="/real.png">` + 无 `<base>`)。
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org