You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "wutongyuonce (via GitHub)" <gi...@apache.org> on 2026/09/08 16:29:24 UTC
[PR] feat(computer-use): keep overlay cursor elevated when the aim point is covered [maka]
wutongyuonce opened a new pull request, #5052:
URL: https://github.com/apache/maka/pull/5052
## Summary
`obscuringRects` already travel from maka-cu through the observation record, and `CuObservation` says the agent cursor should test the point it is about to draw. `keepElevated` still sank whenever a window id existed, so a covered aim point could disappear under the covering window.
This PR:
- adds `obscuringRects` to overlay context
- copies them from the session observation record
- stays elevated only when the aim point hits a rect
- keeps today's sink behaviour when rects are missing or the point is clear
Does not change the desktop sink z-order implementation. Elevation is per point, not per whole window.
Fixes #5048
Does this PR entail a change in behavior?
- [x] Yes — described under Summary above
- [ ] No
## Verification
- `npm --workspace @maka/runtime run build`
- `npm --workspace @maka/computer-use run build`
- `npm --workspace @maka/computer-use run test:dist` — 122 pass, including hook-only point-in-rect cases and `driveRealTool` wiring (covered centre stays up, uncovered centre still sinks)
- Did not run root `lint` / `format:check` / full `typecheck`
- No overlay recording attached (behaviour is asserted in tests; desktop sink is unchanged)
## AI use
Select exactly one:
- [ ] No generative tool made a substantive contribution
- [x] Generative tooling made a substantive contribution
Tool(s) and scope:
pi-coding-agent drafted the overlay context wiring, point-in-rect `keepElevated`, tests, and this description. Commits carry `Generated-by: pi-coding-agent`.
## Checklist
- [x] Tests cover the change and fail without it
- [ ] Lint, format, typecheck and the affected suites pass locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(computer-use): keep overlay cursor elevated when the aim point is covered [maka]
Posted by "hqhq1025 (via GitHub)" <gi...@apache.org>.
hqhq1025 commented on code in PR #5052:
URL: https://github.com/apache/maka/pull/5052#discussion_r4123579677
##########
packages/computer-use/src/__tests__/computer-use-overlay-hook.test.ts:
##########
@@ -332,6 +381,25 @@ test('a semantic action reaches the sink with the point it is aimed at', async (
});
});
+test('a covered aim point from a real observation stays elevated', async () => {
+ // The hook-only cases cannot see computer-use-tools copying record.obscuringRects
+ // onto overlay context. A rect covering the element centre (220,110) must survive
+ // observe → registerObservation → runWithPresentation.
+ const events = await driveRealTool(
+ { obscuringRects: [{ x: 210, y: 100, width: 20, height: 20 }] },
+ { action: 'click_element', element_id: '5' },
+ );
+ assert.equal((events[0]?.input as { keepElevated?: boolean }).keepElevated, true);
Review Comment:
[P1] Fix both new assertions before this can pass the required CI. `events[0]?.input` may evaluate to `undefined`, after which `.keepElevated` is accessed outside the optional chain and throws. Biome reports `lint/correctness/noUnsafeOptionalChaining` here and at line 400; root `npm run lint` fails on both the exact head and the fresh-main merge tree. Please assert that the first event exists before reading `input`, or use non-optional access after the existing event-sequence assertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(computer-use): keep overlay cursor elevated when the aim point is covered [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on PR #5052:
URL: https://github.com/apache/maka/pull/5052#issuecomment-6159238416
#5051 has just been merged, and it touches the same test file (`packages/computer-use/src/__tests__/computer-use-overlay-hook.test.ts`), so this PR now conflicts with main. Could you rebase onto the latest main and resolve the conflict? We'll re-check the new head and merge it once CI is green. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(computer-use): keep overlay cursor elevated when the aim point is covered [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han merged PR #5052:
URL: https://github.com/apache/maka/pull/5052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(computer-use): keep overlay cursor elevated when the aim point is covered [maka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5052:
URL: https://github.com/apache/maka/pull/5052#issuecomment-6073933889
<!-- maka-lifecycle:stale-warning -->
This pull request has had no new commits for 30 days and has been marked stale. It will be closed in 7 days unless a new commit is pushed.
Comments do not reset this timer: only a new commit does. If the pull request is intentionally long-lived, a maintainer can apply the `pinned` label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(computer-use): keep overlay cursor elevated when the aim point is covered [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on PR #5052:
URL: https://github.com/apache/maka/pull/5052#issuecomment-6162912146
Thanks for the quick rebase! Merging now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(computer-use): keep overlay cursor elevated when the aim point is covered [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #5052:
URL: https://github.com/apache/maka/pull/5052#discussion_r4242507301
##########
packages/computer-use/src/__tests__/computer-use-overlay-hook.test.ts:
##########
@@ -332,6 +381,25 @@ test('a semantic action reaches the sink with the point it is aimed at', async (
});
});
+test('a covered aim point from a real observation stays elevated', async () => {
+ // The hook-only cases cannot see computer-use-tools copying record.obscuringRects
+ // onto overlay context. A rect covering the element centre (220,110) must survive
+ // observe → registerObservation → runWithPresentation.
+ const events = await driveRealTool(
+ { obscuringRects: [{ x: 210, y: 100, width: 20, height: 20 }] },
+ { action: 'click_element', element_id: '5' },
+ );
+ assert.equal((events[0]?.input as { keepElevated?: boolean }).keepElevated, true);
Review Comment:
[P2] Fix both unsafe optional-chain assertions so CI can run
This assertion and its companion at line 400 still fail `lint/correctness/noUnsafeOptionalChaining`. Optional chaining can return `undefined`, but the parenthesized cast is then dereferenced unconditionally. The current-head CI run fails in `npm run lint` with these two errors, before the main build/tests; the same five-file Biome check reproduces both locally.
Assert the recorded event exists before reading `input.keepElevated`, or keep the property access inside the safe optional chain. Update both assertions and rerun CI. The rebase did not fix this existing blocker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org