You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/05/24 10:57:49 UTC
[PR] feat: add workflow macro UI support [texera]
carloea2 opened a new pull request, #5179:
URL: https://github.com/apache/texera/pull/5179
### What changes were proposed in this PR?
<img width="2558" height="1511" alt="image" src="https://github.com/user-attachments/assets/ff840606-ddd9-463f-a2cf-901acd8420cc" />
Implements **workflow macros**: a way to group canvas operators into a named, collapsible container, seeded by importing another workflow.
**How it works**
- The user drag and drop a virtual "Workflow Macro" operator. Then in the property panel, the user pick any of their workflows. On selection, the workflow's operators and links are **copied (snapshot)** into the macro with fresh IDs, there's no live link back to the source. The macro keeps the source's id/name only as a display label.
- **Collapsed** mode hides the internal operators and replaces them with a single macro node on the canvas; external links that crossed into the macro are proxied through that node so the graph remains connected. **Expanded** mode shows internals inline within the frame.
- Macros are pure UI grouping. They round-trip through workflow persistence (`WorkflowContent.macros[]`) and across collaborative editing, but the execution engine sees the underlying operator graph unchanged — `macroIdParent` is metadata only.
**Scope of changes**
- *Backend (`common/workflow-operator`)*: adds optional `macroIdParent` to `LogicalOp` and hides it from the auto-generated property form.
- *Frontend types*: new `WorkflowMacro` interface; `WorkflowContent` carries an optional `macros[]`.
- *Frontend graph model*: macro framing, collapse/expand visuals, proxy links, macro-aware auto-layout, collaborative sync.
- *Frontend UI*: new macro side panel; context menu gains **create macro** / **remove from macro**; macro-aware delete/paste.
### Any related issues, documentation, discussions?
Closes #5178.
### How was this PR tested?
New unit specs covering macro ID helpers, frame geometry, internal auto-layout, macro CRUD + undo/redo bundling, create/remove predicates, macro-aware context menu, and execution payload shape:
- `joint-graph-wrapper.spec.ts` (+111)
- `workflow-action.service.spec.ts` (+99)
- `operator-menu.service.spec.ts` (+38)
- `context-menu.component.spec.ts` (+6)
- `execute-workflow.service.spec.ts` (+10)
Manual canvas testing:
1. Drop macro → pick a workflow → its operators copy in with fresh IDs.
2. Toggle collapse/expand → internals hide/show, external links proxy through the macro node.
3. Drag macro → children translate together; no spurious position events.
4. **Remove from macro** on a child → `macroIdParent` cleared, frame reflows.
5. Delete macro
6.
### Was this PR authored or co-authored using generative AI tooling?
No
https://github.com/user-attachments/assets/17bfc0ac-d45b-4b45-b556-e9d97b225be0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 closed pull request #5179: feat: add workflow macro UI support
URL: https://github.com/apache/texera/pull/5179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4528244001
@mengw15 can I get your help to decide a split for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4530470763
Agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4530495393
I also prefer macro to be expanded in the frontend. same logic: I hope the design can capture hash join as well.
I am not looking at code yet. so line of change does not matter. I hope to discuss on the design/user experience first.
Can we move the discussion to issues instead of on particular PR? you can always refer to this PR for implementation. also it might be a good idea to break down the issues for discussions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4530161745
@Yicong-Huang, @carloea2
There are many design decisions to make, but there is one particularly important choice in this PR that I strongly prefer over #5115: all macro-related features should be implemented purely on the frontend. The backend should not compile macros, nor should it even be aware of the existence of macros.
By keeping the implementation entirely on the frontend, we can significantly simplify the architecture. That is also why this PR has roughly 5× less code than #5115.
I don’t have strong opinions on the PR size or the number of PRs, but I strongly recommend keeping the implementation frontend-only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4528244939
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4529506515
Thanks for the PR! It's great to see multiple designs of macros. We definitely need this feature!
Let's talk about the experience before diving into code.
### create a macro
It is not shown or clear to me how to create a macro from scratch?
In @Xiao-zhen-Liu 's version (#5115), user highlights three operators and right click to show the context menu, then click "create Marco", then the three operators disappear and replaced with a macro operator. The Marco operator is same shape/size of other normal operators.
I do like this experience.
### Representation of a macro and reuse a macro operator
In this version, user drags a macro operator (which is a marker, instead of an operator), and selects options from a drop down menu inside property panel.
It is not clear how to reuse a marco in #5115.
I would prefer not to have Marco as a marker. instead, we have always been discussing that a Marco operator should just be the same as other logical operators. For instance, you can view `Hash Join` (we should rename it to Join) operator as a macro. It gets expanded into build and probe operators later after compile. I will be great if Marco operators can just look similar to other normal operators (like inside #5115). Most importantly, Marco operators should have ports, that is derived from the sub-dag that it is representing. For instance, if a macro operator contains a sub-dag that has 2 inputs, and 3 output branches, then it will be good for the Marco to have 2 input ports and 3 output ports. This way user can use the macro operator without knowing its internal shape. During execution, user can see the macro's aggregated statistics instead of internal operators' statistics.
### Display the macro internal sub-Dag and edit it
In this version, the macro's content (internal sub-DAG) is displayed inplace. It is more like the grouping view Zuozhi implemented before in #754. And user can drag and drop a new operator into existing macro to edit it. I assume user can also delete one operator or reconnect an edge between operators.
In Xiaozhen's #5115, user needs to click on the macro to jump to another workflow view, where it expands into the internal dag. The internal view has a dummy source operator and a dummy sink operator for user to understand. I think this experience is also mimicking rapidminer/knime. It is not clear if user can edit the internal sub-DAG in that view.
I do like the experience to jump to another workflow view to edit Marco's internal shape, because usually once a macro is created, it will be likely to be reused by many workflows. To edit a macro in place of a particular workflow would also affect the same macro being used in other workflows (think of modifying a function's definition at its callsite, other invocations places of the same function would also be affected). The marco is better to be edited in a stand alone view.
However I do not like the dummy source and dummy sink in #5115. I think we should prioritize ports and just show input and output ports are sufficient.
### Other ideas
- I think it maybe good idea to show "My macros" in the operator properties directly and let users drag a Marco operator similarly as dragging a normal operator out. This can save the changes on the property panel. Think about this: how many times would user select a marco (marker) and uses the drop down to change it to another macro implementation? I think if in any cases user need to use another macro, they should just drag it out from the operator menu.
Ok those are all my high level comments regarding this PR. I am sure we will have much more detailed comments going deeper.
So let's talk about breaking into smaller PRs. I still think this PR is too large: it has too many design choices that include creating a Marco, reuse a previously created macro, representation of a macro, editing a Marco, runtime behavior of the Marco. There are other life cycles which are not clear or being discussed: removing of a Marco, disable a macro, backend compilation of macro, persistence of macro (how to represent it in DB), search of a Marco (indexing) .... There are also other things we need to change, such as documentation, tests, tutorials of how to use Marco, etc.
The right size of a PR, is we can discuss and review for a single or two design choices. Making the scope smaller can make the discussion easier. For instance, I would recommend you have one PR about creating a macro (without saving it or reuse it), then we can discussion about the experience of creating a macro in the PR. Then you can have a PR to persist the Marco into db, we can discuss where to store macro information and how to make sure search can index it. Then you can have a PR to introduce experience of how user can reuse a previously stored macro in a new workflow. Then another PR to edit a previously macro. For each feature, you can split into frontend, backend to make size even smaller. I think you can see my point now. Making PRs smaller is not the goal: it is because smaller PRs can get proper reviews and discussions on your changes.
You may worry about that: these are partial features, can we merge into main? yes we can, just give a config that macro feature is turned off by default in main, so it doesn't affect any other features. After we have more PRs to implement the whole life cycle, fix bugs, add documentations, we then can turn on the feature by default to let user see and use it.
I hope this make sense to you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4532247449
This is a very important and powerful feature. I suggest we do a live discussion about the details and report the decision here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4559507342
Let's meet to discuss, and report the results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4529523372
BTW, @Xiao-zhen-Liu do you want to merge your version #5115 into main? I do like some of its experience. It will be great if we can unify those two experiences...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4528316128
> @mengw15 can I get your help to decide a split for this PR?
Thanks for the PR! I think this is a pretty important feature for Texera, so it may be worth having a broader discussion to align on the design and direction before moving forward.
I also remember that Xiaozhen worked on a related macro feature during the hackathon, for reference: https://github.com/apache/texera/pull/5115
cc @chenlica @Yicong-Huang @Xiao-zhen-Liu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4529138830
The direction is correct. We just need to split the backend part as a separate PR. The frontend part will be larger than regular PRs, but I think it's already the minimum unit and nonbreakable. It's manageable and reviewable. I can review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4529814833
Let's break them down to discuss one by one. It is not efficient to discuss all those design choices together.
I do hope to make hashjoin a macro operator in the long run. It should not require compilation to expand it: that was a tech debt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4547769290
I also think this topic deserves a in-person discussion. I implemented one version of Marco that has a quite different experience as shown in this demo, and I believe it is critical to have a good design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add workflow macro UI support [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5179:
URL: https://github.com/apache/texera/pull/5179#issuecomment-4528266271
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/5179?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 43.72%. Comparing base ([`0a42fcd`](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`655ae45`](https://app.codecov.io/gh/apache/texera/commit/655ae45d46366a689edec828000cdaf15e8f2a38?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #5179 +/- ##
============================================
+ Coverage 43.71% 43.72% +0.01%
Complexity 2217 2217
============================================
Files 1049 1049
Lines 40578 40525 -53
Branches 4327 4321 -6
============================================
- Hits 17739 17720 -19
+ Misses 21733 21700 -33
+ Partials 1106 1105 -1
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.53% <ø> (ø)` | | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.76% <ø> (ø)` | | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.20% <ø> (+0.03%)` | :arrow_up: | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.18% <ø> (ø)` | | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `35.15% <ø> (-0.03%)` | :arrow_down: | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [python](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.50% <ø> (ø)` | | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/5179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.81% <ø> (ø)` | | Carriedforward from [0a42fcd](https://app.codecov.io/gh/apache/texera/commit/0a42fcdbc20732ada9614bc77a0cfedd05cf6074?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/texera/pull/5179?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org