You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/05/27 07:08:15 UTC
[D] Discuss workflow macro design and lifecycle [texera]
GitHub user carloea2 created a discussion: Discuss workflow macro design and lifecycle
### Feature Summary
Workflow macros are an important feature for Texera, but the current discussion in #5179 mixes several design choices with implementation review. Based on the feedback from @Yicong-Huang and @chenlica, we should move the macro design discussion into an issue, hold a focused live discussion, and record the decisions here before continuing.
References:
- #5179: frontend-oriented workflow macro implementation
- #5178: original feature proposal for importing/grouping/collapsing reusable sub-workflows
- #5115: hackathon macro-operator prototype with a different UX
- #754: previous operator grouping/collapse behavior
### Proposed Solution or Design
Use this issue to align on the macro design before reviewing code in detail. The discussion should decide:
1. Macro creation UX
- Should users create a macro by selecting existing operators and choosing `Create Macro` from the context menu?
- Should users create a macro by importing an existing workflow?
- Should both flows be supported?
2. Macro representation and reuse
- Should a macro look and behave like a normal logical operator?
- Should macro input/output ports be derived from the internal sub-DAG?
- Should reusable macros appear directly in the operator menu, for example under `My Macros`?
- Should the long-term design also be able to represent operators such as Hash Join as macros?
3. Macro internal view and editing
- Should users edit macro internals in-place on the current canvas, or jump into a separate/drill-down workflow view?
- Are edits local to one workflow snapshot, or shared across all uses of the macro?
- Should macro internals expose explicit input/output ports instead of dummy source/sink operators?
4. Runtime and statistics behavior
- Should macro expansion remain frontend-only, with the backend executing the flattened operator graph?
- Should the backend be unaware of macros?
- When collapsed, should a macro show aggregated statistics, internal leaf-operator statistics, or allow users to choose?
5. Macro lifecycle and persistence
- How should macro definitions and macro instances be stored?
- How should users remove, rename, disable, search, share, or version macros?
- What documentation, tutorials, and tests are needed before enabling the feature by default?
### Impact / Priority
?
### Affected Area
- Workflow UI
- Workflow Engine (Amber)
- Storage / Metadata
- Documentation / Tutorials
GitHub link: https://github.com/apache/texera/discussions/5248
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
@chenlica @Yicong-Huang @Xiao-zhen-Liu @aglinxinyuan
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17072050
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
I think we can wait for @Xiao-zhen-Liu to come back since he raised the PR first, and his design was chosen, I could take his code and raise the PRs but I feel it is not right
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17200102
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
I did not request an offline discussion.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17347385
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
@carloea2 Please chime in. If needed, we can have a discussion and report the results here.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17536495
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Discuss workflow macro design and lifecycle
Let me convert this to a Discussion post. sorry if that's not clear. for uncertain topics, we use disucussions. once a consensus is reached, we open an issue (or feel free to use umbrella issue + multiple sub-issues) to track decided todos. Discussions can have back and forth ideas. issues should be clear on what exactly to do.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17072051
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Discuss workflow macro design and lifecycle
Please involve the community in the discussion, instead of "make an offline decision and inform the community".
If you had an offline discussion, please post more details so we can continue the discussion here.
Cc @chenlica @aglinxinyuan @Xiao-zhen-Liu please share more details.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17341175
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Discuss workflow macro design and lifecycle
We discussed this feature offline. We agreed that the design in #5115 is more general in that it has explicit encapsulation of the input and output ports, and that the user chooses which ports of the macro workflow is exposed.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17142331
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "yangzhang75 (via GitHub)" <gi...@apache.org>.
GitHub user yangzhang75 added a comment to the discussion: Discuss workflow macro design and lifecycle
After meeting with @carloea2, here's a summary of what we discussed.
The base macro feature (the #5115 direction — explicit input/output port encapsulation and in-place collapse) will be built first. Once it's merged, I can open separate PRs adding these extra features on top:
- Live preview for a macro — a zoomable mini-map of a collapsed macro's internal sub-DAG (with real operator icons), so you can see what's inside before expanding it.
- More flexible selection when creating a macro — a box/region ("snip") select to turn part of the canvas into a macro, in addition to selecting operators one by one.
- Macro gallery for browsing and reusing saved macros — this fits later, alongside the reuse/sharing phase (v1 doesn't include reuse).
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17586998
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
We had an offline discussion about this topic. @carloea2 : please summarize the results.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17303516
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 closed the discussion with a comment: Discuss workflow macro design and lifecycle
<img width="1200" height="1600" alt="image" src="https://github.com/user-attachments/assets/a703b82c-22fc-4ce3-b0ee-4e320537c22d" />
We have been exploring a revised approach to macros and would like to discuss it here before settling the design.
Some ideas we are considering:
- **Workflow and macro relationship:** At the execution level, workflows and macros are both operator graphs with the same engine capabilities. One possible design choice is to reserve standalone execution for workflows, even though a macro’s DAG may technically be executable independently.
- **Composition:** Under this model, a workflow would act as the entry point, similar to `main` in a program. It could contain macros, and macros could contain other macros, similar to functions calling other functions.
- **Macro representation:** A macro could represent a sub-DAG as a single operator, with exposed input/output ports connected to its internal graph.
- **UI experience:** Reusable macros could appear in the operator panel alongside regular operators. Xiaozhen’s UI could serve as the starting point for editing and navigating macro internals.
- **Independent versioning:** We are considering versioning workflows and macros independently, with macro instances referencing their definitions through pointers. For example, workflow A version 8 could reference macro B version 5.
- **Execution:** An initial approach could be to recursively expand macros in the frontend before submitting the graph for execution. The saved workflow would retain its hierarchy and references, while the execution engine would receive a flat operator graph. This should avoid engine changes, although persistence and APIs may still need work.
This would broaden the earlier proposal that deferred reuse. Whether these capabilities should be included initially or introduced in stages is also open for discussion.
Some questions we would like feedback on:
- **Overall model:** Does the distinction between an executable workflow and a reusable macro make sense? Are there use cases where allowing standalone macro execution would be valuable?
- **Export/import:** How should a workflow carry its macro dependencies when imported into another deployment? Would bundling the referenced definitions and versions be a suitable approach?
- **Form view:** How should exposed properties appear across nested macros? Which values should belong to the shared definition, and which should each instance be able to override?
- **Versioning:** Should references point to specific versions? How should users adopt updates, especially when ports or properties change?
These are starting points for discussion. Suggestions for alternative models, missing use cases, or a simpler initial scope would be welcome.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18788116
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "parshimers (via GitHub)" <gi...@apache.org>.
GitHub user parshimers added a comment to the discussion: Discuss workflow macro design and lifecycle
I'm glad this got revisited. I didn't follow the earlier discussion. I'm glad it led to form views, but I think macros are a separate and useful thing.
I agree with the idea of treating macros as a kind of special workflow, which cannot be run on its own. I think the simple inlining of those workflows into whatever workflow they are used in, as a special kind of operator, is a very common and natural thing. AsterixDB's SQL++ UDFs are implemented in exactly the same way; they are simply queries which get written into some other query where the function appears in the main query.
I think a workflow that references a macro should always refer to it with a reference, and only resolve that reference at compile time. It really can only be the compiler that can take ownership of this part.
The dependencies are indeed a tricky problem. I think at least for "pure" operators it's fine, but for UDFs it can get complicated. I think it might be easier to constrain those out of the design for now if it complicates it.
I don't think a property within a macro should be able to live outside of the context of the macro. It's effectively an operator, after all, and having other workflow side effects seem kind of strange to me.
P.S. (Mentor hat): I noticed this thread kind of devolved into a series of offline discussions, and while the latest post did an OK job of trying to share the state and loop everyone else in, the rest of the thread is really an antipattern. The point of discussing it here is to make the discussion entirely open, and not to agree on when closed discussions should or shouldn't take place. It can't be the case that discussions end up as a dumping ground for designs which have already been agreed to a priori by a clique of contributors before ever being brought out into the open, basically as an afterthought.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18802121
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub) via dev" <de...@texera.apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
@chenlica, the tracking issue is #8913 and the implementation is now available in draft PR #9541. The PR has conflicts with current `main`, so it is ready for design feedback while integration is pending.
One change since my October proposal addresses @parshimers’s point about where references are resolved. A workflow saves each macro’s revision, and the Scala compiler expands those pins when compiling the workflow. The saved hierarchy lets you open either of two usages of the same macro and see that occurrence’s errors and per-port statistics, including after reopening a run. We tested that with five usages of one revision at three depths.
For the lifecycle questions above, editing through a parent updates that occurrence’s path. Other usages keep their saved revisions. Form View exposes fields explicitly, with defaults in the definition and overrides on the instance. A workflow export carries the nested macro definitions and revisions.
Two questions where feedback would help: which external dependencies, such as files or UDF environments, should an import identify for the user? And for @sshiv012’s sample-input suggestion, would a small test workflow that calls the macro meet your needs, or is there a case where you need to test it directly from the macro editor?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-19728818
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
In that case, we can just wait for @chenlica and an internal meeting to decide and report the decision here.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17072232
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "yangzhang75 (via GitHub)" <gi...@apache.org>.
GitHub user yangzhang75 added a comment to the discussion: Discuss workflow macro design and lifecycle
Hi, I was suggested to fold my recent proposal (#6066, now closed) into this discussion.
I've been prototyping something adjacent to macros: a "Blocks" gallery for saving and reusing workflow pieces across workflows. Reading through this thread, my design is different in direction (drop-in reuse vs. in-place collapsing). But a few pieces from the prototype might be useful for the macros work:
- Built-in Sample macros (a small read-only starter set in the panel, for onboarding)
- Mini-map preview of each macro (SVG rendering with real operator icons)
- Snip-region save (box-select a sub-region of the canvas, save only that part as a macro instead of the whole workflow)
- Apply-to-canvas logic (fresh UUIDs, remap links, place on empty area)
Demo of the prototype: https://drive.google.com/file/d/12pPeBUvbLlsrO5t5pp0C9G2xeEordTD2/view?usp=sharing
Would any of these be useful to fold into the macros direction? Thank you!
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17533433
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 edited a comment on the discussion: Discuss workflow macro design and lifecycle
<img width="1200" height="1600" alt="image" src="https://github.com/user-attachments/assets/a703b82c-22fc-4ce3-b0ee-4e320537c22d" />
We have been exploring a revised approach to macros and would like to discuss it here before settling the design.
Some ideas we are considering:
1. - **Workflow and macro relationship:** At the execution level, workflows and macros are both operator graphs with the same engine capabilities. One possible design choice is to reserve standalone execution for workflows, even though a macro’s DAG may technically be executable independently.
2. - **Composition:** Under this model, a workflow would act as the entry point, similar to `main` in a program. It could contain macros, and macros could contain other macros, similar to functions calling other functions.
3. - **Macro representation:** A macro could represent a sub-DAG as a single operator, with exposed input/output ports connected to its internal graph.
4. - **UI experience:** Reusable macros could appear in the operator panel alongside regular operators. Xiaozhen’s UI could serve as the starting point for editing and navigating macro internals. https://www.youtube.com/watch?v=ZYQIoPiEIfo&feature=youtu.be
5. - **Independent versioning:** We are considering versioning workflows and macros independently, with macro instances referencing their definitions through pointers. For example, workflow A version 8 could reference macro B version 5.
6. - **Execution:** An initial approach could be to recursively expand macros in the frontend before submitting the graph for execution. The saved workflow would retain its hierarchy and references, while the execution engine would receive a flat operator graph. This should avoid engine changes, although persistence and APIs may still need work.
This would broaden the earlier proposal that deferred reuse. Whether these capabilities should be included initially or introduced in stages is also open for discussion.
Some questions we would like feedback on:
1. **Overall model:** Does the distinction between an executable workflow and a reusable macro make sense? Are there use cases where allowing standalone macro execution would be valuable?
2. **Export/import:** How should a workflow carry its macro dependencies when imported into another deployment? Would bundling the referenced definitions and versions be a suitable approach?
3. **Form view:** How should exposed properties appear across nested macros? Which values should belong to the shared definition, and which should each instance be able to override?
4. **Versioning:** Should references point to specific versions? How should users adopt updates, especially when ports or properties change?
These are starting points for discussion. Suggestions for alternative models, missing use cases, or a simpler initial scope would be welcome.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18788116
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 edited a comment on the discussion: Discuss workflow macro design and lifecycle
<img width="1200" height="1600" alt="image" src="https://github.com/user-attachments/assets/a703b82c-22fc-4ce3-b0ee-4e320537c22d" />
We have been exploring a revised approach to macros and would like to discuss it here before settling the design.
Some ideas we are considering:
1. - **Workflow and macro relationship:** At the execution level, workflows and macros are both operator graphs with the same engine capabilities. One possible design choice is to reserve standalone execution for workflows, even though a macro’s DAG may technically be executable independently.
2. - **Composition:** Under this model, a workflow would act as the entry point, similar to `main` in a program. It could contain macros, and macros could contain other macros, similar to functions calling other functions.
3. - **Macro representation:** A macro could represent a sub-DAG as a single operator, with exposed input/output ports connected to its internal graph.
4. - **UI experience:** Reusable macros could appear in the operator panel alongside regular operators. Xiaozhen’s UI could serve as the starting point for editing and navigating macro internals.
5. - **Independent versioning:** We are considering versioning workflows and macros independently, with macro instances referencing their definitions through pointers. For example, workflow A version 8 could reference macro B version 5.
6. - **Execution:** An initial approach could be to recursively expand macros in the frontend before submitting the graph for execution. The saved workflow would retain its hierarchy and references, while the execution engine would receive a flat operator graph. This should avoid engine changes, although persistence and APIs may still need work.
This would broaden the earlier proposal that deferred reuse. Whether these capabilities should be included initially or introduced in stages is also open for discussion.
Some questions we would like feedback on:
1. **Overall model:** Does the distinction between an executable workflow and a reusable macro make sense? Are there use cases where allowing standalone macro execution would be valuable?
2. **Export/import:** How should a workflow carry its macro dependencies when imported into another deployment? Would bundling the referenced definitions and versions be a suitable approach?
3. **Form view:** How should exposed properties appear across nested macros? Which values should belong to the shared definition, and which should each instance be able to override?
4. **Versioning:** Should references point to specific versions? How should users adopt updates, especially when ports or properties change?
These are starting points for discussion. Suggestions for alternative models, missing use cases, or a simpler initial scope would be welcome.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18788116
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
Ok.
1. Frontend will do the expansion, If we have workflow 1 with macro A, and macro A has macro B, then the whole hierarchical expansion will be done on frontend, and the backend does not need to be aware of this.
2. We agreed to show the macro UI as in Xiaozhen demo and if I remember correctly to add buttons to add/remove inputs
3. We have not defined details of what the frontend should show about statistics.
A general question: if the goal is to meaningfully involve the broader community, shouldn’t the whole design and decision-making discussions happen in a public, async, and searchable place like GitHub?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17347729
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Discuss workflow macro design and lifecycle
Hi @carloea2 could you please elaborate more? Thanks.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17327259
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Discuss workflow macro design and lifecycle
I don't think we're on the same page. Let's have a group discussion next Monday. If we cannot finish, we can have one more discussion on Wednesday.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17200136
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "sshiv012 (via GitHub)" <gi...@apache.org>.
GitHub user sshiv012 added a comment to the discussion: Discuss workflow macro design and lifecycle
I support treating macros as reusable subgraphs with explicit ports and independently pinned versions. [KNIME components](https://docs.knime.com/ap/latest/analytics_platform_components_guide/) (I believe this was considered as an inspiration) provide a useful precedent for separating local grouping from shared components and adopting updates explicitly.
I think having the below might be nice,
- Allow macros to be tested with sample inputs, even if only workflows are independently executable
- Keep defaults in the definition and overrides in each instance; expose nested properties explicitly.
- Frontend expansion seems reasonable initially, but preserve the hierarchy mapping for errors and statistics, and maybe leave a path towards server-side expansion?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18806035
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "sshiv012 (via GitHub) via dev" <de...@texera.apache.org>.
GitHub user sshiv012 added a comment to the discussion: Discuss workflow macro design and lifecycle
Thanks for the update! A small test workflow calling the macro would be okay. Testing from the macro editor would mainly be a _convenience_ for quickly trying sample inputs while editing. It can be added as a follow-up as well.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-19759542
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 edited a comment on the discussion: Discuss workflow macro design and lifecycle
Ok.
1. Frontend will do the expansion, If we have workflow 1 with macro A, and macro A has macro B, then the whole hierarchical expansion will be done on frontend, and the backend does not need to be aware of this.
2. We agreed to show the macro UI as in Xiaozhen demo and if I remember correctly and introduce buttons to add/remove inputs/outputs.
3. We have not defined details of what the frontend should show about statistics.
A general question: if the goal is to meaningfully involve the broader community, shouldn’t the whole design and decision-making discussions happen in a public, async, and searchable place like GitHub?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17347729
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 edited a comment on the discussion: Discuss workflow macro design and lifecycle
There are no more details at the moment, we just agreed on the user experience.
Design Video for reference https://github.com/apache/texera/pull/5115
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17333500
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Discuss workflow macro design and lifecycle
I shared my preferences in #5179 already. If you are having an offline discussion, please report back here.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17072189
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
@carloea2 Let's do a meeting to discuss the details and report the results here.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17077386
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 reopened a discussion: Discuss workflow macro design and lifecycle
### Feature Summary
Workflow macros are an important feature for Texera, but the current discussion in #5179 mixes several design choices with implementation review. Based on the feedback from @Yicong-Huang and @chenlica, we should move the macro design discussion into an issue, hold a focused live discussion, and record the decisions here before continuing.
References:
- #5179: frontend-oriented workflow macro implementation
- #5178: original feature proposal for importing/grouping/collapsing reusable sub-workflows
- #5115: hackathon macro-operator prototype with a different UX
- #754: previous operator grouping/collapse behavior
### Proposed Solution or Design
Use this issue to align on the macro design before reviewing code in detail. The discussion should decide:
1. Macro creation UX
- Should users create a macro by selecting existing operators and choosing `Create Macro` from the context menu?
- Should users create a macro by importing an existing workflow?
- Should both flows be supported?
2. Macro representation and reuse
- Should a macro look and behave like a normal logical operator?
- Should macro input/output ports be derived from the internal sub-DAG?
- Should reusable macros appear directly in the operator menu, for example under `My Macros`?
- Should the long-term design also be able to represent operators such as Hash Join as macros?
3. Macro internal view and editing
- Should users edit macro internals in-place on the current canvas, or jump into a separate/drill-down workflow view?
- Are edits local to one workflow snapshot, or shared across all uses of the macro?
- Should macro internals expose explicit input/output ports instead of dummy source/sink operators?
4. Runtime and statistics behavior
- Should macro expansion remain frontend-only, with the backend executing the flattened operator graph?
- Should the backend be unaware of macros?
- When collapsed, should a macro show aggregated statistics, internal leaf-operator statistics, or allow users to choose?
5. Macro lifecycle and persistence
- How should macro definitions and macro instances be stored?
- How should users remove, rename, disable, search, share, or version macros?
- What documentation, tutorials, and tests are needed before enabling the feature by default?
### Impact / Priority
?
### Affected Area
- Workflow UI
- Workflow Engine (Amber)
- Storage / Metadata
- Documentation / Tutorials
GitHub link: https://github.com/apache/texera/discussions/5248
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica edited a comment on the discussion: Discuss workflow macro design and lifecycle
@carloea2 Thanks for the summary. Not sure if I fully understand your question. I believe we are allowed to do offline discussions for efficiency reasons, but we need to publish the discussed proposals to the community to maintain the transparency.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17360112
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
@carloea2 Please provide more details as suggested.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17347260
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "zuozhiw (via GitHub)" <gi...@apache.org>.
GitHub user zuozhiw added a comment to the discussion: Discuss workflow macro design and lifecycle
I saw the demo video and I would have absolute no problem and will be very happy if we can ship #5115 as-is
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17198755
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
Do we have any plan to finish this important feature?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17147160
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 closed a discussion: Discuss workflow macro design and lifecycle
### Feature Summary
Workflow macros are an important feature for Texera, but the current discussion in #5179 mixes several design choices with implementation review. Based on the feedback from @Yicong-Huang and @chenlica, we should move the macro design discussion into an issue, hold a focused live discussion, and record the decisions here before continuing.
References:
- #5179: frontend-oriented workflow macro implementation
- #5178: original feature proposal for importing/grouping/collapsing reusable sub-workflows
- #5115: hackathon macro-operator prototype with a different UX
- #754: previous operator grouping/collapse behavior
### Proposed Solution or Design
Use this issue to align on the macro design before reviewing code in detail. The discussion should decide:
1. Macro creation UX
- Should users create a macro by selecting existing operators and choosing `Create Macro` from the context menu?
- Should users create a macro by importing an existing workflow?
- Should both flows be supported?
2. Macro representation and reuse
- Should a macro look and behave like a normal logical operator?
- Should macro input/output ports be derived from the internal sub-DAG?
- Should reusable macros appear directly in the operator menu, for example under `My Macros`?
- Should the long-term design also be able to represent operators such as Hash Join as macros?
3. Macro internal view and editing
- Should users edit macro internals in-place on the current canvas, or jump into a separate/drill-down workflow view?
- Are edits local to one workflow snapshot, or shared across all uses of the macro?
- Should macro internals expose explicit input/output ports instead of dummy source/sink operators?
4. Runtime and statistics behavior
- Should macro expansion remain frontend-only, with the backend executing the flattened operator graph?
- Should the backend be unaware of macros?
- When collapsed, should a macro show aggregated statistics, internal leaf-operator statistics, or allow users to choose?
5. Macro lifecycle and persistence
- How should macro definitions and macro instances be stored?
- How should users remove, rename, disable, search, share, or version macros?
- What documentation, tutorials, and tests are needed before enabling the feature by default?
### Impact / Priority
?
### Affected Area
- Workflow UI
- Workflow Engine (Amber)
- Storage / Metadata
- Documentation / Tutorials
GitHub link: https://github.com/apache/texera/discussions/5248
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
Thanks for the summary. It's based on an offline discussion by @carloea2 @aglinxinyuan @Xiao-zhen-Liu @mengw15 @parshimers @sshiv012 and @chenlica .
For "idea 1," a macro is viewed as a workflow that cannot run by itself, similar to a non-main function in programming languages.
For "idea 4," please add a link to "Xiaozhen’s UI."
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18788588
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "sshiv012 (via GitHub)" <gi...@apache.org>.
GitHub user sshiv012 edited a comment on the discussion: Discuss workflow macro design and lifecycle
I support treating macros as reusable subgraphs with explicit ports and independently pinned versions.
I think having the below might be nice,
- Allow macros to be tested with sample inputs, even if only workflows are independently executable
- Keep defaults in the definition and overrides in each instance; expose nested properties explicitly.
- Frontend expansion seems reasonable initially, but preserve the hierarchy mapping for errors and statistics, and maybe leave a path towards server-side expansion?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18806035
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub) via dev" <de...@texera.apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
@carloea2 If you have issues and PRs related to this discussion, please include them in this discussion.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-19702835
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
@carloea2 Thanks for the summary. Not sure if I fully understand your question. I believe we are allowed to do offline discussions for efficiency reasons, but we need to publish the results and decisions here to maintain the transparency.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17360112
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
In summary:
- We will keep the first implementation simple.
- First version will not allow reusability of a macro or sharing.
- The main changes will be frontend related according to the experience Xiaozhen designed, by adding some properties to the workflow json.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17316303
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
There are no more details at the moment, we just agreed on the user experience.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17333500
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
Maybe @carloea2 can take the lead to finish it? Notice that @Xiao-zhen-Liu's availability in the summer is limited. It will be good to have a discussion before that, and report the plan here.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17200005
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Discuss workflow macro design and lifecycle
@yangzhang75 do you want to have a zoom meeting so we can understand your proposal and design in comparison with the one presented here?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17539924
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "yangzhang75 (via GitHub)" <gi...@apache.org>.
GitHub user yangzhang75 added a comment to the discussion: Discuss workflow macro design and lifecycle
Sure. What times work for you this week?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17540800
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
I suggest we do not do this type "scheduling meeting" messages on GitHub.
@Yicong-Huang : what's a good practice for such messages?
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-17543309
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Discuss workflow macro design and lifecycle [texera]
Posted by "chenlica (via GitHub) via dev" <de...@texera.apache.org>.
GitHub user chenlica added a comment to the discussion: Discuss workflow macro design and lifecycle
@carloea2 Please give an update.
GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-19466062
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org