You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "zyratlo (via GitHub)" <gi...@apache.org> on 2026/10/07 18:49:26 UTC
[PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
zyratlo opened a new pull request, #8910:
URL: https://github.com/apache/texera/pull/8910
### What changes were proposed in this PR?
This is the first of three PRs adding R support to the migration tool (#8009). It is a pure refactor with no behavior change: Python conversions send the same prompts and build the same operators as before.
The migration tool hardcodes Python in three places that R needs to vary: the documentation and prompts sent to the model, the operator type built from each generated UDF (`PythonUDFV2`), and the kernelspec of the notebook derived for a script. This PR gathers those into one `MigrationPromptSet` per source language, held in a `PROMPT_SETS` table that only has a Python entry for now.
* `convertNotebookToWorkflow` and `convertScriptToWorkflow` take a `language` argument, defaulting to `"python"`, and read everything from the set.
* `buildWorkflow`, `requestConversion` and `finishConversion` receive the operator type and notebook metadata as arguments instead of hardcoding them.
* The three Python documentation arrays move from private statics on the class to module level constants, unchanged, so the table can reference them.
* The folder path uses the Python set explicitly. R folders come in a later PR.
`MigrationLanguage` is `"python"` only in this PR. The next PR widens it to `"python" | "r"` and adds the R set.
### Any related issues, documentation, discussions?
Part of #8009
Parent-issue #4301
The plan for R support is in https://github.com/apache/texera/discussions/8866. The follow up PRs add the R prompts and the R option in the import modal. They are stacked on this one.
### How was this PR tested?
Three new tests in `migration-llm.spec.ts` pin the Python output so later changes cannot alter it silently:
* The notebook path sends exactly the existing system prelude, workflow prompt and mapping prompt.
* The script path does the same with the script prelude and prompts.
* Generated operators are built as `PythonUDFV2`.
The existing notebook, script and folder tests pass unchanged. Coverage of `migration-llm.ts` is 100% of lines; the one uncovered branch is the `@Injectable()` decorator, as before this change. The full frontend suite passes.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4288391840
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.spec.ts:
##########
@@ -316,6 +331,74 @@ describe("NotebookMigrationLLM", () => {
});
});
+ describe("Python prompts", () => {
+ const workflowResponse = JSON.stringify({ code: { UDF1: "# UDF1" }, edges: [], outputs: {} });
+ const documentation = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+ ];
+
+ function contentsFor(role: string): string[] {
+ return (callModelSpy.mock.calls[0][0] as { role: string; content: string }[])
+ .filter(message => message.role === role)
+ .map(message => message.content);
+ }
+
+ it("sends the notebook prelude and prompts unchanged", async () => {
+ mockResponses(workflowResponse, JSON.stringify({ UDF1: ["CELL1"] }));
+
+ await makeLLM().convertNotebookToWorkflow({ cells: [codeCell("CELL1", "print(1)")] });
+
+ expect(contentsFor("system")).toEqual(documentation);
+ const [workflowPrompt, mappingPrompt] = contentsFor("user");
+ expect(workflowPrompt.startsWith(`${WORKFLOW_PROMPT}\n`)).toBe(true);
+ expect(mappingPrompt).toBe(MAPPING_PROMPT);
+ });
+
+ it("sends the script prelude and prompts unchanged", async () => {
+ mockResponses(workflowResponse, JSON.stringify({ UDF1: [[1, 1]] }));
+
+ await makeLLM().convertScriptToWorkflow("print(1)");
+
+ expect(contentsFor("system")).toEqual(
+ documentation.map(doc =>
+ doc === EXAMPLE_OF_MULTIPLE_UDF_CONVERSION ? EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT : doc
+ )
+ );
+ const [workflowPrompt, mappingPrompt] = contentsFor("user");
+ expect(workflowPrompt.startsWith(`${SCRIPT_WORKFLOW_PROMPT}\n`)).toBe(true);
+ expect(mappingPrompt).toBe(SCRIPT_MAPPING_PROMPT);
Review Comment:
contentsFor()` always inspects `callModelSpy.mock.calls[0]` (the first model invocation). Since conversions call the model at least twice (workflow prompt then mapping prompt), the first call’s message list typically won’t include the mapping prompt—making `mappingPrompt` `undefined` and the assertions incorrect/flaky. Consider either (a) updating `contentsFor` to accept a call index and asserting workflow prompt from call 0 and mapping prompt from call 1, or (b) using the existing conversation helper (seen elsewhere in this spec) that already supports selecting which call to inspect.
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.spec.ts:
##########
@@ -316,6 +331,74 @@ describe("NotebookMigrationLLM", () => {
});
});
+ describe("Python prompts", () => {
+ const workflowResponse = JSON.stringify({ code: { UDF1: "# UDF1" }, edges: [], outputs: {} });
+ const documentation = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+ ];
+
+ function contentsFor(role: string): string[] {
+ return (callModelSpy.mock.calls[0][0] as { role: string; content: string }[])
+ .filter(message => message.role === role)
+ .map(message => message.content);
+ }
Review Comment:
contentsFor()` always inspects `callModelSpy.mock.calls[0]` (the first model invocation). Since conversions call the model at least twice (workflow prompt then mapping prompt), the first call’s message list typically won’t include the mapping prompt—making `mappingPrompt` `undefined` and the assertions incorrect/flaky. Consider either (a) updating `contentsFor` to accept a call index and asserting workflow prompt from call 0 and mapping prompt from call 1, or (b) using the existing conversation helper (seen elsewhere in this spec) that already supports selecting which call to inspect.
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -111,15 +111,66 @@ function toDerivedNotebook(cells: DerivedCell[]): Notebook {
outputs: [],
execution_count: null,
})),
- metadata: {
- kernelspec: { display_name: "Python 3", language: "python", name: "python3" },
- language_info: { name: "python" },
- },
+ metadata: structuredClone(metadata),
Review Comment:
structuredClone` is not available in all runtime environments/browsers without a polyfill; if the frontend’s supported browser matrix includes older Safari/Chromium versions, this will throw at runtime when deriving notebooks from scripts/folders. If broad browser support is required, consider switching to an existing project-standard deep-clone utility (or a guarded fallback) for cloning notebook metadata.
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.spec.ts:
##########
@@ -316,6 +331,74 @@ describe("NotebookMigrationLLM", () => {
});
});
+ describe("Python prompts", () => {
+ const workflowResponse = JSON.stringify({ code: { UDF1: "# UDF1" }, edges: [], outputs: {} });
+ const documentation = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+ ];
+
+ function contentsFor(role: string): string[] {
+ return (callModelSpy.mock.calls[0][0] as { role: string; content: string }[])
+ .filter(message => message.role === role)
+ .map(message => message.content);
+ }
+
+ it("sends the notebook prelude and prompts unchanged", async () => {
+ mockResponses(workflowResponse, JSON.stringify({ UDF1: ["CELL1"] }));
+
+ await makeLLM().convertNotebookToWorkflow({ cells: [codeCell("CELL1", "print(1)")] });
+
+ expect(contentsFor("system")).toEqual(documentation);
+ const [workflowPrompt, mappingPrompt] = contentsFor("user");
+ expect(workflowPrompt.startsWith(`${WORKFLOW_PROMPT}\n`)).toBe(true);
+ expect(mappingPrompt).toBe(MAPPING_PROMPT);
Review Comment:
contentsFor()` always inspects `callModelSpy.mock.calls[0]` (the first model invocation). Since conversions call the model at least twice (workflow prompt then mapping prompt), the first call’s message list typically won’t include the mapping prompt—making `mappingPrompt` `undefined` and the assertions incorrect/flaky. Consider either (a) updating `contentsFor` to accept a call index and asserting workflow prompt from call 0 and mapping prompt from call 1, or (b) using the existing conversation helper (seen elsewhere in this spec) that already supports selecting which call to inspect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4330327678
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -111,15 +111,66 @@ function toDerivedNotebook(cells: DerivedCell[]): Notebook {
outputs: [],
execution_count: null,
})),
- metadata: {
- kernelspec: { display_name: "Python 3", language: "python", name: "python3" },
- language_info: { name: "python" },
- },
+ metadata: structuredClone(metadata),
Review Comment:
No change needed. The frontend's browserslist (`defaults`, minus IE) resolves to Chrome 109+, Firefox 140+, Edge 145+, Safari 26.3+ and iOS Safari 18.5+. `structuredClone` has been available in all of them since 2022 (Chrome 98, Firefox 94, Safari 15.4), so it cannot throw in any supported browser.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4391200272
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -444,10 +481,12 @@ export class NotebookMigrationLLM {
throw new Error("LLM session not initialized");
}
+ const prompts = PROMPT_SETS.python;
const { workflowJSON, udfIdToOperatorId, workflowResponse, reportedRanges } = await this.requestConversion(
- NotebookMigrationLLM.FOLDER_DOCUMENTATION,
+ PYTHON_FOLDER_DOCUMENTATION,
FOLDER_WORKFLOW_PROMPT,
FOLDER_MAPPING_PROMPT,
+ prompts.operatorType,
Review Comment:
Kept out on purpose. The folder fields join the prompt set in the R folders PR, the first point where a second language fills them in. Adding them now would need an optional group and a "no folder prompts" error that only exist until then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4252217002
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -111,15 +111,66 @@ function toDerivedNotebook(cells: DerivedCell[]): Notebook {
outputs: [],
execution_count: null,
})),
- metadata: {
- kernelspec: { display_name: "Python 3", language: "python", name: "python3" },
- language_info: { name: "python" },
- },
+ metadata,
Review Comment:
`PROMPT_SETS` owns one mutable metadata object, and this returns that same reference from every script/folder conversion. Mutating one returned notebook's nested metadata will therefore change later conversion results; before this refactor, each derived notebook received a fresh metadata object. Clone it when constructing the notebook to preserve conversion isolation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4391230927
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -111,15 +111,66 @@ function toDerivedNotebook(cells: DerivedCell[]): Notebook {
outputs: [],
execution_count: null,
})),
- metadata: {
- kernelspec: { display_name: "Python 3", language: "python", name: "python3" },
- language_info: { name: "python" },
- },
+ metadata: structuredClone(metadata),
nbformat: 4,
nbformat_minor: 4,
};
}
+const PYTHON_NOTEBOOK_DOCUMENTATION: string[] = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+];
+
+// The script prelude differs in exactly one entry. The notebook worked example is written in
+// `# START CELL1` form, and a system-message example of that weight would push the model to
+// answer in cell ids for an input that has no cells. The notebook array is left untouched so
+// existing conversions see byte-identical context.
+const PYTHON_SCRIPT_DOCUMENTATION: string[] = PYTHON_NOTEBOOK_DOCUMENTATION.map(doc =>
+ doc === EXAMPLE_OF_MULTIPLE_UDF_CONVERSION ? EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT : doc
+);
+
+// Differs from the script prelude in the same single entry: its worked example shows banner
+// lines, an entry point calling into other files, and definitions inlined rather than imported.
+const PYTHON_FOLDER_DOCUMENTATION: string[] = PYTHON_SCRIPT_DOCUMENTATION.map(doc =>
+ doc === EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT ? EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_FOLDER : doc
+);
+
+export type MigrationLanguage = "python";
+
+interface MigrationPromptSet {
+ notebookDocumentation: string[];
+ notebookWorkflowPrompt: string;
+ notebookMappingPrompt: string;
+ scriptDocumentation: string[];
+ scriptWorkflowPrompt: string;
+ scriptMappingPrompt: string;
+ operatorType: string;
Review Comment:
The frontend has no compile time operator registry. Operator types come from backend metadata at runtime, and `OperatorPredicate["operatorType"]` is `string`. The one literal per language is covered by tests: the spec asserts `PythonUDFV2` for Python, and the R PR's specs assert `RUDF`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8910:
URL: https://github.com/apache/texera/pull/8910#issuecomment-6044649407
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- No candidates found from `git blame` history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4330335986
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.spec.ts:
##########
@@ -316,6 +331,74 @@ describe("NotebookMigrationLLM", () => {
});
});
+ describe("Python prompts", () => {
+ const workflowResponse = JSON.stringify({ code: { UDF1: "# UDF1" }, edges: [], outputs: {} });
+ const documentation = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+ ];
+
+ function contentsFor(role: string): string[] {
+ return (callModelSpy.mock.calls[0][0] as { role: string; content: string }[])
+ .filter(message => message.role === role)
+ .map(message => message.content);
+ }
Review Comment:
The tests are correct. Every model call receives the same shared `messages` array, so the first call's argument already holds the mapping prompt when the test reads it. Added a comment on the helper saying so. Added a comment in [55f56c8](https://github.com/apache/texera/pull/8910/commits/55f56c89942e529be47869c9582c290dae833375) so that this is cleared up
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.spec.ts:
##########
@@ -316,6 +331,74 @@ describe("NotebookMigrationLLM", () => {
});
});
+ describe("Python prompts", () => {
+ const workflowResponse = JSON.stringify({ code: { UDF1: "# UDF1" }, edges: [], outputs: {} });
+ const documentation = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+ ];
+
+ function contentsFor(role: string): string[] {
+ return (callModelSpy.mock.calls[0][0] as { role: string; content: string }[])
+ .filter(message => message.role === role)
+ .map(message => message.content);
+ }
+
+ it("sends the notebook prelude and prompts unchanged", async () => {
+ mockResponses(workflowResponse, JSON.stringify({ UDF1: ["CELL1"] }));
+
+ await makeLLM().convertNotebookToWorkflow({ cells: [codeCell("CELL1", "print(1)")] });
+
+ expect(contentsFor("system")).toEqual(documentation);
+ const [workflowPrompt, mappingPrompt] = contentsFor("user");
+ expect(workflowPrompt.startsWith(`${WORKFLOW_PROMPT}\n`)).toBe(true);
+ expect(mappingPrompt).toBe(MAPPING_PROMPT);
Review Comment:
The tests are correct. Every model call receives the same shared `messages` array, so the first call's argument already holds the mapping prompt when the test reads it. Added a comment on the helper saying so. Added a comment in [55f56c8](https://github.com/apache/texera/pull/8910/commits/55f56c89942e529be47869c9582c290dae833375) so that this is cleared up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 merged PR #8910:
URL: https://github.com/apache/texera/pull/8910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8910:
URL: https://github.com/apache/texera/pull/8910#issuecomment-6045059188
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8910?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 92.59%. Comparing base ([`4c982a7`](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8a287a2`](https://app.codecov.io/gh/apache/texera/commit/8a287a20622160fde87bca8e3fe013fea88139e6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8910 +/- ##
============================================
- Coverage 92.60% 92.59% -0.01%
Complexity 5044 5044
============================================
Files 1252 1252
Lines 53580 53586 +6
Branches 6672 6673 +1
============================================
+ Hits 49616 49619 +3
Misses 2317 2317
- Partials 1647 1650 +3
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.38% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.16% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.09% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.48% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.37% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.53% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.58% <100.00%> (-0.02%)` | :arrow_down: | |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.73% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.52% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8910/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.09% <ø> (ø)` | | Carriedforward from [4c982a7](https://app.codecov.io/gh/apache/texera/commit/4c982a7a8abba119b326b19456f4ef7fda78f059?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8910?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4330339106
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.spec.ts:
##########
@@ -316,6 +331,74 @@ describe("NotebookMigrationLLM", () => {
});
});
+ describe("Python prompts", () => {
+ const workflowResponse = JSON.stringify({ code: { UDF1: "# UDF1" }, edges: [], outputs: {} });
+ const documentation = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+ ];
+
+ function contentsFor(role: string): string[] {
+ return (callModelSpy.mock.calls[0][0] as { role: string; content: string }[])
+ .filter(message => message.role === role)
+ .map(message => message.content);
+ }
+
+ it("sends the notebook prelude and prompts unchanged", async () => {
+ mockResponses(workflowResponse, JSON.stringify({ UDF1: ["CELL1"] }));
+
+ await makeLLM().convertNotebookToWorkflow({ cells: [codeCell("CELL1", "print(1)")] });
+
+ expect(contentsFor("system")).toEqual(documentation);
+ const [workflowPrompt, mappingPrompt] = contentsFor("user");
+ expect(workflowPrompt.startsWith(`${WORKFLOW_PROMPT}\n`)).toBe(true);
+ expect(mappingPrompt).toBe(MAPPING_PROMPT);
+ });
+
+ it("sends the script prelude and prompts unchanged", async () => {
+ mockResponses(workflowResponse, JSON.stringify({ UDF1: [[1, 1]] }));
+
+ await makeLLM().convertScriptToWorkflow("print(1)");
+
+ expect(contentsFor("system")).toEqual(
+ documentation.map(doc =>
+ doc === EXAMPLE_OF_MULTIPLE_UDF_CONVERSION ? EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT : doc
+ )
+ );
+ const [workflowPrompt, mappingPrompt] = contentsFor("user");
+ expect(workflowPrompt.startsWith(`${SCRIPT_WORKFLOW_PROMPT}\n`)).toBe(true);
+ expect(mappingPrompt).toBe(SCRIPT_MAPPING_PROMPT);
Review Comment:
The tests are correct. Every model call receives the same shared `messages` array, so the first call's argument already holds the mapping prompt when the test reads it. Added a comment on the helper saying so. Added a comment in [55f56c8](https://github.com/apache/texera/pull/8910/commits/55f56c89942e529be47869c9582c290dae833375) so that this is cleared up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on PR #8910:
URL: https://github.com/apache/texera/pull/8910#issuecomment-6044592419
/request-review @mengw15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4287604015
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -111,15 +111,66 @@ function toDerivedNotebook(cells: DerivedCell[]): Notebook {
outputs: [],
execution_count: null,
})),
- metadata: {
- kernelspec: { display_name: "Python 3", language: "python", name: "python3" },
- language_info: { name: "python" },
- },
+ metadata,
Review Comment:
Fixed in [4a1f3bd](https://github.com/apache/texera/pull/8910/commits/4a1f3bd21e412c5c34dc112286efe6ca6c8f4bda). `toDerivedNotebook` now clones the prompt set's metadata, so each derived notebook gets its own copy, as it did before the refactor. A new test mutates one conversion's metadata and checks that the next conversion is unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(notebook-migration): select prompts and operator type by source language [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8910:
URL: https://github.com/apache/texera/pull/8910#discussion_r4364580152
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -444,10 +481,12 @@ export class NotebookMigrationLLM {
throw new Error("LLM session not initialized");
}
+ const prompts = PROMPT_SETS.python;
const { workflowJSON, udfIdToOperatorId, workflowResponse, reportedRanges } = await this.requestConversion(
- NotebookMigrationLLM.FOLDER_DOCUMENTATION,
+ PYTHON_FOLDER_DOCUMENTATION,
FOLDER_WORKFLOW_PROMPT,
FOLDER_MAPPING_PROMPT,
+ prompts.operatorType,
Review Comment:
With `MigrationPromptSet` introduced to centralize language-specific variations, the folder conversion path still hardcodes Python documentation and prompts outside the prompt set. This leaves an additional place to update when adding R folder support. Consider adding `folderDocumentation`, `folderWorkflowPrompt`, and `folderMappingPrompt` to `MigrationPromptSet` now (even if only the Python entry is populated) and routing this call through `prompts` for consistency.
##########
frontend/src/app/workspace/service/notebook-migration/migration-llm.ts:
##########
@@ -111,15 +111,66 @@ function toDerivedNotebook(cells: DerivedCell[]): Notebook {
outputs: [],
execution_count: null,
})),
- metadata: {
- kernelspec: { display_name: "Python 3", language: "python", name: "python3" },
- language_info: { name: "python" },
- },
+ metadata: structuredClone(metadata),
nbformat: 4,
nbformat_minor: 4,
};
}
+const PYTHON_NOTEBOOK_DOCUMENTATION: string[] = [
+ TEXERA_OVERVIEW,
+ TUPLE_DOCUMENTATION,
+ TABLE_DOCUMENTATION,
+ OPERATOR_DOCUMENTATION,
+ EXAMPLE_OF_GOOD_CONVERSION,
+ VISUALIZER_DOCUMENTATION,
+ UDF_INPUT_PORT_DOCUMENTATION,
+ EXAMPLE_OF_MULTIPLE_UDF_CONVERSION,
+];
+
+// The script prelude differs in exactly one entry. The notebook worked example is written in
+// `# START CELL1` form, and a system-message example of that weight would push the model to
+// answer in cell ids for an input that has no cells. The notebook array is left untouched so
+// existing conversions see byte-identical context.
+const PYTHON_SCRIPT_DOCUMENTATION: string[] = PYTHON_NOTEBOOK_DOCUMENTATION.map(doc =>
+ doc === EXAMPLE_OF_MULTIPLE_UDF_CONVERSION ? EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT : doc
+);
+
+// Differs from the script prelude in the same single entry: its worked example shows banner
+// lines, an entry point calling into other files, and definitions inlined rather than imported.
+const PYTHON_FOLDER_DOCUMENTATION: string[] = PYTHON_SCRIPT_DOCUMENTATION.map(doc =>
+ doc === EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT ? EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_FOLDER : doc
+);
+
+export type MigrationLanguage = "python";
+
+interface MigrationPromptSet {
+ notebookDocumentation: string[];
+ notebookWorkflowPrompt: string;
+ notebookMappingPrompt: string;
+ scriptDocumentation: string[];
+ scriptWorkflowPrompt: string;
+ scriptMappingPrompt: string;
+ operatorType: string;
Review Comment:
`operatorType` being typed as a plain `string` makes it easy to introduce a typo that only fails at runtime (e.g., when calling `getNewOperatorPredicate`). Consider tightening the type to an existing operator-type type/union (if available in the codebase), or deriving it from `OperatorPredicate[\"operatorType\"]` / a shared operator registry type so invalid operator types are caught at compile time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org