You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "chenlica (via GitHub)" <gi...@apache.org> on 2026/05/13 21:58:36 UTC
[D] Feature ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
GitHub user chenlica created a discussion: Feature ideas for the dkNet-AI · Apache Texera Agent Hackathon
Please use this post to share ideas you want to pursue in the dkNET-Texera hackathon (https://texera.io/hackathon/) this week at UCI. They can be about all kinds of topics such as
- Human-Agent Collaboration
- Data Experience
- Productivity Enhancement
- Innovation
You are welcome to be creative :-)
GitHub link: https://github.com/apache/texera/discussions/5059
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "odabir33 (via GitHub)" <gi...@apache.org>.
GitHub user odabir33 edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
## Idea 1A: Plan-then-Build: draft before building
**Frustration:** The agent jumps straight to creating operators. When it guesses wrong, you watch 6+ wrong operators appear, pay tokens for each, and regenerate the workflow or clean up manually.
**Fix:** Insert a text-only plan phase before any mutation.
`Request → Clarify questions → Markdown plan → Approve → Build`
The agent emits a short plan (operators, params, assumptions, open questions) and waits. User refines in plain English (*"skip step 3"*). Nothing touches the canvas until **Approve**.
**Why this matters?**
- Wrong plans cost a 10-second read, not a 5-minute cleanup
- Skips workflow mutation, recompilation, and per-step context rebuilds — one cheap call instead of many expensive ones
- Clarifying questions help novices articulate vague intents
---
## Idea 1B: Sample-then-Scale: try it on 100 rows first
**Frustration:** *"I waited 8 minutes for it to crash at op12. Now I wait 8 more."* Most errors (schema, join keys, filters) show up instantly on a sample.
**Fix:** Default the first run to **100 rows** per source. User reviews, tweaks, then clicks **Run on full data**.
`Build → Sample run (~3s) → Review → Full run`
**Why it matters?**
- Catches ~90% of bugs in seconds, not minutes
- Stacks with Idea 1: plan → build → sample → scale
---
Idea 1 catches *intent* errors before any work happens. Idea 2 catches *data* errors before any waiting happens.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925445
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
My PR: https://github.com/apache/texera/pull/5086
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936847
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "jaeyun0503 (via GitHub)" <gi...@apache.org>.
GitHub user jaeyun0503 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Floating Agent:
A small floating agent that can function as:
1. Notification Center: As a regular user, the floating agent will notify users when a workflow finishes running, or runs into an issue. When other users clone or like a user's workflow, the agent will notify the user as well. Admin users will also receive notifications when a new user joins the system and requests approval.
2. Workflow Monitor: Can be used to monitor the progress of workflows
3. Side-by-side workflow builder/suggester: in the workflow page, it can function as an agent that helps users to build workflow with human input.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16923980
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
we had video tutorials before but it was not very easy to use. it will be great if agent can adapt easily on the current system and introduce!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16934656
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
# Live Canvas Analytics for Texera
## Motivation
- Today, every Texera operator looks the same regardless of zoom — and to actually see results, users have to break flow into a separate popout panel
- The canvas should adapt to how the user is reading it: a high-level overview when zoomed out, full details and live results when zoomed in
- This is the difference between *"a workflow editor with a results tab"* and *"a true analytics workspace"*
## The headline: semantic zoom
- Zoom **out** and operators collapse to high-level boxes — the workflow reads at a glance, like a map
- Zoom **in** on an operator and it expands in place to reveal its parameters, sample data, and a live chart of its output
- One interaction scales the view from *"what is this workflow about?"* to *"what's happening inside this step?"* — no popouts, no panels, no context switches
## Supporting features
- **Free-form notes.** Drop a sticky note anywhere on the canvas to capture an observation, hypothesis, or question. Notes stay readable even when zoomed out, so they act as signposts on the overview.
- **Agent-authored insights.** The Texera agent watches results as they stream in and drops short sticky notes next to each chart (*"median age 34, right-skewed bimodal"*), so the zoomed-out view tells you what mattered without inspecting every operator.
- **Live dashboards.** Pin the most important charts into a dashboard view that updates as the workflow runs. Semantic zoom gives you depth; the dashboard gives you a curated cross-section.
## Why it fits the Agent Hackathon
- Brings the Texera agent out of the chat sidebar and onto the canvas, where users actually work
- A single visible UX shift judges can grasp in a 60-second demo
- Solo-sized — the headline mechanic plus three self-contained supporting pieces, each demoable on its own
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924434
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
TexeraClaw:
We want Texera to be able to connect to the user local machine and run commands and basically become manager by defining workflows. The followings are some examples of how this can help:
- Read local files: user can just give address of a file and workflow can read it directly
- Develop local application: if user asks for something that involves running specific process on user local, agent can create a code, run it in user laptop and connect it to the workflow
This work extends Texera from a DAG data science to a fully local agent which can turn tasks into transparent workflows.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924684
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
GitHub user eugenegujing added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
> Is it specific to data cleaning use case ? Seems it is a general permission mechanism
I'm not explaining really well here in the prompt. I'm thinking about implementing for 2 features basically: First, when dataset is imported to the workflow, agent would automatically examine the whole dataset to find out potential dataset errors, like how grammarly ai detects any grammar errors in the text, then giving users some prompts about whether to fix them, if yes the agent could fix them. Second, when using agent to modify the workflow, I want to add a permission mechanism before agent modifying anything in the workflow. Is this clear enough?
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925025
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
GitHub user mengw15 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
https://github.com/apache/texera/pull/5081
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936515
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
GitHub user kz930 edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Added an AI Workflow Wizard to Texera that guides users through four steps: (1) define the analysis goal, (2) select the data source, (3) choose a scientific framework, and (4) set guardrails and constraints. After these steps, the wizard automatically generates a Texera workflow using the existing AI services and lets users review any missing required settings before applying it. The feature was built entirely on the frontend without changing Texera’s core backend code.
See #5095
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936868
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "jaeyun0503 (via GitHub)" <gi...@apache.org>.
GitHub user jaeyun0503 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Link to PR: PR #5093
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942013
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "tanishqgandhi1908 (via GitHub)" <gi...@apache.org>.
GitHub user tanishqgandhi1908 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
**Explainable Result Pane for Texera**
Texera currently lets users preview operator outputs, but the result pane behaves mostly like a static table. Users can inspect rows and search column names, but they cannot interact with the data the way they expect from modern spreadsheet-like tools: filtering by values, sorting columns, searching across row contents, or quickly narrowing down records of interest. They also still have to manually compare outputs across operators to understand what changed.
I propose upgrading the result pane into an Interactive and Explainable Result Pane that combines two improvements:
1. Spreadsheet-like data exploration + true data search instead of only column search
2. Transformation explanation
This would help users not only see their data, but also understand their workflow. Instead of repeatedly switching between operators and manually inspecting outputs, users could explore, compare, and debug transformations from one place.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925950
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
GitHub user bobbai00 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Very good idea. Here are some related work in the recent literatures: https://github.com/lotus-data/lotus https://arxiv.org/abs/2407.11418
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16935315
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "MelihErduran (via GitHub)" <gi...@apache.org>.
GitHub user MelihErduran added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Simple start stop commans to launch dev environment.
See [#5079](https://github.com/apache/texera/pull/5079)
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936834
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "claire687 (via GitHub)" <gi...@apache.org>.
GitHub user claire687 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
https://github.com/apache/texera/pull/5087
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936849
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "galenjauss (via GitHub)" <gi...@apache.org>.
GitHub user galenjauss added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
### Idea 1 — Persistent Memory (a Texera agent that grows with you)
Today the agent forgets you between sessions. What if it didn't?
A **persistent profile** that observes your work across every workflow you build — the operators you reach for, your fix patterns, your domain expertise, how you've grown. The agent uses it to **pre-seed new workflows in your style** ("you usually filter nulls before joining, want me to scaffold that?") and references your past work when you hit familiar errors. You see everything that's remembered in a "Your Data Science Self" panel and can edit or delete any of it.
Not just chat memory — closer to **personal informatics for data scientists.**
### Idea 2 — Workflow Theater (watch your data flow)
Workflow execution today is progress bars and status icons. What if instead, you **watched actual data tuples flow through your DAG as animated particles**?
Bottlenecks visible as traffic jams. Filters showing tuples dropped to the floor. Joins showing two streams merging. Click any tuple mid-flight to inspect it. Scrub a timeline to rewind execution.
Builds on the time-travel debugging work in IcedTea — adds the visual layer that makes it legible to humans.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924332
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
GitHub user xuang7 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Idea 1: AI-Guided Tutorial for New Users
- Issue: Onboarding is hard, new users don't know where to start.
- Idea: A first-time user mode that walks the user through building their first workflow end-to-end, to reduce onboarding friction and lower the barrier for new users:
- A welcome modal asks new users if they want a guided tour.
- The tutorial highlights the next UI element to interact with (e.g., "drag a CSV Source from the operator menu") with a spotlight and tooltip.
- Each step auto-advances when the user completes the expected action.
- A built-in AI chat sidebar answers in-context questions like "what does this operator do?" using Texera's existing agent service.
Idea 2: AI Workflow Doc Generator
- Issue: After building a workflow, there's no easy way to document, and users rarely write READMEs manually.
- Idea: A one-click action that uses the AI agent to generate a document for a given workflow, describing its purpose, inputs, outputs, and key steps. Useful for sharing workflows and improving reproducibility.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924305
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "yangzhang75 (via GitHub)" <gi...@apache.org>.
GitHub user yangzhang75 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5122
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16950006
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
GitHub user eugenegujing edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
> Is it specific to data cleaning use case ? Seems it is a general permission mechanism
I'm not explaining really well here in the prompt. I'm thinking about implementing this: when dataset is imported to the workflow, agent would automatically examine the whole dataset to find out potential dataset errors, like how grammarly ai detects any grammar errors in the text, then giving users some prompts about whether to fix them, if yes the agent could fix them. Is this clear enough?
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925025
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "yangzhang75 (via GitHub)" <gi...@apache.org>.
GitHub user yangzhang75 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
BioFlow Genesis: AI guesses what you want to do, from the dataset itself.
Drop a CSV. Genesis profiles the data, column types, target candidates, value distributions, and proposes 4 domain-appropriate analyses as cards. A medical dataset gets classification cards. A numerical dataset gets regression cards. A text-heavy one gets exploration. The AI infers intent from the data, not from the user.
One click on a card builds a wired Texera workflow with sklearn training and an AI Insight node that writes a 5-section interpretation from the results.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16947070
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Improve the dataset previewer to show more metadata of each file to get a better insight on the dataset.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942056
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Here are some example ideas:
1. AI agent
- NL-to-workflow auto-suggest — extend agent-service to suggest the next operator given the current canvas state (not just whole-workflow generation). Inline ghost-node previews on the canvas.
- "Explain this workflow" — agent reads a workflow + recent execution stats and produces a plain-English summary, including bottleneck and data-shape narration. Great demo material.
- Voice-driven workflow editing — Web Speech API → agent-service → workflow mutation. "Add a filter on age > 30 after the CSV scan."
- Self-healing workflows — when a run fails, the agent reads the error + operator config and proposes a fix as a diff on the workflow JSON.
2. Workflow engine (Amber)
- Operator marketplace / share-a-UDF — one-click publish of a Python UDF (with its requirements) to a shared registry; one-click import into someone else's workflow. Hits the pytexera SDK and file-service.
- GPU-aware scheduler hint — let an operator declare requires: gpu and have computing-unit-managing-service route it to a GPU pod. Even a stub that just labels pods is demo-worthy.
3. Frontend
- Workflow diff & PR-style review — visualize the diff between two workflow versions (357K versions in the badge — clearly important) as a side-by-side DAG with added/removed/changed nodes highlighted. Comment threads on nodes.
- "Replay another user's session" — record canvas edits + executions and play them back as a tutorial. Pairs well with the real-time collab infra already in place.
- Mobile / tablet read-only viewer — render workflows + live execution status on a phone for monitoring long-running jobs.
4. Data / AI for Science
- One-click LLM eval workflow template — drag a dataset, a prompt template, and a model operator; get a leaderboard. Showcases Texera as an experiment harness.
- RAG-as-a-workflow — chunker → embedder → vector store → retriever → LLM, all as composable operators. A killer-demo template that ships with the platform.
5. Dev/infra
- In-browser Python sandbox for UDFs — use Pyodide to validate operator code (lint + smoke run) before shipping it to a worker. Faster feedback than the round-trip through Amber.
- texera dev CLI — single command that brings up the minimum stack (Amber + one service + frontend) using Docker compose, replacing the multi-step single-node setup in bin/single-node.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16911198
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "odabir33 (via GitHub)" <gi...@apache.org>.
GitHub user odabir33 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
## Idea 1: Plan-then-Build: draft before building
**Frustration:** The agent jumps straight to creating operators. When it guesses wrong, you watch 6+ wrong operators appear, pay tokens for each, and regenerate the workflow or clean up manually.
**Fix:** Insert a text-only plan phase before any mutation.
`Request → Clarify questions → Markdown plan → Approve → Build`
The agent emits a short plan (operators, params, assumptions, open questions) and waits. User refines in plain English (*"skip step 3"*). Nothing touches the canvas until **Approve**.
**Why this matters?**
- Wrong plans cost a 10-second read, not a 5-minute cleanup
- Skips workflow mutation, recompilation, and per-step context rebuilds — one cheap call instead of many expensive ones
- Clarifying questions help novices articulate vague intents
---
## Idea 2: Sample-then-Scale: try it on 100 rows first
**Frustration:** *"I waited 8 minutes for it to crash at op12. Now I wait 8 more."* Most errors (schema, join keys, filters) show up instantly on a sample.
**Fix:** Default the first run to **100 rows** per source. User reviews, tweaks, then clicks **Run on full data**.
`Build → Sample run (~3s) → Review → Full run`
**Why it matters?**
- Catches ~90% of bugs in seconds, not minutes
- Just inject `TableLimit(100)` on sources at submit; strip for full run
- Stacks with Idea 1: plan → build → sample → scale
---
Idea 1 catches *intent* errors before any work happens. Idea 2 catches *data* errors before any waiting happens.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925445
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
GitHub user mengw15 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Great point — the schema-aware context can definitely extend to other operators. For this hackathon I'm focusing on Python UDFs since the 4 interaction modes need polish together, but the context plumbing is operator-agnostic so it should generalize.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16935379
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "PG1204 (via GitHub)" <gi...@apache.org>.
GitHub user PG1204 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Workflow Performance Profiler:
1. A visual profiling layer on top of existing Texera runtime statistics that helps users quickly see where their workflows are slow and how to fix them.
2. Bottleneck Heatmap: On the workflow canvas, each operator node is colored based on relative cost, using metrics like runtime and rows processed. After or during a run, users see “traffic light” colors on the graph that highlight the hottest operators at a glance.
3. Detailed Per‑Operator Stats: Hovering or clicking an operator shows a small panel with richer metrics such as total runtime, input and output row counts, throughput, and data size, turning raw JSON stats into human‑readable information.
4. Optimization Hints: A lightweight cost model and a few rule‑based checks (for example, a join with huge input and tiny output) generate simple suggestions like “add a Filter before this Join” or “increase parallelism for this hot operator,” guiding users toward concrete performance improvements without digging into logs.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924043
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
````markdown
# AI Workflows in Texera: LLM, RAG & Agentic DAG
**Pitch.** Today Texera can run ML inference but has *no first-class LLM, RAG, or agentic-DAG operators* — the only AI surface is the workflow-editing chat in `agent-service`. We add drag-and-drop AI building blocks so users compose RAG and agent workflows in the same visual canvas they already use for ETL.
**What we'll ship**
- **LLM operators** — `LLMChat`, `LLMEmbedding`. Reuse the existing `agent-service` LLM gateway (`LLM_ENDPOINT`, OpenAI-compatible) — no new keys, no new infra.
- **RAG operators** — `DocumentChunker`, `VectorStoreSink`, `VectorStoreRetriever` on top of a pluggable Python `VectorStore` abstraction. Ship **Chroma** in embedded mode (zero-deploy MVP) as the first backend.
- **Agentic primitive** — `Agent` operator: a single visual node containing a ReAct loop with declarative tools (calculator / web-search / SQL / Python). No DAG-cycle changes needed.
- **Stretch** — a `SubWorkflow` operator that runs a saved workflow as a child of the parent execution → enables agents-as-reusable-workflows.
**Why it's a good hackathon fit**
- Most of the work follows the established `PythonOperatorDescriptor` + HuggingFace pattern → fast to land, low review risk.
- Frontend palette auto-updates from operator metadata → zero Angular work for the demo.
- Each operator is independently demoable on day one.
**Demo storyboard**
1. *Phase-1 demo:* `CSV(reviews)` → `LLMChat("summarize")` → results panel.
2. *Phase-2 demo:* ingest PDFs → chunk → embed → vector-store; ask question → retrieve → LLM answers with citations.
3. *Phase-3 demo:* one `Agent` node with `WebSearch` + `Calculator` plans a trip end-to-end; console streams the ReAct trace.
**Impact.** Turns Texera into a viable visual environment for RAG pipelines and agent prototypes — opening data-science / education use cases (UCI courses, research demos) that currently need LangChain glue code.
````
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924278
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Nicoleee1108 (via GitHub)" <gi...@apache.org>.
GitHub user Nicoleee1108 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Row-Level Data Lineage: "Why Is This Row Here?"
Categories: Data Experience · Innovation
The problem
Every data analyst, at some point, stares at a number in a result table and asks: why is this number what it is? Which input rows produced it? Was it inflated by a duplicate join key? Skewed by one outlier? In SQL or Pandas, this question is essentially unanswerable after the fact — once you've aggregated, the input rows are gone, and you have to re-run with manual instrumentation to find out.
The idea
Right-click any output row in Texera's result panel → click "Why?" → the canvas dims, then visually traces backwards through every operator, highlighting the upstream tuples that contributed to that row. Click any intermediate operator and the result panel shows that operator's contributing rows. The workflow explains itself.
A region = West, total = $1.2M row becomes a story you can audit:
- 1,847 rows survived the Filter
- Came from 1,847 Join outputs
- Sourced from 864 customer rows and 1,847 order rows in the CSV scans
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924906
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
GitHub user bobbai00 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Is it specific to data cleaning use case ? Seems it is a general permission mechanism
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924976
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Mobile App for Texera.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924102
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Implemented a prototype **Compiled C++ UDF Operator** for Texera.
The operator lets users write C++ directly inside the workflow UI, configure input/output columns, compile the code, and run it as a workflow operator. The MVP supports typed tuple/batch/table-style APIs, compiler errors, retained input columns, configurable compiler flags, timeout, and batching.
To demonstrate why this is useful, I benchmarked the same deterministic CPU-heavy matrix multiplication workload across C++, Java UDF, and Python UDF. Each runtime receives its own independent source with the same seed/workload.
### Benchmark Results
| Runtime | Avg ms / row | Min ms | Max ms | Runs | Speedup vs Python |
|---|---:|---:|---:|---:|---:|
| C++ | 0.122 | 0.103 | 0.477 | 80 | 132.7x |
| Java | 1.073 | 0.388 | 5.673 | 80 | 15.1x |
| Python | 16.220 | 9.235 | 34.053 | 80 | 1.0x |
This shows the motivation clearly: Python is great for usability, but compiled C++ can provide major speedups for CPU-heavy UDFs while staying inside Texera’s visual workflow model.
### C++ UDF
```cpp
#include <chrono>
class MatrixMultiplyOperator : public texera::UDFOperator {
public:
texera::TupleOutput process_tuple(const texera::Tuple& tuple, int port) override {
int trial = tuple.get("trial").as_int();
int n = tuple.get("matrix_size").as_int();
long long seed = tuple.get("seed").as_long();
auto start = std::chrono::high_resolution_clock::now();
double checksum = 0.0;
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
double cell = 0.0;
for (int k = 0; k < n; k++) {
double a = ((seed + trial * 97LL + i * 31LL + k * 17LL) % 1000LL) / 1000.0;
double b = ((seed + trial * 53LL + k * 13LL + j * 29LL) % 1000LL) / 1000.0;
cell += a * b;
}
checksum += cell;
}
}
auto end = std::chrono::high_resolution_clock::now();
double elapsed_ms = std::chrono::duration<double, std::milli>(end - start).count();
return { texera::TupleLike{
texera::Value::string_value("cpp"),
texera::Value::double_value(checksum),
texera::Value::double_value(elapsed_ms)
}};
}
};
using TexeraUDFOperator = MatrixMultiplyOperator;
```
### Java UDF
```java
public TupleLike processTuple(Tuple tuple) {
int trial = ((Number) tuple.getField("trial")).intValue();
int n = ((Number) tuple.getField("matrix_size")).intValue();
long seed = ((Number) tuple.getField("seed")).longValue();
long start = System.nanoTime();
double checksum = 0.0;
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
double cell = 0.0;
for (int k = 0; k < n; k++) {
double a = ((seed + trial * 97L + i * 31L + k * 17L) % 1000L) / 1000.0;
double b = ((seed + trial * 53L + k * 13L + j * 29L) % 1000L) / 1000.0;
cell += a * b;
}
checksum += cell;
}
}
double elapsedMs = (System.nanoTime() - start) / 1_000_000.0;
Object[] inputFields = tuple.getFields();
Object[] outputFields = Arrays.copyOf(inputFields, inputFields.length + 3);
outputFields[inputFields.length] = "java";
outputFields[inputFields.length + 1] = checksum;
outputFields[inputFields.length + 2] = elapsedMs;
return TupleLike$.MODULE$.apply(Arrays.asList(outputFields));
}
```
### Python UDF
```python
from pytexera import *
import time
class ProcessTupleOperator(UDFOperatorV2):
@overrides
def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
trial = int(tuple_["trial"])
n = int(tuple_["matrix_size"])
seed = int(tuple_["seed"])
start = time.perf_counter()
checksum = 0.0
for i in range(n):
for j in range(n):
cell = 0.0
for k in range(n):
a = ((seed + trial * 97 + i * 31 + k * 17) % 1000) / 1000.0
b = ((seed + trial * 53 + k * 13 + j * 29) % 1000) / 1000.0
cell += a * b
checksum += cell
elapsed_ms = (time.perf_counter() - start) * 1000.0
output = tuple_.as_dict()
output["runtime"] = "python"
output["checksum"] = checksum
output["elapsed_ms"] = elapsed_ms
yield output
```
https://github.com/user-attachments/assets/10fc6eee-8742-403e-9b11-11221edb1b9b
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16926806
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
PR #5083
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936539
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "galenjauss (via GitHub)" <gi...@apache.org>.
GitHub user galenjauss added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Idea 2 shipped — submitted as **Workflow Theater** for the **Best Data Experience** track.
PR: https://github.com/galenjauss/texera/pull/1
The visualization is built on the WebSocket events Texera already emits (`OperatorStatisticsUpdateEvent`, `OperatorCurrentTuplesUpdateEvent`, `ConsoleUpdateEvent`) — no engine changes. The time-travel scrubber is the **IcedTea** nod; the Python UDF side panel with the current-line highlight is the **UDON** nod.
2-minute demo video link to follow.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942066
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "SarahAsad23 (via GitHub)" <gi...@apache.org>.
GitHub user SarahAsad23 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5102
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942107
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Idea: Python UDF to a sub-DAG
A user creates a Python UDF operator and adds a Python script. Add a button to the operator to automatically translate into a sub-DAG in the existing workflow.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16935920
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5112
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942131
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
@galenjauss did you submit your video? we could not find it.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16954146
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
GitHub user eugenegujing added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
# DataGuard: Claude-Code-style Permissioned AI for Data Cleaning
**Themes:** Human-Agent Collaboration · Data / AI for Science
**The gap.** Texera can run cleaning logic, but data-cleaning *decisions* today are made one of two ways: silently by an auto-clean script (no audit trail, no human control), or manually in a notebook (no reproducibility, doesn't scale). Neither works for scientific data, where placeholder values, group-imbalanced missingness, and "outliers that are actually meaningful rare cases" need a domain expert in the loop.
**The idea.** Bring the Claude Code permission model into Texera, but applied to *data* instead of code. A conversational `DataGuard` agent profiles a dataset, proposes one fix at a time, and asks the user to **Allow / Allow & remember / Deny / Modify** before any mutating action. Every approved action is logged with evidence and confidence, producing an auditable, replayable decision trail.
**Why it fits the Agent Hackathon**
- **No engine changes.** Lives entirely in `agent-service` + chat panel; doesn't require Amber pause/resume, no new operators, no new protocol — the hardest path for HITL workflows is sidestepped by going chat-first.
- **Targets domain-expert users** (bio, medicine, social science) who consume data but aren't developers — addressed at the data-decision layer rather than the code layer.
- **Reproducible by design.** `--replay decision_log.csv` rebuilds the cleaned dataset without re-invoking the LLM. Concrete, measurable correctness.
- **Aligned with the Data / AI for Science theme** — every cleaning decision is explained, justified, and reversible.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924731
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
YES! I would love to see more themes that are concise across entire site!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16933015
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
I love both ideas! for idea 1, maybe we can combine memory and style, you can give more personality to the agent as well.
for idea 2, I recall @aglinxinyuan has an attempt, maybe you can make it even better! if we can visualize time traveled workflows that's be nice!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16934617
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "ImJordanNetz (via GitHub)" <gi...@apache.org>.
GitHub user ImJordanNetz added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
AI Agent as an operator. You send each row + system prompt into an LLM, which returns text and puts it into a new column,
This is useful for classification, NLP, and much more!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16935288
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
GitHub user mengw15 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
**UDF Copilot: a schema-aware AI inside the Python UDF editor**
Themes: Human-Agent Collaboration · Productivity Enhancement
**The gap.** Today, `agent-service` is great at building/editing whole workflows, and the editor has a single "Add Type Annotation" button. But once you start *writing* a UDF, you're on your own — no autocomplete that knows the upstream schema, no "fix this", no inline edits, no chat.
**The idea.** Bring a Cursor-style assistant into the Monaco UDF editor, with one thing GitHub Copilot can't have: **Texera context** — the upstream operator's output schema, a real sample row, the downstream consumer, and the UDF's API contract (`ProcessTupleOperator`, tuple vs table API).
**Four interaction modes** sharing that context:
1. **Schema-aware ghost text** — typing `tuple_["` suggests real column names from the upstream operator, not generic guesses.
2. **Cmd+K inline rewrite** — "add an `is_adult` column for age > 18" → AI rewrites the lines, shows a diff, Accept/Reject.
3. **Quick Fix on Pyright errors** — `KeyError: 'age'` → one-click fix that knows the schema actually has `user_age`.
4. **Side chat panel** — "why is this slow?", "convert tuple API to table API", "generate pytest with null edge cases", each with an "Apply to editor" button.
**Why it matters.** A large share of Texera's audience are domain experts (biology, social science, medicine) who write UDFs but aren't full-time programmers. A schema-aware copilot collapses the learning curve from "read pytexera docs first" to "just describe what you want, with real column names autocompleting."
**Implementation sketch.** Frontend: extend `code-editor.component.ts`, add Monaco `InlineCompletionsProvider`, register a Cmd+K widget, add a side panel. Backend: 4 new endpoints in `agent-service` reusing the Vercel AI SDK already wired up. Context plumbing: pull upstream schema from `WorkflowActionService` already in memory on the frontend.
**Demo.** Open a Python UDF after a CSVScan → ghost text suggests real columns → Cmd+K rewrites a loop as vectorized pandas → force a schema typo → one-click Fix → side chat generates pytest. ~3 minutes end-to-end.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924109
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
GitHub user eugenegujing added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5101
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16940389
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "jaeyun0503 (via GitHub)" <gi...@apache.org>.
GitHub user jaeyun0503 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Link to PR: #5093
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942017
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "nathant27 (via GitHub)" <gi...@apache.org>.
GitHub user nathant27 edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
My idea is to implement Operators to allow for reading data from a news api in real time through web sockets, or polled at intervals. Depending on the daily sentiment, it will make a decision, and send a POST request to an external API that outputs its decisions. I'll have it make buy/sell decisions of stocks, or will have it make bets based on Kalshi.
Key operators: something that polls from an API endpoint at fixed intervals. another operator that makes POST requests to an endpoint. Might make it poll at fixed intervals? Might just have an operator that makes requests, and instead of polling an endpoint, and just make a timer operator that repeats it while the workflow is active? Not too sure on this but will clarify as I develop more.
#5103
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16927257
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "claire687 (via GitHub)" <gi...@apache.org>.
GitHub user claire687 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Expand the current AI agent into a global Texera AI agent that lets users perform any action, in any tab, using natural language.
Here are some things the updated AI agent can do:
- Directly allow user to upload files and move around different pages to complete the entire workflow
- This will make our current single AI agent into a multi agent system
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924699
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
idea 2: I am very interested in this document. if we can make it embed with the workflow and result, it self can be a data science report!!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924784
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "MelihErduran (via GitHub)" <gi...@apache.org>.
GitHub user MelihErduran added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Very simple but added themes beyond light and dark mode. You can edit font, font size, theme color etc. Applies to all areas of the screen.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925713
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "nathant27 (via GitHub)" <gi...@apache.org>.
GitHub user nathant27 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
My idea is to implement Operators to allow for reading data from a news api in real time through web sockets, or polled at intervals. Depending on the daily sentiment, it will make a decision, and send a POST request to an external API that outputs its decisions. I'll have it make buy/sell decisions of stocks, or will have it make bets based on Kalshi.
Key operators: something that polls from an API endpoint at fixed intervals. another operator that makes POST requests to an endpoint. Might make it poll at fixed intervals? Might just have an operator that makes requests, and instead of polling an endpoint, and just make a timer operator that repeats it while the workflow is active? Not too sure on this but will clarify as I develop more.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16927257
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "MelihErduran (via GitHub)" <gi...@apache.org>.
GitHub user MelihErduran added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
@Yicong-Huang see [#5084](https://github.com/apache/texera/pull/5084)
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936673
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
we do have some of those operators, checkout source ops. One old example is twitter source reading from tweets. But would be nice to have a general streaming operator that is always running, to accept events, it can serve the purpose for ingestion.
I would recommend don't limit to adding an operator. make your project bigger scope that includes those operators, make a use case out of them, so that they are justified!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16932929
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
GitHub user mengw15 edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
PR: https://github.com/apache/texera/pull/5081
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936515
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "yangzhang75 (via GitHub)" <gi...@apache.org>.
GitHub user yangzhang75 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5122
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16950004
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Yes we could try rust as well, thanks
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16935819
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "ImJordanNetz (via GitHub)" <gi...@apache.org>.
GitHub user ImJordanNetz added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
https://github.com/apache/texera/pull/5111
Submitted!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942093
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
it would be great to have it! also consider support folders? (we have a lot of use cases of reading a folder of images, etc).
I actually had another direction in mind before: have a LLM operator that reads a sample (or a part) of a file, and create an operator on the fly to read/parse the entire file into table format. then run that operator as a source. so it is not a pre-designed generic operator that supports all kinds of files, but a dynamically generated operator designed specifically for that single file (or a folder of similar files). One use case: I have a business report in pdf which embeds some tables, or other information inside of it, read it out would be very useful. But all pdfs are having different structures which we may not be able to know before seeing the file. same source of the pdfs may share similar sturcutre (e.g., business report generated by the same company across different months).
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16933200
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Theospe (via GitHub)" <gi...@apache.org>.
GitHub user Theospe added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
A sports-betting pipeline for Valorant kill prop bets on Underdog Fantasy, where Texera does the data work and a machine learning model makes the actual predictions. Texera cleans up raw match data, scrapes daily betting lines, and feeds everything into the model in a repeatable, visual way. The followings are some examples of how this can help:
Clean data, ready for the model: Texera handles the messy work — combining stats from multiple sources, weighting recent games more heavily, and packaging it all into the format the model expects.
Daily picks with explanations: Texera pulls today's betting lines, runs them through the model, and then an LLM writes a short reason for each suggested bet. The math comes from the model; the LLM only explains it in plain language.
Honest performance checks: a backtest workflow replays past games to see if the model actually has an edge, with a strict gate — no real money is placed until the model proves itself across at least 1,000 historical picks.
Self-correction over time: as real bets get resolved, Texera updates how confident the model should be in its own predictions, and flags when results start drifting in the wrong direction.
Human-in-the-loop reviews: a diagnostics workflow lets an LLM read the model's recent performance and suggest where it's doing well or poorly — like a coach reviewing game tape.
The idea is to use Texera as the backbone that turns a tangle of scripts into a clean, transparent pipeline, with the model doing the predicting and AI assistants writing the reasoning — so a human can trust, verify, and improve the system over time.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925830
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
it would be a great project if you can polish the result table!!! one concern to keep in mind is about size: size of data could potentially be much larger than frontend's memory. and tables can be very wide.
we could also do some frontend views of the same backend result: temporary reorder result, filter rows, hide columns, etc.
also can combine visualization and tables.
don't let me restrict your imagination though!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16932975
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Texera also supports ML operators, especially useful for inference. I think this can be an App idea as I mentioned before. you can even design a new interface for Apps, and user can optionally click into details to see the underlying workflow. I hope to see this came out true!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16932993
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
GitHub user xuang7 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
PR #5104
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16941514
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "jaeyun0503 (via GitHub)" <gi...@apache.org>.
GitHub user jaeyun0503 deleted a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Link to PR: PR #5093
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942013
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Theospe (via GitHub)" <gi...@apache.org>.
GitHub user Theospe added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
PR https://github.com/apache/texera/pull/5110
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16941906
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
GitHub user bobbai00 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Does this context selection idea have to be limited in the Python code editor? Can it be generalized to apply for the whole workflow workspace ?
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924913
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
# AI Workflows in Texera: LLM, RAG & Agentic DAG
**Pitch.** Today Texera can run ML inference but has *no first-class LLM, RAG, or agentic-DAG operators* — the only AI surface is the workflow-editing chat in `agent-service`. We add drag-and-drop AI building blocks so users compose RAG and agent workflows in the same visual canvas they already use for ETL.
**What we'll ship**
- **LLM operators** — `LLMChat`, `LLMEmbedding`. Reuse the existing `agent-service` LLM gateway (`LLM_ENDPOINT`, OpenAI-compatible) — no new keys, no new infra.
- **RAG operators** — `DocumentChunker`, `VectorStoreSink`, `VectorStoreRetriever` on top of a pluggable Python `VectorStore` abstraction. Ship **Chroma** in embedded mode (zero-deploy MVP) as the first backend.
- **Agentic primitive** — `Agent` operator: a single visual node containing a ReAct loop with declarative tools (calculator / web-search / SQL / Python). No DAG-cycle changes needed.
- **Stretch** — a `SubWorkflow` operator that runs a saved workflow as a child of the parent execution → enables agents-as-reusable-workflows.
**Why it's a good hackathon fit**
- Most of the work follows the established `PythonOperatorDescriptor` + HuggingFace pattern → fast to land, low review risk.
- Frontend palette auto-updates from operator metadata → zero Angular work for the demo.
- Each operator is independently demoable on day one.
**Demo storyboard**
1. *Phase-1 demo:* `CSV(reviews)` → `LLMChat("summarize")` → results panel.
2. *Phase-2 demo:* ingest PDFs → chunk → embed → vector-store; ask question → retrieve → LLM answers with citations.
3. *Phase-3 demo:* one `Agent` node with `WebSearch` + `Calculator` plans a trip end-to-end; console streams the ReAct trace.
**Impact.** Turns Texera into a viable visual environment for RAG pipelines and agent prototypes — opening data-science / education use cases (UCI courses, research demos) that currently need LangChain glue code.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924278
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Nicoleee1108 (via GitHub)" <gi...@apache.org>.
GitHub user Nicoleee1108 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5082
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936602
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
GitHub user zyratlo added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
## Workflow Auto-Documententation Agent
A new tool that automatically generates documentation for non-trivial workflows using AI.
### Problem
As workflows grow in size and complexity, their intent becomes increasingly opaque to collaborators and even to their original authors. A pipeline of twenty or more operators offers no concise, human-readable explanation of what it does, what data it consumes, or what it produces. Comment boxes provide a partial remedy but rely on author discipline and are rarely kept current. The platform currently offers no mechanism for summarizing a workflow's purpose and structure at a glance.
### Proposed Solution
Introduce an on-demand documentation action available from the workflow editor. When invoked, the system inspects the current workflow's logical structure — its operators, their configurations, and the connections between them — and produces a concise, structured narrative describing the workflow.
The generated documentation includes:
- Purpose — a short summary of the workflow's overall intent, inferred from operator composition.
- Inputs — the data sources consumed by the workflow.
- Pipeline stages — a grouping of operators into logical phases (e.g., ingestion, transformation, modeling).
- Outputs — the destinations or sinks produced by the workflow.
- Caveats — notable observations a reader should be aware of, such as unconfigured parameters, embedded credentials, or operators whose behavior depends on user-supplied code.
The result is rendered alongside the canvas as a readable artifact that can be reviewed, copied, or shared.
### Design Goals
- Non-intrusive. Documentation is generated on demand and never alters the workflow itself.
- Self-contained. The action requires no additional configuration, persistence, or external state.
- Composable. The generated output is plain markdown, allowing it to be embedded into existing collaboration surfaces such as comment boxes or external documentation.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924312
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
# Streaming Texera: Real-Time Workflows on Amber
## Problem
Texera's Amber engine is micro-batch and bounded-input today: every workflow assumes its sources finish. Live data — sockets, WebSockets, sensor feeds, chat streams, agent event logs — cannot be expressed as a Texera workflow without polling hacks.
## Idea
Add a first-class **streaming mode** to Texera workflows. Sources can produce data forever; downstream operators react to it in real time; the workflow runs indefinitely until the user stops it.
## What users get
- **Live source operators** — connect a workflow to a TCP socket or WebSocket feed and watch results update as events arrive.
- **Event-time windows** — tumbling, sliding, and session windows for aggregating streams (counts, sums, sessionization).
- **Watermarks** — principled handling of out-of-order events, so windows fire correctly under network jitter.
- **Continuous workflows** — a new "running indefinitely" workflow state with explicit Stop & Flush vs. Kill controls.
- **Python streaming UDFs** — users yield tuples (and watermarks) forever from Python, same SDK ergonomics as today's UDFs.
## Why it matters
- Unlocks live dashboards, alerting pipelines, and real-time AI-agent observability — directly aligned with the Agent Hackathon theme.
- Keeps Texera's visual-workflow model intact; no new UI paradigm to learn.
- Bounded workflows are unchanged — streaming is opt-in.
## Demo
A WebSocket feed of agent tool-call events → tumbling 1-minute window → live "tool usage by agent" chart in the Texera UI, updating as events arrive.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924319
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "odabir33 (via GitHub)" <gi...@apache.org>.
GitHub user odabir33 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
## Idea: Operator Stash — Git Stash for Properties
**Frustration:** Every property edit is destructive. The agent overwrites your config and you can't get it back. You want to just test two settings but there's no place to hold one aside. Undo only goes back in a straight line.
**Fix:** Auto-save the previous version of an operator's properties on every change. A tiny stash icon on the node lets you **toggle** or **keep/drop**. Scoped per-operator, not global.
**Why it matters?**
- Accept agent edits fearlessly
- Stash affects one node, not the whole workflow
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925506
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica edited a discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Please use this post to share ideas you want to pursue in the dkNET-Texera hackathon (https://texera.io/hackathon/) this week at UCI. They can be about all kinds of topics such as
- Human-Agent Collaboration
- Data Experience
- Productivity Enhancement
- Innovation
You are welcome to be creative :-)
GitHub link: https://github.com/apache/texera/discussions/5059
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "yangzhang75 (via GitHub)" <gi...@apache.org>.
GitHub user yangzhang75 deleted a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5122
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16950004
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
I want to encourage you to utilize our workflow canvas: it is more intuitive than "text" interface!
for example for planning, instead of showing in "Request → Clarify questions → Markdown plan → Approve → Build" text, just render temporary operators (without detailed properties, just a skeleton operator) on the canvas, would be much more visual and intuitive!
for sample, that's a great idea! many other systems have this, always want to see it for texera!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16933072
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
GitHub user kz930 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Added an AI Workflow Wizard to Texera that guides users through four steps: (1) define the analysis goal, (2) select the data source, (3) choose a scientific framework, and (4) set guardrails and constraints. After these steps, the wizard automatically generates a Texera workflow using the existing AI services and lets users review any missing required settings before applying it. The feature was built entirely on the frontend without changing Texera’s core backend code.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936868
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "tanishqgandhi1908 (via GitHub)" <gi...@apache.org>.
GitHub user tanishqgandhi1908 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
#5099
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16941987
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
@carloea2 nice results!
however let's keep this post brief for idea discussions, if you went ahead implemented it out, feel free to include detailed plans/results on a separate PR!
back to the topic, one quick question: can we try rust as well? it is also very trendy nowadays
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16932884
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
I think we have versioning already! checkout version history of workflow, it also covers individual operator version as well.
But love the idea for accept edits, just like what we have in IDEs for accept code change!
can have auto mode/ auto accept mode for ppl who don't want to accept manually as well
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16933041
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
related https://github.com/apache/texera/discussions/5059#discussioncomment-16927257
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16934496
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
@bobbai00 Can you chime in?
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-17643196
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Compiled C++ UDF Operator
Category: Productivity Enhancement / Innovation
I want to build a Compiled C++ UDF operator for Texera. The idea is to let users write small performance-critical UDFs in C++ directly inside a workflow, compile them from the UI, and run them on incoming data.
This would be useful when Python UDFs are easy to write but too slow for CPU-heavy tasks like numeric transformations, feature extraction, custom parsing, or scientific preprocessing.
MVP:
- Select input columns
- Write a simple C++ function
- Compile from the UI
- Show compiler errors in the editor
- Run the compiled function as a workflow operator
- Produce one output column
Example:
double process(double age, double income) {
return income / (age + 1.0);
}
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924396
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "SarahAsad23 (via GitHub)" <gi...@apache.org>.
GitHub user SarahAsad23 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Side-by-Side Workflow Version Comparison
Adding support for side-by-side workflow version execution and comparison within Texera. The idea is to allow users to run two versions of the same workflow in parallel and automatically detect where their behavior or outputs begin to diverge. By comparing intermediate results at each operator, the system could identify the exact stage where differences are introduced. This feature would help users debug workflow changes, validate new optimizations, and safely iterate on workflows by providing clear visibility into the impact of modifications between versions.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924490
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "tanishqgandhi1908 (via GitHub)" <gi...@apache.org>.
GitHub user tanishqgandhi1908 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Idea: Smart File Source for Texera
Texera currently exposes file input through separate operators such as CSV, JSONL, Arrow, and generic file scan. For new users, this creates friction: before building a workflow, they must already know which scan operator to choose and how to configure it. Real datasets are also messy: CSV vs TSV, missing headers, wrong delimiters, JSONL, Excel sheets, Parquet exports, and files with unclear schemas.
I want to build a Smart File Source operator that lets the user select a file once and lets Texera infer how to read it.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925420
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "richardchen874-sys (via GitHub)" <gi...@apache.org>.
GitHub user richardchen874-sys added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
The NL-to-workflow and "explain this workflow" ideas in this thread are especially interesting because they turn the agent service into a repeated workflow assistant, not just a one-off demo.
I would meter agent calls by workflow, operator, model, and user action. That makes it easier to see which ideas are practical at scale and which need cheaper routing or caching before they become production features.
I am testing an OpenAI-compatible multi-model API layer, and agentic data workflows are a strong use case for cost-aware routing. Would Texera's agent calls run centrally, or would users bring their own model/API configuration?
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-17618091
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "tanishqgandhi1908 (via GitHub)" <gi...@apache.org>.
GitHub user tanishqgandhi1908 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Divided this idea into 3 features with smart operator, visual workflow and llm parser
1. #5094
2. #5107
3. #5105
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16941978
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
PR: https://github.com/apache/texera/pull/5115
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16942700
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "PG1204 (via GitHub)" <gi...@apache.org>.
GitHub user PG1204 edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Workflow Performance Profiler:
1. A visual profiling layer on top of existing Texera runtime statistics that helps users quickly see where their workflows are slow and how to fix them.
2. Bottleneck Heatmap: On the workflow canvas, each operator node is colored based on relative cost, using metrics like runtime and rows processed. After or during a run, users see “traffic light” colors on the graph that highlight the hottest operators at a glance.
3. Detailed Per‑Operator Stats: Hovering or clicking an operator shows a small panel with richer metrics such as total runtime, input and output row counts, throughput, and data size, turning raw JSON stats into human‑readable information.
4. Optimization Hints: A lightweight cost model and a few rule‑based checks (for example, a join with huge input and tiny output) generate simple suggestions like “add a Filter before this Join” or “increase parallelism for this hot operator,” guiding users toward concrete performance improvements without digging into logs.
PR #5098
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924043
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Texera also support ML operators, especially useful for inference. I think this can be an App idea as I mentioned before. you can even design a new interface for Apps, and user can optionally click into details to see the underlying workflow. I hope to see this came out true!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16932993
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
glad to see your demos yesterday! I think it is already awesome! keep the good work.
I want to reiterate my idea of two-directional binding. now we use data workflows to represent a task, your documentation/summary could be a text version describing the task, it can be coming form the same source data, and modify any (workflow or text) would also update the other one.
don't be limited by my thoughts!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16934650
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Theospe (via GitHub)" <gi...@apache.org>.
GitHub user Theospe added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
I have it working in the terminal right now. Will look into making an interface today. Thank you for the advice.
If anyone is curious about the results: Gave the model $30 currently at $632.37 after 1 week.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16934644
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
https://github.com/apache/texera/pull/5078
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936518
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
GitHub user bobbai00 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
sounds good!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936557
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
GitHub user zyratlo added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
PR #5088
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16936923
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
Taken by me. Starting to work on this idea.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924479
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
# AI-Augmented Macro Operators for Texera
## Problem
Texera workflows grow into 20–50+ operator DAGs with no encapsulation. Users copy-paste the same subgraphs across projects, and pipelines run slower than they need to because of inter-operator serialization.
## What we'll build
- **Macro operators** — collapse a selection of operators into one reusable, version-pinned node (KNIME wrapped-metanode style). Drill-down to edit; drag from a library to reuse.
- **Agent tool: `suggestMacros`** — the agent inspects the `LogicalPlan` and proposes ranked subgraphs to encapsulate, each with a one-line rationale ("looks like a reusable text-preprocessing block"). Highlights candidates on the canvas; one click materializes.
- **Agent tool: `fuseMacro`** — for a macro whose internals the user no longer needs to inspect, the agent synthesizes an equivalent `PythonUDFOpDescV2`, runs original and fused on a sample, diffs outputs, and only swaps in after verification passes.
## Why it fits the Agent Hackathon
- Plugs straight into the existing `agent-service` (Vercel AI SDK + ReAct loop + tool framework). Two new tools, no new LLM plumbing.
- The agent doesn't just suggest — it **verifies** (sample-run diff for fusion) and rolls back on mismatch. Concrete, measurable correctness.
- Showcases capabilities a generic chatbot can't: structural reasoning over a DAG, codegen for a known runtime, and a built-in verification harness.
## Demo (~3 min)
1. Open a 15-operator workflow → "Suggest Macros (AI)" → three highlighted candidates appear with rationales.
2. Accept one → subgraph collapses into a single macro node.
3. Run → note baseline time.
4. Right-click macro → "Fuse for performance" → agent generates UDF, verifies ("matched on 1000 sample rows"), swaps in.
5. Re-run → show **2–5× speedup** on the stateless chain.
## Stretch
- Cross-workflow pattern mining: "you've built this subgraph 4 times — save as a macro?"
- Auto-publish recurring patterns to the workflow hub as community macros.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924132
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
# Texera Workflow Completion Forecasting
**What exists today.** Each operator box already displays cumulative *processed input* and *produced output* tuple counts (rendered by `JointUIService` via `.texera-operator-processed-count` and `.texera-operator-output-count`). The toolbar shows an elapsed-time counter, and the workflow state is conveyed via color badges. Users can already see *what has happened* and *for how long*.
**The actual gap.** What they cannot see:
- **Rate** — is the operator moving 230k rows/sec, slowing down, or stalled?
- **How much is left** — counts have no denominator, so "halfway done" is not expressible.
- **When will it finish** — no ETA, per operator or workflow-wide.
- **One-glance workflow health** — elapsed time keeps climbing whether work is happening or not.
**Proposal.** Turn the existing tuple counters into a *forecasting layer*. Build on the per-operator `ExecutionStatsUpdate` already streamed every ~500 ms; derive rate, predict remaining work, and surface ETA. No new transport, no engine rework.
**Scope (four small, independently shippable phases):**
- **Throughput and stall detection** — compute rows/sec from successive count snapshots (exponentially smoothed). Render a rate label next to each operator's existing counters (e.g. `230 k/s`); surface a stall warning when rate → 0 while state is still Running. Side panel adds a rate sparkline so users can see trends, not just instantaneous values.
- **Source cardinality estimation** — cheap pre-execution estimates of total input size for bounded sources: exact for Arrow/Parquet (header `numRows`), `FileLister` (file count), `URLFetcher`; file-size-based estimates for CSV / JSONL / text scans. Unknown for Python/R UDF sources and streaming APIs — those keep showing counts + rate, no fake estimate. No SQL `COUNT(*)` probes (Texera is not a database engine).
- **Per-operator ETA** — divide remaining work by smoothed rate; show "~14 s remaining" next to the operator. Each downstream operator's remaining work is bounded by its slowest input — no selectivity heuristics (which produce confidently-wrong forecasts).
- **Workflow-level ETA** — toolbar gains a "~ETA" label beside the existing elapsed counter, taken as the max over operator ETAs (longest-remaining operator gates the workflow). Hidden when no operator has an estimate — the elapsed counter remains the only indicator, same as today.
**Honesty principle.** Operators rooted in unbounded sources keep showing counts + rate only — no fabricated forecast. Where the system genuinely cannot predict, it stays silent rather than misleading.
**Deliverables.**
- One protobuf field (`estimated_output_count`) + a `CardinalityEstimable` trait implemented by ~8 source operators.
- Frontend additions in `WorkflowStatusService`, `JointUIService`, the operator property panel, and the workflow menu toolbar.
- Unit tests for each estimator and the derivation service; demo on a CSV → Filter → Aggregate → Sink workflow with before/after screenshots.
**Impact.** The existing counters answer *what happened*. This adds the answers users actually ask out loud: *how fast is it going*, *how much is left*, *when will it finish* — while being explicit when prediction is genuinely impossible.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924286
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
1. that's nice! make the notifications fancy, if it can show different things and effects on different events, that'd be nice!
2. monitor is quite hard to build, but can quickly hack a prototype! We need monitors.
3. we already have texera agent that's on the side, you can expand the power of it!
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924749
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
GitHub user bobbai00 added a comment to the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
I like this one. You can even take a step further to consider wrap this profiler as a tool that can be used by the `texera-agent`, so that agent can leverage this information to automatically optimize the workflow execution
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16924893
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Task ideas for the dkNet-AI · Apache Texera Agent Hackathon [texera]
Posted by "odabir33 (via GitHub)" <gi...@apache.org>.
GitHub user odabir33 edited a comment on the discussion: Task ideas for the dkNet-AI · Apache Texera Agent Hackathon
## Idea 1: Plan-then-Build: draft before building
**Frustration:** The agent jumps straight to creating operators. When it guesses wrong, you watch 6+ wrong operators appear, pay tokens for each, and regenerate the workflow or clean up manually.
**Fix:** Insert a text-only plan phase before any mutation.
`Request → Clarify questions → Markdown plan → Approve → Build`
The agent emits a short plan (operators, params, assumptions, open questions) and waits. User refines in plain English (*"skip step 3"*). Nothing touches the canvas until **Approve**.
**Why this matters?**
- Wrong plans cost a 10-second read, not a 5-minute cleanup
- Skips workflow mutation, recompilation, and per-step context rebuilds — one cheap call instead of many expensive ones
- Clarifying questions help novices articulate vague intents
---
## Idea 2: Sample-then-Scale: try it on 100 rows first
**Frustration:** *"I waited 8 minutes for it to crash at op12. Now I wait 8 more."* Most errors (schema, join keys, filters) show up instantly on a sample.
**Fix:** Default the first run to **100 rows** per source. User reviews, tweaks, then clicks **Run on full data**.
`Build → Sample run (~3s) → Review → Full run`
**Why it matters?**
- Catches ~90% of bugs in seconds, not minutes
- Stacks with Idea 1: plan → build → sample → scale
---
Idea 1 catches *intent* errors before any work happens. Idea 2 catches *data* errors before any waiting happens.
GitHub link: https://github.com/apache/texera/discussions/5059#discussioncomment-16925445
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org