[D] Running an operator inside a container [texera]

["chenlica (via GitHub)" <gi...@apache.org>]

GitHub user chenlica created a discussion: Running an operator inside a container

In one of the recent use cases of the system, a collaborator (@hhvu0102) expressed the need of running an operator (of workflow) inside a container, using a docker image possibly hosted on Docker Hub.  I would like start a discussion about this feature.

@hhvu0102: If possible, please provide more details of the use case.  Thanks.

GitHub link: https://github.com/apache/texera/discussions/7475

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org

Re: [D] Running an operator inside a container [texera]

["sshiv012 (via GitHub)" <gi...@apache.org>]

GitHub user sshiv012 added a comment to the discussion: Running an operator inside a container

Adding details on the use case and what we built for it.

**The use case: isolation** Ha's pipeline is three Python steps that each pin their own dependencies. Running them in one computing unit means one Python environment, and a single environment can hold only one version of a shared library. The steps may pin different numpy versions (1.26.4 / 2.1.0 / 2.0.2), one of them needs `pysam`, another `scikit-image`, and in practice any two bio tools disagree on something. The failure mode is not performance; it is "this workflow cannot be installed." Nextflow solves this by giving each process its own container image, which is why the pipeline existed as a Nextflow workflow first. The ask was: let a Texera operator do the same.

**How it looks from the UI.** On top of Chris's (@kunwp1) rented-compute prototype we made the image a per-operator property. In the operator's property panel there is an Offload toggle; turning it on reveals *Instance type* (a catalog entry such as `local-2g`, which sets the container's memory cap) and *Image* (any Docker tag). Leave *Image* blank and the operator runs on the platform's default worker image; set it to `myorg/qc-step:1.0` and that image is used for that operator only. When the workflow runs, the master issues `docker run <image>` per offloaded operator, which pulls the tag from Docker Hub (or any registry the daemon is logged into) if it is not already local, exactly like Nextflow's `container = 'myorg/qc-step:1.0'` directive. The one constraint today: the image must be built `FROM texera-computing-unit-worker`, because the container has to start a Texera worker and join the cluster. The step's own libraries go on top of that base, so each operator gets its own envir
 onment.

```mermaid
flowchart LR
  subgraph UI["Workflow editor"]
    P["Operator property panel<br/>Offload: on<br/>Instance type: local-2g<br/>Image: myorg/qc-step:1.0"]
  end
  subgraph CU["Computing-unit master (JVM, Pekko cluster seed :2552)"]
    O["RuntimeOffloadOrchestrator<br/>rent one container per offloaded operator<br/>pin the operator's workers to it"]
  end
  subgraph D["Docker daemon"]
    C1["container: qc<br/>image qc:1.0 (numpy 1.26)<br/>Texera worker :2560"]
    C2["container: plot_qc<br/>image plot-qc:1.0 (numpy 2.1)<br/>Texera worker :2561"]
    C3["container: interactive<br/>image interactive:1.0 (numpy 2.0)<br/>Texera worker :2562"]
  end
  P -->|run| O
  O -->|docker run --memory=2g image| C1 & C2 & C3
  C1 & C2 & C3 <-->|"Pekko Artery TCP (Kryo tuples + control)"| CU
```

**Same design as Nextflow, different transport.** In both systems the unit of isolation is "one container per step, each with its own image." The difference is what crosses the container boundary. Nextflow stages input files into the task's work directory and runs a shell command; the container is a process sandbox and data moves as files. In Texera the container runs a full worker that joins the master's Pekko cluster over Artery TCP, and the operator's input and output tuples are streamed to it the same way they are streamed between in-process workers. Nothing in the operator changes; placement is the only difference, and the dataflow engine keeps working across the boundary (backpressure, pause/resume, stats).

GitHub link: https://github.com/apache/texera/discussions/7475#discussioncomment-18807240

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org

Re: [D] Running an operator inside a container [texera]

["sshiv012 (via GitHub)" <gi...@apache.org>]

GitHub user sshiv012 added a comment to the discussion: Running an operator inside a container

Python virtual environments were also considered, but Ha might be using R workflows - hence it is not a complete solution.

GitHub link: https://github.com/apache/texera/discussions/7475#discussioncomment-18807283

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org