You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "HaruspexSan (via GitHub)" <gi...@apache.org> on 2026/02/26 22:33:01 UTC
[D] Best practice for scheduling multiple workflows (Docker Compose & Kubernetes)? (hop)
GitHub user HaruspexSan created a discussion: Best practice for scheduling multiple workflows (Docker Compose & Kubernetes)?
Hello Hop community,
I am building a data platform with Apache Hop (v2.17) and looking for the recommended way to schedule multiple .hwf workflows as part of my Master Thesis. Currently, I am using Hop-Web to create the workflows in the UI and I want to use something more lightweight to execute the workflows. My idea was splitting the workflows into topics, having one to parse and store data for each data domain. This includes saving prod data in a dedicated bi database.
I know from previous Q&A (#6105) that Hop Server isn't a time-based scheduler, and the standard approach is triggering the hop-run.sh CLI via a native scheduler (like cron) to run short-lived execution containers.
### My environments are:
- Local: Docker Compose (using apache/hop-web as the IDE)
- Production: Kubernetes via Helm
### Questions:
- Hop Execution: Instead of triggering 10 different containers for 10 workflows, is the standard Hop practice to create a single "Master Workflow" that calls the other .hwf files, and just schedule that one Master container?
- Local Scheduling: What is the recommended, lightweight approach for handling these schedules locally in Docker Compose without introducing a heavy orchestrator? Especially, if i want to run the workflows scheduled with cron or something similar.
- K8s Scaling (10-50+ workflows): When moving to Kubernetes, what is the best practice that works cleanly with the provided Apache Hop Helm charts? Is it standard to generate dozens of K8s CronJob manifests, or should I be looking at something else?
Finally, thank you for taking your time and helping me out.
Cheers,
Max
GitHub link: https://github.com/apache/hop/discussions/6676
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Best practice for scheduling multiple workflows (Docker Compose & Kubernetes)? (hop)
Posted by "dmainou (via GitHub)" <gi...@apache.org>.
GitHub user dmainou added a comment to the discussion: Best practice for scheduling multiple workflows (Docker Compose & Kubernetes)?
Hi,
you can trigger n number of short lived containers
or
- start a container running hop server.
- execute jobs on the server as required
- kill the server
or use and metal to run individual jobs using the crontabs
or execute in dataflow or similar
sky is the limit
GitHub link: https://github.com/apache/hop/discussions/6676#discussioncomment-15941838
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org