You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "pcserna (via GitHub)" <gi...@apache.org> on 2025/08/29 05:35:35 UTC
[D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
GitHub user pcserna created a discussion: Apache Airflow vs Hop Server (vs Hop-run)
Please, help me understand why it is favorable using Apache Airflow for orchestration: as I can see, with the DAG python script Airflow set up a new instance of Apache Hop, so if I run many DAGs in parallel, they would eat up resources, not to mention the warm-up time-overhead.
Ok, it is a good thing, if your Workflow fails you can start over, you can see the KPIs of your runs and you can set up cron in a pythonic way, but what else?
In a production environment you have to set up a backend database, too (which may not be the one you operate). Not to mention - as I see - it is preferred to use docker.
So, under the very end Apache Hop will do the ETL, right?
If I have to operate many Workflows and Pipelines e.g. in every 15 minutes in parallel, I think I would use as simple backend as I could with the lowest necessary hardware and software need. Do you think that doing many tasks with separated hop-runs have a comparable computing load to one hop-server?
Thanks for sharing your experiences!
GitHub link: https://github.com/apache/hop/discussions/5654
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "NicoDupont (via GitHub)" <gi...@apache.org>.
GitHub user NicoDupont added a comment to the discussion: Apache Airflow vs Hop Server (vs Hop-run)
95% of my Hop workflows are executed on a Hop server because it's convenient to be able to choose the execution machine (ram/cpu/access/env).
The logs are in Apache Airflow and it doesn't matter if the Hop server restarts.
@pcserna How many workflows do you have to execute?
You can just use cron tab if it's very low or a python script with the shedule package or APScheduler package.
I use Airflow because I have multiple scenarios (ssh, http, docker, dbt, python script) and over 200 DAGs...
You can see at Dagster, Prefect, Kestra, Luigi, etc..
There is a lot of alternatives to airflow.
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14254036
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "pcserna (via GitHub)" <gi...@apache.org>.
GitHub user pcserna added a comment to the discussion: Apache Airflow vs Hop Server (vs Hop-run)
Thanks for your opinions and use cases!
@NicoDupont I have lots of workflows on two servers. The parallelism and the load is optimized for reaching the results within a specified timeframe (there are also failsafe procedures). The servers are mounting shared spaces of mirrored fileservers (it is needed to start up backup servers in case of power or internet outages). Here I have the repos, batch files, python programs, settings, DBs, etc.
Many of the workflows refer to sub-workflows and Python procedures, other 3rd party programs. If a highlighted workflow fails an email is sent about it and there is no time to restart (because the same procedure will be started within minutes).
I use Carte as an orchestrator now and evaluating / migrating to Hop. The pdi.log is another info-source I use. The batch files are scheduled form Cron.
I started to use Carte because running workflows separately from a batch took a lot of time to warm up, eat-up resources, no visual feedback … it was straightforward.
I have to decide where to go, which technology should I go with. I prefer to use open-source platforms, of course, and I thank you for forking your work to Hop.
As I see now, running the same workflows in Airflow takes a lot of work: I should learn, integrate and operate a considerable amount of new technologies and separate the complex workflows in DAGs, and at the moment I do not have the impression / information that it is worth investing in. I have to consider the pros and cons.
I just got used to and I tried to maximize the capabilities of Carte – I know, this is the past, but my life would be easier if “**Export linked resources to server**” Hop procedure (for example) would create one (zipped ) xml and could be called from command line. :-)
Am I right that each Airflow calls a new running instance of Hop?
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14254925
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer edited a comment on the discussion: Apache Airflow vs Hop Server (vs Hop-run)
> If I have to operate many Workflows and Pipelines e.g. in every 15 minutes in parallel, I think I would use as simple backend as I could with the lowest necessary hardware and software need
what @hansva said, and in addition to that, if you have workloads that run every 15 minutes in parallel, I would do the opposite of using a backend as simple as possible and try to get all the help I can in terms of visual overview, easy access to logs etc. None of that is impossible with hop server, it just won't be easy/simple to maintain, monitor, troubleshoot etc.
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14253739
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Apache Airflow vs Hop Server (vs Hop-run)
It should still all be possible with the current Hop Server, it depends on how you used to start workflows.
The only thing that isn't available anymore is to use filepaths on the URL to start workflows but you can work around those limitations by (ab)using a werbservice to start your workflows.
Correct the airflow calls would start a new instance of Hop.
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14255506
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Apache Airflow vs Hop Server (vs Hop-run)
The reasons we tell people not to use the Hop Server:
- It is stateless (reboot and everything is gone)
- It has no scheduling
- It does not have an easy API to add workflows/pipelines to the server
- It is lacking good user management and dashboarding
The server was originally designed to be an extension of your local GUI. It was used to act as a remote location to test and run the things you were designing locally because you might not have access to a DB on your local machine, or not enough resources.
After that, it was extended to add some fun features, such as using it as an endpoint to provide the results of an executed pipeline.
Now for the actual scheduling problem... It all depends on your needs, requirements, and architecture.
If you are building workflows that need to run on fixed intervals, and you know the resources required.
A single instance and crontab with regular Hop run will be the simplest deployment.
You don't know your workloads, they can be triggered externally, users need an overview of what happened, it needs to be scalable?
Then you end up with an airflow deployment with scalable infrastructure.
From personal experience, except for the we need an endpoint to provide data to another system use case. We never use Hop Server.
Kr,
Hans
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14252393
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "mocoxk (via GitHub)" <gi...@apache.org>.
GitHub user mocoxk added a comment to the discussion: Apache Airflow vs Hop Server (vs Hop-run)
@pcserna I have a Hop server running and I use systemctl to ensure it automatically restarts in case the machine shuts down or reboots.
With Airflow, I use SSH Operator to connect on the apache hop and execute hop-run.sh, specifying that the execution should occur in "server". For the hop-run.sh process, I assign a maximum of 100MB to its JVM, since its sole purpose is to submit the job execution request to the Hop server, which itself is configured with greater resources to handle multiple parallel processes and several executions throughout the day.
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14265483
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Apache Airflow vs Hop Server (vs Hop-run)
> If I have to operate many Workflows and Pipelines e.g. in every 15 minutes in parallel, I think I would use as simple backend as I could with the lowest necessary hardware and software need
what @hansva said, and in addition to that, if you have workloads that run every 15 minutes in parallel, I would do the opposite of using a backend as simple as possible and try to get all the help I can in terms of visual overview, easy access to logs etc. None of that is impossible with hop server, it just won't be easy/simple to maintain, monitor, troubleshoot etc.
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14253739
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "NicoDupont (via GitHub)" <gi...@apache.org>.
GitHub user NicoDupont edited a comment on the discussion: Apache Airflow vs Hop Server (vs Hop-run)
95% of my Hop workflows are executed on a Hop server because it's convenient to be able to choose the execution machine (ram/cpu/access/env).
The logs are in Apache Airflow and it doesn't matter if the Hop server restarts.
And the apache airflow instance is not calibrate to execute hop workflows.
@pcserna How many workflows do you have to execute?
You can just use cron tab if it's very low or a python script with the shedule package or APScheduler package.
I use Airflow because I have multiple scenarios (ssh, http, docker, dbt, python script) and over 200 DAGs...
You can see at Dagster, Prefect, Kestra, Luigi, etc..
There is a lot of alternatives to airflow.
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14254036
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Airflow vs Hop Server (vs Hop-run) (hop)
Posted by "NicoDupont (via GitHub)" <gi...@apache.org>.
GitHub user NicoDupont added a comment to the discussion: Apache Airflow vs Hop Server (vs Hop-run)
Personally, I start Hop with the dockeroperator also with low memory and cpu, and execution takes place on a Hop server depending on the context.
GitHub link: https://github.com/apache/hop/discussions/5654#discussioncomment-14273261
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org