You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "usbrandon (via GitHub)" <gi...@apache.org> on 2024/12/29 19:50:07 UTC
[I] [Feature Request]: Add watchdog / auto kill functionality to "Execute a Process" actions. (hop)
usbrandon opened a new issue, #4748:
URL: https://github.com/apache/hop/issues/4748
### What would you like to happen?
Many times when functionality is limited within Hop it is useful to execute a process that carries out the tasks and then returns. Problems arise when the process hangs or never finishes. I would like to request some kind of traditional watchdog timer / parameterization that would automatically kill the process if it takes to long or never occurs so my pipelines and workflows can eventually finish.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add watchdog / auto kill functionality to "Execute a Process" actions. (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4748:
URL: https://github.com/apache/hop/issues/4748#issuecomment-2578131638
Hi @bamaer. These are great ideas for sure. The Apache Airflow "Retry" task can do these things, and many enterprise schedulers as well. What you usually can't do however, is have the Hop process itself killed if it takes too long to complete. That is what this issue tries to solve.
Doing a re-start of pipelines and workflows, even remembering until where a workflow executed succesfully and restarting from that point, is something that can be done. I wrote something similar for another similar tool many years ago. Perhaps it deserves its own issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add watchdog / auto kill functionality to "Execute a Process" actions. (hop)
Posted by "usbrandon (via GitHub)" <gi...@apache.org>.
usbrandon commented on issue #4748:
URL: https://github.com/apache/hop/issues/4748#issuecomment-2578500889
Not sure if I intended to have Hop kill itself after some timeout, but the
point was that I may build a bash command line that runs a docker container
with a python script inside. If anything goes wrong or hangs in the
process that Hop let's it start, then there is no recovery but to kill all
of Hop. I wish that I could kill off that process that I had hop start in
within the pipeline, or workflow, whichever causes the sidecar execution.
I agree on Airflow. We are exploring that. I am interested in how SLA's
are declared and to what effect. From what I understand they are only an
event marker and Airflow does nothing about it but note that such and such
an SLA was violated, but it doesn't seem to kill the job or trigger other
logic. I am still exploring so I might come back with a different
understanding. Please teach me if you know differently. It is very
appealing to have a single pane of glass for scheduling. We have a mixture
of things executing like serverless functions, scripts that put elements on
an SQS queue that cause processing, and then Hop for ingestion of landed
files once the scaled up serverless data gathering tasks are done
downloading where Hop can see them.
Thanks for the consideration on this request. I hope the extra context
helps.
On Wed, Jan 8, 2025 at 10:41 AM Matt Casters ***@***.***>
wrote:
> Hi @bamaer <https://github.com/bamaer>. These are great ideas for sure.
> The Apache Airflow "Retry" task can do these things, and many enterprise
> schedulers as well. What you usually can't do however, is have the Hop
> process itself killed if it takes too long to complete. That is what this
> issue tries to solve.
> Doing a re-start of pipelines and workflows, even remembering until where
> a workflow executed succesfully and restarting from that point, is
> something that can be done. I wrote something similar for another similar
> tool many years ago. Perhaps it deserves its own issue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hop/issues/4748#issuecomment-2578131638>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAJNF5SOAMNKVI4IDSLEW4L2JVIMVAVCNFSM6AAAAABULCSOCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZYGEZTCNRTHA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add watchdog / auto kill functionality to "Execute a Process" actions. (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4748:
URL: https://github.com/apache/hop/issues/4748#issuecomment-2572811359
I would perhaps make this a general option in the Workflow and Pipeline run configurations.
In case the timeout option is set, a separate thread will be started alongside the Workflow or Pipeline to stop after the given time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add watchdog / auto kill functionality to "Execute a Process" actions. (hop)
Posted by "rvirgolireply (via GitHub)" <gi...@apache.org>.
rvirgolireply commented on issue #4748:
URL: https://github.com/apache/hop/issues/4748#issuecomment-2593247445
> I would perhaps make this a general option in the Workflow and Pipeline run configurations. In case the timeout option is set, a separate thread will be started alongside the Workflow or Pipeline to stop after the given time.
Hey, this would be really appreciated! I was about to open a feature request on this exact topic. Being able to stop a workflow or a pipeline after a given time would be so handy!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add watchdog / auto kill functionality to "Execute a Process" actions. (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4748:
URL: https://github.com/apache/hop/issues/4748#issuecomment-2573695529
if we want to kill pipelines/workflows based on a timeout or other parameters (available disk space etc), we could also look at restart options (nb of retries, wait time between retries).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org