You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "ErickTrindade-Orit (via GitHub)" <gi...@apache.org> on 2025/05/12 13:45:22 UTC
[I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
ErickTrindade-Orit opened a new issue, #5304:
URL: https://github.com/apache/hop/issues/5304
### What would you like to happen?
Hi team,
It would be extremely helpful to have a feature that allows users to "enter" or open a running pipeline or workflow to see its current execution status — similar to how it's possible in Pentaho.
This would make it easier to troubleshoot long-running jobs or monitor processes without having to wait for them to finish or relying only on logs. Ideally, we’d be able to see which step is currently being executed and the progress so far.
### Issue Priority
Priority: 3
### Issue Component
Component: Hop Gui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "CarlosJuncher03 (via GitHub)" <gi...@apache.org>.
CarlosJuncher03 commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3891446905
I tested this activity and it's working perfectly. However, would it be difficult to add metadata to the ETL? I created a new activity for this purpose, but I still have this question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva closed issue #5304: [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows
URL: https://github.com/apache/hop/issues/5304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "steewel (via GitHub)" <gi...@apache.org>.
steewel commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-2887366504
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3069303464
That's exactly what an execution information location can do for you.
https://hop.apache.org//manual/latest/hop-gui/perspective-execution-information.html
I can recommend sending the execution information to local files, Neo4j or an Elastic server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3852912103
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3891591002
> I tested this activity and it's working perfectly. However, would it be difficult to add metadata to the ETL? I created a new activity for this purpose, but I still have this question.
Do you mean the ETL Metadata Injection transform? I noticed that we can't look into these executions.
I'll create a new issue for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3853741727
Fair point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3853449105
Indeed @hansva. The Previous Devepment Instance only tracked things in the GUI and not very well. I think that it should be possible to implement something that links a running sub-pipeline or sub-workflow to the executing transform or action.
You can store this information in `IExtensionData.getExtensionDataMap()`.
The main concern would be leaking memory. We'd probably need options to "forget" completed work ... or not. Visual indicators on the actions and transforms like `running`, `failed` or `completed` icons are already present. Perhaps we can implement a click-through options for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3891602961
See also: #6559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3853659314
At first all it needs to be is practical. We can make it beautiful later. If you go down the route of storing the information, you might as well get it from an Execution Information Location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3853146866
I think both can live next to each other, but it has to be bettern then what it was in said previous platform.
We can use live drill down for development and the execution information for history and runs using the CLI or other platforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3853618315
I've been drafting something that I'll bring forward for discussion. But I am not unhappy with the progress I have made; the main issue I am pondering about is how to integrate it cleanly into the UI.
<img width="898" height="511" alt="Image" src="https://github.com/user-attachments/assets/d7378df9-0b8f-42f6-aba4-2c0793e378c0" />
Currently, I have a dialog popping up where you can select which instance of a loop you want to open and what its current state and parameters were. But it might be cleaner to also implement the `Info` tab the execution information perspective has.
I would like the result to be something actionable. If you have a loop and it fails on iteration x, we would want to be able to restart a pipeline on that iteration with the state it had to make debugging even more powerful, or at least have great insights on what and why it is going wrong.
Also wondering if we should include/store the data sniffers as it would be great information to have for each iteration but it might blow up memory consumption during development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Ability to Monitor Live Execution of Pipelines and Workflows (hop)
Posted by "CarlosJuncher03 (via GitHub)" <gi...@apache.org>.
CarlosJuncher03 commented on issue #5304:
URL: https://github.com/apache/hop/issues/5304#issuecomment-3891665556
That's right, I even opened an issue improvement thread (https://github.com/apache/hop/issues/6559) because I didn't know if it was a bug or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org