You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "alexdesousapereira (via GitHub)" <gi...@apache.org> on 2025/04/16 11:11:27 UTC
[D] Logging Pipelines to Database from Workflow Execution (hop)
GitHub user alexdesousapereira created a discussion: Logging Pipelines to Database from Workflow Execution
Description:
Hi team,
I found [this article](https://www.know.bi/blog/our-blog-1/step-by-step-guide-to-workflow-logging-26) that explains how to configure logging for a workflow and a pipeline separately, and it works great.
However, what I'm looking for is a way to automatically store the pipeline logs when executing a workflow.
I tried creating separate log database connections for both workflow and pipelines, but it didnโt work as expected โ only the workflow log is being saved.
๐ My question is:
Is there currently any supported way to make the workflow execution also persist the logs of the pipelines it triggers? Or maybe thereโs something Iโm missing in the configuration?
Thanks in advance for any guidance!
GitHub link: https://github.com/apache/hop/discussions/5191
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Logging Pipelines to Database from Workflow Execution (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Logging Pipelines to Database from Workflow Execution
I have created #5199 to change the defaults and update the documentation a bit
GitHub link: https://github.com/apache/hop/discussions/5191#discussioncomment-12864052
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Logging Pipelines to Database from Workflow Execution (hop)
Posted by "alexdesousapereira (via GitHub)" <gi...@apache.org>.
GitHub user alexdesousapereira added a comment to the discussion: Logging Pipelines to Database from Workflow Execution
Good morning, team! โ๏ธ
You can solve this by disabling the "Logging parent workflows only" option, as highlighted in the screenshot below. โ
When this option is enabled, Apache Hop only logs the parent workflow, ignoring the logs of any pipelines it triggers.
๐ง By unchecking this option, the system will also capture and store the logs of the pipelines executed within the workflow.
๐ธ Please see the visual evidence highlighting the correct setting:
GitHub link: https://github.com/apache/hop/discussions/5191#discussioncomment-12865602
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Logging Pipelines to Database from Workflow Execution (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Logging Pipelines to Database from Workflow Execution
Hi All,
If you disable the `Logging parent pipelines only` flag in the pipeline log it will start logging the triggered pipelines.
I suspect you also want to turn off that flag in the workflow log to have logging activated for subsequent workflow actions.
I'll change the defaults to disable that flag
GitHub link: https://github.com/apache/hop/discussions/5191#discussioncomment-12863910
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Logging Pipelines to Database from Workflow Execution (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc added a comment to the discussion: Logging Pipelines to Database from Workflow Execution
Hello @alexdesousapereira,
usually with the default options the nested pipelines/workflows log is included in the main workflow log, although with some limitations.
Beware that you can find the full workflow log in the `workflowLogging` field from the Workflow Logging transform, but you'll find it as you could read it from a normal console. To give it some structure you need to split the field by lines, do some parsing on the date part to get the log "moment", and possibly add some constant fields containing the workflow and/or other references.
_(reference discussions: #4968, #4896)_
GitHub link: https://github.com/apache/hop/discussions/5191#discussioncomment-12854126
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Logging Pipelines to Database from Workflow Execution (hop)
Posted by "alexdesousapereira (via GitHub)" <gi...@apache.org>.
GitHub user alexdesousapereira deleted a comment on the discussion: Logging Pipelines to Database from Workflow Execution
Good morning, team! โ๏ธ
You can solve this by disabling the "Logging parent workflows only" option, as highlighted in the screenshot below. โ
When this option is enabled, Apache Hop only logs the parent workflow, ignoring the logs of any pipelines it triggers.
๐ง By unchecking this option, the system will also capture and store the logs of the pipelines executed within the workflow.
๐ธ Please see the visual evidence highlighting the correct setting:
GitHub link: https://github.com/apache/hop/discussions/5191#discussioncomment-12865602
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Logging Pipelines to Database from Workflow Execution (hop)
Posted by "alexdesousapereira (via GitHub)" <gi...@apache.org>.
GitHub user alexdesousapereira added a comment to the discussion: Logging Pipelines to Database from Workflow Execution
Good morning
You can solve this by disabling the "Logging parent workflows only" option, as highlighted in the screenshot below. โ
When this option is enabled, Apache Hop only logs the parent workflow, ignoring the logs of any pipelines it triggers.
๐ง By unchecking this option, the system will also capture and store the logs of the pipelines executed within the workflow.
๐ธ Please see the visual evidence highlighting the correct setting:
GitHub link: https://github.com/apache/hop/discussions/5191#discussioncomment-12865613
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org