You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hop.apache.org by Matt Casters <ma...@neo4j.com.INVALID> on 2022/06/28 07:57:55 UTC
Execution Information
Dear Hop-devs,
I've started the mother of all cases around a number of requirements I
captured of the last couple of years around "Execution Information":
https://issues.apache.org/jira/browse/HOP-4024
Make sure to leave your comments. I've linked the cases I thought were
relevant. Thanks for those!
Cheers,
Matt
Re: Execution Information
Posted by Matt Casters <ma...@neo4j.com.INVALID>.
Absolutely Ricardo. I think there are a lot of suboptimal things right now
in that scenario.
Capturing and keeping track of the execution lineage will benefit the
person looking at the list of executing workflows.
Drilling down into the various executing workflows, pipelines, transforms
and actions is just something that should just work naturally.
The main issue is just to have a flexible plugin system that allows all
sorts of back-ends to be used for the capturing and storing of this
information.
The user interface also needs to be disconnected from the editing of
pipelines and workflows in a new perspective.
In other words: a lot of work to be done :-)
Cheers,
Matt
On Tue, Jun 28, 2022 at 1:51 PM Ricardo Gouvea <ri...@openin.com.br>
wrote:
> Hi Matt,
>
> First, congratulations on the initiative, this work will have a very usable
> result.
>
> Well, some time ago I asked if it would be possible to capture the
> execution status (idle, running, finished) of the transforms of a pipeline
> that was executed by a workflow via hop-server. Just to confirm, would this
> be included in this scope of work?
>
> Best regards,
>
> Em ter., 28 de jun. de 2022 às 04:58, Matt Casters
> <ma...@neo4j.com.invalid> escreveu:
>
> > Dear Hop-devs,
> >
> > I've started the mother of all cases around a number of requirements I
> > captured of the last couple of years around "Execution Information":
> >
> > https://issues.apache.org/jira/browse/HOP-4024
> >
> > Make sure to leave your comments. I've linked the cases I thought were
> > relevant. Thanks for those!
> >
> > Cheers,
> > Matt
> >
>
>
> --
>
>
>
> <https://www.openin.com.br>
>
> Patrocinadora Oficial do
>
> Pentaho Day 2019
>
> Ricardo Gouvêa
>
> CEO
>
> ricardo@openin.com.br
>
> Mobile: +55 11 9 9638-9238
>
>
> Openin - Your Partner in Data
>
> www.openin.com.br
>
> [image: Openin no Linkedin] [image: Openin no Facebook]
> <https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
> <https://www.instagram.com/openinbr/> [image: Openin no Twitter]
> <https://twitter.com/OpeninBigData> [image: Openin no Youtube]
> <https://www.youtube.com/openinbigdata>
>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: Execution Information
Posted by Matt Casters <ma...@neo4j.com.INVALID>.
Absolutely Ricardo. I think there are a lot of suboptimal things right now
in that scenario.
Capturing and keeping track of the execution lineage will benefit the
person looking at the list of executing workflows.
Drilling down into the various executing workflows, pipelines, transforms
and actions is just something that should just work naturally.
The main issue is just to have a flexible plugin system that allows all
sorts of back-ends to be used for the capturing and storing of this
information.
The user interface also needs to be disconnected from the editing of
pipelines and workflows in a new perspective.
In other words: a lot of work to be done :-)
Cheers,
Matt
On Tue, Jun 28, 2022 at 1:51 PM Ricardo Gouvea <ri...@openin.com.br>
wrote:
> Hi Matt,
>
> First, congratulations on the initiative, this work will have a very usable
> result.
>
> Well, some time ago I asked if it would be possible to capture the
> execution status (idle, running, finished) of the transforms of a pipeline
> that was executed by a workflow via hop-server. Just to confirm, would this
> be included in this scope of work?
>
> Best regards,
>
> Em ter., 28 de jun. de 2022 às 04:58, Matt Casters
> <ma...@neo4j.com.invalid> escreveu:
>
> > Dear Hop-devs,
> >
> > I've started the mother of all cases around a number of requirements I
> > captured of the last couple of years around "Execution Information":
> >
> > https://issues.apache.org/jira/browse/HOP-4024
> >
> > Make sure to leave your comments. I've linked the cases I thought were
> > relevant. Thanks for those!
> >
> > Cheers,
> > Matt
> >
>
>
> --
>
>
>
> <https://www.openin.com.br>
>
> Patrocinadora Oficial do
>
> Pentaho Day 2019
>
> Ricardo Gouvêa
>
> CEO
>
> ricardo@openin.com.br
>
> Mobile: +55 11 9 9638-9238
>
>
> Openin - Your Partner in Data
>
> www.openin.com.br
>
> [image: Openin no Linkedin] [image: Openin no Facebook]
> <https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
> <https://www.instagram.com/openinbr/> [image: Openin no Twitter]
> <https://twitter.com/OpeninBigData> [image: Openin no Youtube]
> <https://www.youtube.com/openinbigdata>
>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: Execution Information
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Ricardo,
You are right, I did not think about the fact that workflows do not expand
to multiple pipelines on the hop server status page.
Which on the other hand would also be a good added functionality on that
overview.
Cheers,
Hans
On Tue, 28 Jun 2022 at 16:23, Ricardo Gouvea <ri...@openin.com.br> wrote:
> Hi @Hans Van Akelyen <ha...@leanwithdata.com>
>
> The conversation was as follows:
> We would like to create a real-time monitoring panel of the executions of
> pipelines and workflows at the level of their transforms and actions,
> however the pipelineStatus/workflowStatus still does not allow the return
> of the status of each of the transforms/actions and when executing a
> pipeline through a workflow it is also not possible to retrieve
> pipelineStatus.
>
> Best regards,
>
>
> Em ter., 28 de jun. de 2022 às 09:29, Hans Van Akelyen <
> hans.van.akelyen@gmail.com> escreveu:
>
> > Hi All,
> >
> > Great writeup Matt, I think a centralized spot to store all this
> > information will be a good step forward.
> >
> > @Ricardo, I might have missed that discussion but what is holding you
> back
> > from using the pipelineStatus endpoint to fetch this information?
> >
> > Kr,
> > Hans
> >
> > On Tue, 28 Jun 2022 at 13:51, Ricardo Gouvea <ri...@openin.com.br>
> > wrote:
> >
> > > Hi Matt,
> > >
> > > First, congratulations on the initiative, this work will have a very
> > usable
> > > result.
> > >
> > > Well, some time ago I asked if it would be possible to capture the
> > > execution status (idle, running, finished) of the transforms of a
> > pipeline
> > > that was executed by a workflow via hop-server. Just to confirm, would
> > this
> > > be included in this scope of work?
> > >
> > > Best regards,
> > >
> > > Em ter., 28 de jun. de 2022 às 04:58, Matt Casters
> > > <ma...@neo4j.com.invalid> escreveu:
> > >
> > > > Dear Hop-devs,
> > > >
> > > > I've started the mother of all cases around a number of requirements
> I
> > > > captured of the last couple of years around "Execution Information":
> > > >
> > > > https://issues.apache.org/jira/browse/HOP-4024
> > > >
> > > > Make sure to leave your comments. I've linked the cases I thought
> were
> > > > relevant. Thanks for those!
> > > >
> > > > Cheers,
> > > > Matt
> > > >
> > >
> > >
> > > --
> > >
> > >
> > >
> > > <https://www.openin.com.br>
> > >
> > > Patrocinadora Oficial do
> > >
> > > Pentaho Day 2019
> > >
> > > Ricardo Gouvêa
> > >
> > > CEO
> > >
> > > ricardo@openin.com.br
> > >
> > > Mobile: +55 11 9 9638-9238
> > >
> > >
> > > Openin - Your Partner in Data
> > >
> > > www.openin.com.br
> > >
> > > [image: Openin no Linkedin] [image: Openin no Facebook]
> > > <https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
> > > <https://www.instagram.com/openinbr/> [image: Openin no Twitter]
> > > <https://twitter.com/OpeninBigData> [image: Openin no Youtube]
> > > <https://www.youtube.com/openinbigdata>
> > >
> >
>
>
> --
>
>
>
> <https://www.openin.com.br>
>
> Patrocinadora Oficial do
>
> Pentaho Day 2019
>
> Ricardo Gouvêa
>
> CEO
>
> ricardo@openin.com.br
>
> Mobile: +55 11 9 9638-9238
>
>
> Openin - Your Partner in Data
>
> www.openin.com.br
>
> [image: Openin no Linkedin] [image: Openin no Facebook]
> <https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
> <https://www.instagram.com/openinbr/> [image: Openin no Twitter]
> <https://twitter.com/OpeninBigData> [image: Openin no Youtube]
> <https://www.youtube.com/openinbigdata>
>
Re: Execution Information
Posted by Ricardo Gouvea <ri...@openin.com.br>.
Hi @Hans Van Akelyen <ha...@leanwithdata.com>
The conversation was as follows:
We would like to create a real-time monitoring panel of the executions of
pipelines and workflows at the level of their transforms and actions,
however the pipelineStatus/workflowStatus still does not allow the return
of the status of each of the transforms/actions and when executing a
pipeline through a workflow it is also not possible to retrieve
pipelineStatus.
Best regards,
Em ter., 28 de jun. de 2022 às 09:29, Hans Van Akelyen <
hans.van.akelyen@gmail.com> escreveu:
> Hi All,
>
> Great writeup Matt, I think a centralized spot to store all this
> information will be a good step forward.
>
> @Ricardo, I might have missed that discussion but what is holding you back
> from using the pipelineStatus endpoint to fetch this information?
>
> Kr,
> Hans
>
> On Tue, 28 Jun 2022 at 13:51, Ricardo Gouvea <ri...@openin.com.br>
> wrote:
>
> > Hi Matt,
> >
> > First, congratulations on the initiative, this work will have a very
> usable
> > result.
> >
> > Well, some time ago I asked if it would be possible to capture the
> > execution status (idle, running, finished) of the transforms of a
> pipeline
> > that was executed by a workflow via hop-server. Just to confirm, would
> this
> > be included in this scope of work?
> >
> > Best regards,
> >
> > Em ter., 28 de jun. de 2022 às 04:58, Matt Casters
> > <ma...@neo4j.com.invalid> escreveu:
> >
> > > Dear Hop-devs,
> > >
> > > I've started the mother of all cases around a number of requirements I
> > > captured of the last couple of years around "Execution Information":
> > >
> > > https://issues.apache.org/jira/browse/HOP-4024
> > >
> > > Make sure to leave your comments. I've linked the cases I thought were
> > > relevant. Thanks for those!
> > >
> > > Cheers,
> > > Matt
> > >
> >
> >
> > --
> >
> >
> >
> > <https://www.openin.com.br>
> >
> > Patrocinadora Oficial do
> >
> > Pentaho Day 2019
> >
> > Ricardo Gouvêa
> >
> > CEO
> >
> > ricardo@openin.com.br
> >
> > Mobile: +55 11 9 9638-9238
> >
> >
> > Openin - Your Partner in Data
> >
> > www.openin.com.br
> >
> > [image: Openin no Linkedin] [image: Openin no Facebook]
> > <https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
> > <https://www.instagram.com/openinbr/> [image: Openin no Twitter]
> > <https://twitter.com/OpeninBigData> [image: Openin no Youtube]
> > <https://www.youtube.com/openinbigdata>
> >
>
--
<https://www.openin.com.br>
Patrocinadora Oficial do
Pentaho Day 2019
Ricardo Gouvêa
CEO
ricardo@openin.com.br
Mobile: +55 11 9 9638-9238
Openin - Your Partner in Data
www.openin.com.br
[image: Openin no Linkedin] [image: Openin no Facebook]
<https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
<https://www.instagram.com/openinbr/> [image: Openin no Twitter]
<https://twitter.com/OpeninBigData> [image: Openin no Youtube]
<https://www.youtube.com/openinbigdata>
Re: Execution Information
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi All,
Great writeup Matt, I think a centralized spot to store all this
information will be a good step forward.
@Ricardo, I might have missed that discussion but what is holding you back
from using the pipelineStatus endpoint to fetch this information?
Kr,
Hans
On Tue, 28 Jun 2022 at 13:51, Ricardo Gouvea <ri...@openin.com.br> wrote:
> Hi Matt,
>
> First, congratulations on the initiative, this work will have a very usable
> result.
>
> Well, some time ago I asked if it would be possible to capture the
> execution status (idle, running, finished) of the transforms of a pipeline
> that was executed by a workflow via hop-server. Just to confirm, would this
> be included in this scope of work?
>
> Best regards,
>
> Em ter., 28 de jun. de 2022 às 04:58, Matt Casters
> <ma...@neo4j.com.invalid> escreveu:
>
> > Dear Hop-devs,
> >
> > I've started the mother of all cases around a number of requirements I
> > captured of the last couple of years around "Execution Information":
> >
> > https://issues.apache.org/jira/browse/HOP-4024
> >
> > Make sure to leave your comments. I've linked the cases I thought were
> > relevant. Thanks for those!
> >
> > Cheers,
> > Matt
> >
>
>
> --
>
>
>
> <https://www.openin.com.br>
>
> Patrocinadora Oficial do
>
> Pentaho Day 2019
>
> Ricardo Gouvêa
>
> CEO
>
> ricardo@openin.com.br
>
> Mobile: +55 11 9 9638-9238
>
>
> Openin - Your Partner in Data
>
> www.openin.com.br
>
> [image: Openin no Linkedin] [image: Openin no Facebook]
> <https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
> <https://www.instagram.com/openinbr/> [image: Openin no Twitter]
> <https://twitter.com/OpeninBigData> [image: Openin no Youtube]
> <https://www.youtube.com/openinbigdata>
>
Re: Execution Information
Posted by Ricardo Gouvea <ri...@openin.com.br>.
Hi Matt,
First, congratulations on the initiative, this work will have a very usable
result.
Well, some time ago I asked if it would be possible to capture the
execution status (idle, running, finished) of the transforms of a pipeline
that was executed by a workflow via hop-server. Just to confirm, would this
be included in this scope of work?
Best regards,
Em ter., 28 de jun. de 2022 às 04:58, Matt Casters
<ma...@neo4j.com.invalid> escreveu:
> Dear Hop-devs,
>
> I've started the mother of all cases around a number of requirements I
> captured of the last couple of years around "Execution Information":
>
> https://issues.apache.org/jira/browse/HOP-4024
>
> Make sure to leave your comments. I've linked the cases I thought were
> relevant. Thanks for those!
>
> Cheers,
> Matt
>
--
<https://www.openin.com.br>
Patrocinadora Oficial do
Pentaho Day 2019
Ricardo Gouvêa
CEO
ricardo@openin.com.br
Mobile: +55 11 9 9638-9238
Openin - Your Partner in Data
www.openin.com.br
[image: Openin no Linkedin] [image: Openin no Facebook]
<https://www.facebook.com/Openinbr/> [image: Openin no Instagram]
<https://www.instagram.com/openinbr/> [image: Openin no Twitter]
<https://twitter.com/OpeninBigData> [image: Openin no Youtube]
<https://www.youtube.com/openinbigdata>