Pipeline preview

[Dominik Riemer <ri...@apache.org>]

Hi all,

 

after talking to some first-time users I discovered that our live dashboard is often used as some kind of “debugging feature” to better see if the created pipeline performs as expected. This leads to quite a confusing and time-consuming pipeline development approach (creating a pipeline, appending dashboard sinks everywhere, going to the dashboard, checking the output, going back to the pipeline editor to change the pipeline and so on).

So I quickly implemented a “preview feature” that can be enabled in the pipeline editor and then shows the output for each pipeline element in form of a table below the element (data streams & data processors and limited to 1 event/sec to avoid web browser performance issues).

In the background, a pipeline is started and stopped and we reuse the pipeline element live data fetcher that was already implemented for data streams.

 

It would be great if you could check if the feature is working as expected or if any changes are needed or critical bugs occur, as the implementation of the feature seemed to be a bit too easy and I’ve expected it to be much more complicated 😉

 

Dominik

 

 

Re: Pipeline preview

[Dominik Riemer <ri...@apache.org>]

Hi Philipp,
thanks for the feedback.
An alternative would be a continuous web socket connection from the core, but this has the disadvantage that the browser could get overloaded with events from high-frequency stream.
We could extend the long-polling interval so that the chance that events get lost is reduced.

I absolutely agree that the preview feature has not much value if any events get lost.

Dominik

On 2021/05/23 20:27:26, Philipp Zehnder <ze...@apache.org> wrote: 
> Hi Dominik,
> 
> This is a very cool feature and I already tried it.
> I really like how fast it is and it definitely improves the development experience and reduces the amount of (debug) dashboard sinks in a pipeline ;)
> 
> While trying, I have encountered a problem. I had a pipeline with a filter, which should only let an event through from time to time. 
> Unfortunately this was not displayed in the preview. When I started this with a data lake sink, the correct values arrived.
> I suspect the reason for this behavior is because we poll the values every two seconds in the preview component. 
> Do you have any ideas how to fix this? Because as a user you then think that the pipeline does not work as expected.
> 
> Philipp
> 
> > On 21. May 2021, at 12:24, Dominik Riemer <ri...@apache.org> wrote:
> > 
> > Hi all,
> > 
> > 
> > 
> > after talking to some first-time users I discovered that our live dashboard is often used as some kind of “debugging feature” to better see if the created pipeline performs as expected. This leads to quite a confusing and time-consuming pipeline development approach (creating a pipeline, appending dashboard sinks everywhere, going to the dashboard, checking the output, going back to the pipeline editor to change the pipeline and so on).
> > 
> > So I quickly implemented a “preview feature” that can be enabled in the pipeline editor and then shows the output for each pipeline element in form of a table below the element (data streams & data processors and limited to 1 event/sec to avoid web browser performance issues).
> > 
> > In the background, a pipeline is started and stopped and we reuse the pipeline element live data fetcher that was already implemented for data streams.
> > 
> > 
> > 
> > It would be great if you could check if the feature is working as expected or if any changes are needed or critical bugs occur, as the implementation of the feature seemed to be a bit too easy and I’ve expected it to be much more complicated 😉
> > 
> > 
> > 
> > Dominik
> > 
> > 
> > 
> > 
> > 
> 
> 

Re: Pipeline preview

[Philipp Zehnder <ze...@apache.org>]

Hi Dominik,

This is a very cool feature and I already tried it.
I really like how fast it is and it definitely improves the development experience and reduces the amount of (debug) dashboard sinks in a pipeline ;)

While trying, I have encountered a problem. I had a pipeline with a filter, which should only let an event through from time to time. 
Unfortunately this was not displayed in the preview. When I started this with a data lake sink, the correct values arrived.
I suspect the reason for this behavior is because we poll the values every two seconds in the preview component. 
Do you have any ideas how to fix this? Because as a user you then think that the pipeline does not work as expected.

Philipp

> On 21. May 2021, at 12:24, Dominik Riemer <ri...@apache.org> wrote:
> 
> Hi all,
> 
> 
> 
> after talking to some first-time users I discovered that our live dashboard is often used as some kind of “debugging feature” to better see if the created pipeline performs as expected. This leads to quite a confusing and time-consuming pipeline development approach (creating a pipeline, appending dashboard sinks everywhere, going to the dashboard, checking the output, going back to the pipeline editor to change the pipeline and so on).
> 
> So I quickly implemented a “preview feature” that can be enabled in the pipeline editor and then shows the output for each pipeline element in form of a table below the element (data streams & data processors and limited to 1 event/sec to avoid web browser performance issues).
> 
> In the background, a pipeline is started and stopped and we reuse the pipeline element live data fetcher that was already implemented for data streams.
> 
> 
> 
> It would be great if you could check if the feature is working as expected or if any changes are needed or critical bugs occur, as the implementation of the feature seemed to be a bit too easy and I’ve expected it to be much more complicated 😉
> 
> 
> 
> Dominik
> 
> 
> 
> 
>