You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Dominik Riemer <do...@bytefabrik.ai> on 2022/10/19 12:37:18 UTC

Pipeline Monitoring / STREAMPIPES-598

Hi,

I've worked on a new feature to collect monitoring infos for pipelines and adapters.
In the current release, only very basic monitoring info was displayed for running pipelines, which only worked when using Kafka as message broker and often displayed strange values as we were receiving the info directly from the Kafka admin client. The new implementation, independent from the underlying broker, collects metrics for adapters and pipeline elements (messages received, messages sent and last timestamp of received messages) and makes runtime exceptions available in the UI. Exceptions are shown in the "Logs" section within the details view of pipelines and adapters. On the SDK side, there is also a new method in the RuntimeContext of processors and sinks to forward custom exceptions to the core.

I hope this greatly improves transparency for users as runtime exceptions were previously only available in the log files.

There are two things left:

  *   I know that some community members have asked for a central REST endpoint which provides monitoring info in a single document, e.g., to read this from external monitoring software. This shouldn't be hard to implement now and I'm happy to assist.
  *   Exceptions for adapters are currently very basic, as there is now easy way to globally catch exceptions from adapters. I'll try to improve the StreamPipes Connect API and codebase first, which should then also ease the definition of integration tests Zike has started to work on.

I'll merge the changes into dev soon and I'm happy for any feedback!

Cheers
Dominik




Re: AW: Pipeline Monitoring / STREAMPIPES-598

Posted by Dominik Riemer <ri...@apache.org>.
Hi Philipp,

good point - I've directly added this and the adapter overview now shows the time of the last event as well as the number of events sent.

Dominik

On 2022/10/21 15:59:14 Philipp Zehnder wrote:
> Hi Dominik,
> 
> I really like this feature. For me, the view is now much clearer.
> 
> I have a suggestion for a further improvement. Can we integrate the date of the last published event into the “My adapters” view for each adapter.
> This would really help to directly see if all adapters are still running as expected.
> Maybe we can also directly include a rule into the UI that color codes when the last event exceeded a certain threshold.
> 
> What do you think?
> 
> Philipp
> 
> Von: Dominik Riemer <do...@bytefabrik.ai>
> Datum: Mittwoch, 19. Oktober 2022 um 14:37
> An: dev@streampipes.apache.org <de...@streampipes.apache.org>
> Betreff: Pipeline Monitoring / STREAMPIPES-598
> Hi,
> 
> I've worked on a new feature to collect monitoring infos for pipelines and adapters.
> In the current release, only very basic monitoring info was displayed for running pipelines, which only worked when using Kafka as message broker and often displayed strange values as we were receiving the info directly from the Kafka admin client. The new implementation, independent from the underlying broker, collects metrics for adapters and pipeline elements (messages received, messages sent and last timestamp of received messages) and makes runtime exceptions available in the UI. Exceptions are shown in the "Logs" section within the details view of pipelines and adapters. On the SDK side, there is also a new method in the RuntimeContext of processors and sinks to forward custom exceptions to the core.
> 
> I hope this greatly improves transparency for users as runtime exceptions were previously only available in the log files.
> 
> There are two things left:
> 
>   *   I know that some community members have asked for a central REST endpoint which provides monitoring info in a single document, e.g., to read this from external monitoring software. This shouldn't be hard to implement now and I'm happy to assist.
>   *   Exceptions for adapters are currently very basic, as there is now easy way to globally catch exceptions from adapters. I'll try to improve the StreamPipes Connect API and codebase first, which should then also ease the definition of integration tests Zike has started to work on.
> 
> I'll merge the changes into dev soon and I'm happy for any feedback!
> 
> Cheers
> Dominik
> 
> 
> 

AW: Pipeline Monitoring / STREAMPIPES-598

Posted by Philipp Zehnder <ph...@bytefabrik.ai>.
Hi Dominik,

I really like this feature. For me, the view is now much clearer.

I have a suggestion for a further improvement. Can we integrate the date of the last published event into the “My adapters” view for each adapter.
This would really help to directly see if all adapters are still running as expected.
Maybe we can also directly include a rule into the UI that color codes when the last event exceeded a certain threshold.

What do you think?

Philipp

Von: Dominik Riemer <do...@bytefabrik.ai>
Datum: Mittwoch, 19. Oktober 2022 um 14:37
An: dev@streampipes.apache.org <de...@streampipes.apache.org>
Betreff: Pipeline Monitoring / STREAMPIPES-598
Hi,

I've worked on a new feature to collect monitoring infos for pipelines and adapters.
In the current release, only very basic monitoring info was displayed for running pipelines, which only worked when using Kafka as message broker and often displayed strange values as we were receiving the info directly from the Kafka admin client. The new implementation, independent from the underlying broker, collects metrics for adapters and pipeline elements (messages received, messages sent and last timestamp of received messages) and makes runtime exceptions available in the UI. Exceptions are shown in the "Logs" section within the details view of pipelines and adapters. On the SDK side, there is also a new method in the RuntimeContext of processors and sinks to forward custom exceptions to the core.

I hope this greatly improves transparency for users as runtime exceptions were previously only available in the log files.

There are two things left:

  *   I know that some community members have asked for a central REST endpoint which provides monitoring info in a single document, e.g., to read this from external monitoring software. This shouldn't be hard to implement now and I'm happy to assist.
  *   Exceptions for adapters are currently very basic, as there is now easy way to globally catch exceptions from adapters. I'll try to improve the StreamPipes Connect API and codebase first, which should then also ease the definition of integration tests Zike has started to work on.

I'll merge the changes into dev soon and I'm happy for any feedback!

Cheers
Dominik