Tracking message processing in my application

["Halfon, Roey" <ro...@sap.com>]

Hi,
We are looking for a monitoring solution for our dataflow - Track the progress of incoming messages while they are processed.
I'll clarify - we want to build some service which will show status for each incoming message. And in case of failures to give some detailed information.

I thought about the following:
First, every incoming message will be assigned with some id.
We can create a "Reporter" (A logger with some additional capabilities)  which each operator can communicate with, and update a status and more relevant information. These details can be stroed in kibana (ES) for example.
Then, we need to create another service which will query kibana and shows the results.

What do you think about that? Is there any built-in solution for that? (flink built in metrics are not relevant here because they don't help to track a single message)

How are you logging and tracking your processed messages?
Is there any documentation or some use cases that I can learn from?

Thanks,
Roey.

Re:Tracking message processing in my application

[Haibo Sun <su...@163.com>]

Hi,  Roey


> What do you think about that? 


I would have some concerns about throughput and latency, so I think that the operators should report state data asynchronously and in batches to minimize the impact of monitoring on the normal business processing. In addition, If the amount of business data is too large in a certain period of time, which leads to the operator-side state data backlog exceeding the set capacity, how to deal with the operator-side state data also needs to be considered, whether to discard or block the business data processing, or other ways?


Best,
Haibo 

At 2019-07-04 20:29:02, "Halfon, Roey" <ro...@sap.com> wrote:


Hi,

We are looking for a monitoring solution for our dataflow – Track the progress of incoming messages while they are processed.
I'll clarify – we want to build some service which will show status for each incoming message. And in case of failures to give some detailed information.

I thought about the following:
First, every incoming message will be assigned with some id.
We can create a "Reporter" (A logger with some additional capabilities)  which each operator can communicate with, and update a status and more relevant information. These details can be stroed in kibana (ES) for example.
Then, we need to create another service which will query kibana and shows the results.

What do you think about that? Is there any built-in solution for that? (flink built in metrics are not relevant here because they don't help to track a single message)

How are you logging and tracking your processed messages?
Is there any documentation or some use cases that I can learn from?

Thanks,
Roey.