You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Sachin Pasalkar <Sa...@symantec.com> on 2015/11/23 14:38:16 UTC
Trident topology showing improper values in UI
Hi,
We are developing the trident topology, we observed that the acking in UI increase continuously even though we do not process any message. I found description what those acks are https://github.com/miguno/kafka-storm-starter/issues/5. However, is there any plan to show users processed message? As a user, I am not interested in seeing these acks. I am very much interesting in acks for my messages.
Thanks,
Sachin
Re: Trident topology showing improper values in UI
Posted by Sachin Pasalkar <Sa...@symantec.com>.
Thanks :) Will do.
From: Bobby Evans <ev...@yahoo-inc.com>>
Reply-To: Bobby Evans <ev...@yahoo-inc.com>>
Date: Monday, 23 November 2015 11:34 pm
To: Sachin Pasalkar <sa...@symantec.com>>, "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Re: Trident topology showing improper values in UI
File a JIRA in issues.apache.org/jira under STORM describing what you want to see with storm and the UI especially for trident. We are in the process of merging with the JStorm project and they have completely redone how metrics work, so with their new code it might be more doable.
- Bobby
On Monday, November 23, 2015 10:52 AM, Sachin Pasalkar <Sa...@symantec.com>> wrote:
I also found that we can configure our own matrices to check how many tuples are processed
http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
Can’t we have this as part of trident doing internally?
From: Sachin Pasalkar <sa...@symantec.com>>
Date: Monday, 23 November 2015 10:14 pm
To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>, Bobby Evans <ev...@yahoo-inc.com>>
Subject: Re: Trident topology showing improper values in UI
Thanks Bobby for reply. However, I am aware of these things but my question different, when topology is not processing message still count on UI increases (Both on topology page/component page). Also, I have hardly seen the expected count of message processed on component page.
From: Bobby Evans <ev...@yahoo-inc.com.INVALID>>
Reply-To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>, Bobby Evans <ev...@yahoo-inc.com>>
Date: Monday, 23 November 2015 8:17 pm
To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Re: Trident topology showing improper values in UI
The metrics are not always the most clear on the UI, and trident does not make it any clearer. First of all there are several different modes on the UI. You can look at the metrics over different time periods, 10 mins, 3 hours, 1 day, and all time. If you have clicked on one of these links the page will display numbers for just this range, except in the section labeled for specific time periods. There is also a button that decides if we should include system stats or not. If this is selected the aggregate numbers will include acks. If it is not selected they will not.
The other thing to be aware of is that trident is a layer that sits on top of storm so the UI displays what the trident topology was compiled down to, not as much of what your code looks like. Trident spouts actually run in a bolt, but control logic is in the actual spout. You can still pull out the number of messages that a trident spout emitted in most cases, but you need to know where to look, and you might need to click on the component page for the bolt running the spout to be able to divide the counts up by which stream they were written to. - Bobby
On Monday, November 23, 2015 7:38 AM, Sachin Pasalkar <Sa...@symantec.com>> wrote:
Hi,
We are developing the trident topology, we observed that the acking in UI increase continuously even though we do not process any message. I found description what those acks are https://github.com/miguno/kafka-storm-starter/issues/5. However, is there any plan to show users processed message? As a user, I am not interested in seeing these acks. I am very much interesting in acks for my messages.
Thanks,
Sachin
Re: Trident topology showing improper values in UI
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
File a JIRA in issues.apache.org/jira under STORM describing what you want to see with storm and the UI especially for trident. We are in the process of merging with the JStorm project and they have completely redone how metrics work, so with their new code it might be more doable.
- Bobby
On Monday, November 23, 2015 10:52 AM, Sachin Pasalkar <Sa...@symantec.com> wrote:
I also found that we can configure our own matrices to check how many tuples are processed http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
Can’t we have this as part of trident doing internally?
From: Sachin Pasalkar <sa...@symantec.com>
Date: Monday, 23 November 2015 10:14 pm
To: "dev@storm.apache.org" <de...@storm.apache.org>, Bobby Evans <ev...@yahoo-inc.com>
Subject: Re: Trident topology showing improper values in UI
Thanks Bobby for reply. However, I am aware of these things but my question different, when topology is not processing message still count on UI increases (Both on topology page/component page). Also, I have hardly seen the expected count of message processed on component page.
From: Bobby Evans <ev...@yahoo-inc.com.INVALID>
Reply-To: "dev@storm.apache.org" <de...@storm.apache.org>, Bobby Evans <ev...@yahoo-inc.com>
Date: Monday, 23 November 2015 8:17 pm
To: "dev@storm.apache.org" <de...@storm.apache.org>
Subject: Re: Trident topology showing improper values in UI
The metrics are not always the most clear on the UI, and trident does not make it any clearer. First of all there are several different modes on the UI. You can look at the metrics over different time periods, 10 mins, 3 hours, 1 day, and all time. If you have clicked on one of these links the page will display numbers for just this range, except in the section labeled for specific time periods. There is also a button that decides if we should include system stats or not. If this is selected the aggregate numbers will include acks. If it is not selected they will not.
The other thing to be aware of is that trident is a layer that sits on top of storm so the UI displays what the trident topology was compiled down to, not as much of what your code looks like. Trident spouts actually run in a bolt, but control logic is in the actual spout. You can still pull out the number of messages that a trident spout emitted in most cases, but you need to know where to look, and you might need to click on the component page for the bolt running the spout to be able to divide the counts up by which stream they were written to. - Bobby
On Monday, November 23, 2015 7:38 AM, Sachin Pasalkar <Sa...@symantec.com> wrote:
Hi,
We are developing the trident topology, we observed that the acking in UI increase continuously even though we do not process any message. I found description what those acks arehttps://github.com/miguno/kafka-storm-starter/issues/5. However, is there any plan to show users processed message? As a user, I am not interested in seeing these acks. I am very much interesting in acks for my messages.
Thanks,Sachin
Re: Trident topology showing improper values in UI
Posted by Sachin Pasalkar <Sa...@symantec.com>.
I also found that we can configure our own matrices to check how many tuples are processed
http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
Can’t we have this as part of trident doing internally?
From: Sachin Pasalkar <sa...@symantec.com>>
Date: Monday, 23 November 2015 10:14 pm
To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>, Bobby Evans <ev...@yahoo-inc.com>>
Subject: Re: Trident topology showing improper values in UI
Thanks Bobby for reply. However, I am aware of these things but my question different, when topology is not processing message still count on UI increases (Both on topology page/component page). Also, I have hardly seen the expected count of message processed on component page.
From: Bobby Evans <ev...@yahoo-inc.com.INVALID>>
Reply-To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>, Bobby Evans <ev...@yahoo-inc.com>>
Date: Monday, 23 November 2015 8:17 pm
To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Re: Trident topology showing improper values in UI
The metrics are not always the most clear on the UI, and trident does not make it any clearer. First of all there are several different modes on the UI. You can look at the metrics over different time periods, 10 mins, 3 hours, 1 day, and all time. If you have clicked on one of these links the page will display numbers for just this range, except in the section labeled for specific time periods. There is also a button that decides if we should include system stats or not. If this is selected the aggregate numbers will include acks. If it is not selected they will not.
The other thing to be aware of is that trident is a layer that sits on top of storm so the UI displays what the trident topology was compiled down to, not as much of what your code looks like. Trident spouts actually run in a bolt, but control logic is in the actual spout. You can still pull out the number of messages that a trident spout emitted in most cases, but you need to know where to look, and you might need to click on the component page for the bolt running the spout to be able to divide the counts up by which stream they were written to. - Bobby
On Monday, November 23, 2015 7:38 AM, Sachin Pasalkar <Sa...@symantec.com>> wrote:
Hi,
We are developing the trident topology, we observed that the acking in UI increase continuously even though we do not process any message. I found description what those acks are https://github.com/miguno/kafka-storm-starter/issues/5. However, is there any plan to show users processed message? As a user, I am not interested in seeing these acks. I am very much interesting in acks for my messages.
Thanks,
Sachin
Re: Trident topology showing improper values in UI
Posted by Sachin Pasalkar <Sa...@symantec.com>.
Thanks Bobby for reply. However, I am aware of these things but my question different, when topology is not processing message still count on UI increases (Both on topology page/component page). Also, I have hardly seen the expected count of message processed on component page.
From: Bobby Evans <ev...@yahoo-inc.com.INVALID>>
Reply-To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>, Bobby Evans <ev...@yahoo-inc.com>>
Date: Monday, 23 November 2015 8:17 pm
To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Re: Trident topology showing improper values in UI
The metrics are not always the most clear on the UI, and trident does not make it any clearer. First of all there are several different modes on the UI. You can look at the metrics over different time periods, 10 mins, 3 hours, 1 day, and all time. If you have clicked on one of these links the page will display numbers for just this range, except in the section labeled for specific time periods. There is also a button that decides if we should include system stats or not. If this is selected the aggregate numbers will include acks. If it is not selected they will not.
The other thing to be aware of is that trident is a layer that sits on top of storm so the UI displays what the trident topology was compiled down to, not as much of what your code looks like. Trident spouts actually run in a bolt, but control logic is in the actual spout. You can still pull out the number of messages that a trident spout emitted in most cases, but you need to know where to look, and you might need to click on the component page for the bolt running the spout to be able to divide the counts up by which stream they were written to. - Bobby
On Monday, November 23, 2015 7:38 AM, Sachin Pasalkar <Sa...@symantec.com>> wrote:
Hi,
We are developing the trident topology, we observed that the acking in UI increase continuously even though we do not process any message. I found description what those acks are https://github.com/miguno/kafka-storm-starter/issues/5. However, is there any plan to show users processed message? As a user, I am not interested in seeing these acks. I am very much interesting in acks for my messages.
Thanks,
Sachin
Re: Trident topology showing improper values in UI
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
The metrics are not always the most clear on the UI, and trident does not make it any clearer. First of all there are several different modes on the UI. You can look at the metrics over different time periods, 10 mins, 3 hours, 1 day, and all time. If you have clicked on one of these links the page will display numbers for just this range, except in the section labeled for specific time periods. There is also a button that decides if we should include system stats or not. If this is selected the aggregate numbers will include acks. If it is not selected they will not.
The other thing to be aware of is that trident is a layer that sits on top of storm so the UI displays what the trident topology was compiled down to, not as much of what your code looks like. Trident spouts actually run in a bolt, but control logic is in the actual spout. You can still pull out the number of messages that a trident spout emitted in most cases, but you need to know where to look, and you might need to click on the component page for the bolt running the spout to be able to divide the counts up by which stream they were written to. - Bobby
On Monday, November 23, 2015 7:38 AM, Sachin Pasalkar <Sa...@symantec.com> wrote:
Hi,
We are developing the trident topology, we observed that the acking in UI increase continuously even though we do not process any message. I found description what those acks are https://github.com/miguno/kafka-storm-starter/issues/5. However, is there any plan to show users processed message? As a user, I am not interested in seeing these acks. I am very much interesting in acks for my messages.
Thanks,
Sachin