You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by 刘睿 <24...@qq.com> on 2019/03/08 05:15:56 UTC
Monitor Module in IoTDB
Hi all,
I am LiuRui (second-year graduated student) from the IoTDB group. Recently am working on the Monitor Module in IoTDB which is to store some monitoring statistics like data file size and successfully ingested data point number into an internal storage group (root.stats). Monitor module is an important part of a TSDB system and other TSDBs like InfluxDB or KairosDB also have some internal metrics. The related JIRA issue is https://issues.apache.org/jira/browse/IOTDB-23 <https://issues.apache.org/jira/browse/IOTDB-23>. I would be very glad to discuss the monitoring function with you all.
Best wishes,
LiuRui
Re: Monitor Module in IoTDB
Posted by 刘睿 <24...@qq.com>.
Hi,
I agree that the previous design and implementation of the monitor module have many issues and we should redesign and refactor the code. The last PR#88 is developed from early commit, which means its implementation is based on the previous design. However, since recently we have many improvements and modifications in the code, some are highly related to this module like FileNodeManager and FileNodeProcessor refactor, which causes some issues if I merge the latest master. I think it is necessary to reconsider the design and implementation.
Best,
LiuRui
> 在 2019年3月9日，上午12:36，Xiangdong Huang <sa...@gmail.com> 写道：
>
> Hi,
>
>> 1. How can you prevent user to create a timeseries named as
> 'root.stats.xxx', will conflict occur?
> Maybe move `root.stats.xxx` to `root.system.stats` is better... I think
> it is acceptable that set `root.system` as a reserved word...
>
>> 2. Can you provide an user-guide to demonstrate what do these timeseries
> mean and how can i query them?
>> 3. Do i need to write some sql statements by jdbc to get these statistics
> , or there are interfaces to directly get them?
>
> Indeed, do not forget modify the user-manual. But because the PR's title is
> [phase 1.1], maybe the author will submit it latter...
>
> I have reviewed the PR, I think it is basically OK.....
>
> BUT!
> As now I am thinking how to refactor FileNodeManager and FileNodeProcessor,
> I find that the IStatistic interface has too many functions to be
> implemented and many works can be done by StatMonitor service in fact.
>
> Now, Each IStatistic implementation handles its monitored data and provids
> a function `getAllStatisticsValue()` (and 3 other functions) to let
> StatMonitor service get data and then let StatMonitor service write data
> on disk. It is ok... but lead to FileNodeProcessor and FileNodeManager (and
> all other implementations of IStatistic) having too many logic codes that
> they actually do not care.
>
> So, how about just leave two functions in the interface: (1)
> registerToStatMonitor(String name); and (2) reportToStatMonitor(TsRecord
> record).
>
> The StatMonitor just needs to provide a function to these monitored
> instances: isRegistered(String name, String measurement), and provide
> query interfaces to users by JMX.
>
> That is, all monitored instances do not need to temporary store monitored
> data in memory, they just push the data to the StatMonitor Service.
> StatMonitor Service can handle that.
>
> It also requires that,
> (1) Given a instance name, StatMonitor knows how many measurements the name
> has.
> (2) If we use a Map<instance name, Map<measurement name, value>> or
> Map<instance name, Map<measurement name, List<value>>> to save data in
> memory in StatMonitor, we need to consider the loss caused by concurrency
> control of the first level of the Map (which is no need to consider in
> current implementation).
>
> The new implementation is totally an event-driven pattern or Observer
> Pattern (in design Pattern). StatMonitor can decide how to take action: put
> a report into memory temporarily and then flush it on disk per 5 seconds,
> or just discard some of them if the system is overhead...
>
>
> Best,
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> 徐毅 <xu...@126.com> 于2019年3月8日周五 下午8:57写道：
>
>> Hi
>>
>>
>> I have some questions：
>> 1）what is the meaning of ' successfully ingested data point number'?
>> 2）As you mentioned, statistics information is stored in root.stats,
>> 1. How can you prevent user to create a timeseries named as
>> 'root.stats.xxx', will conflict occur?
>> 2. Can you provide an user-guide to demonstrate what do these
>> timeseries mean and how can i query them?
>> 3. Do i need to write some sql statements by jdbc to get these
>> statistics , or there are interfaces to directly get them?
>> 3) Have your tested your codes and make sure your statistics is correct?
>>
>>
>> Thanks
>> XuYi
>>
>>
>> On 3/8/2019 13:15，刘睿<24...@qq.com> wrote：
>> Hi all,
>>
>> I am LiuRui (second-year graduated student) from the IoTDB group. Recently
>> am working on the Monitor Module in IoTDB which is to store some monitoring
>> statistics like data file size and successfully ingested data point number
>> into an internal storage group (root.stats). Monitor module is an important
>> part of a TSDB system and other TSDBs like InfluxDB or KairosDB also have
>> some internal metrics. The related JIRA issue is
>> https://issues.apache.org/jira/browse/IOTDB-23 <
>> https://issues.apache.org/jira/browse/IOTDB-23>. I would be very glad to
>> discuss the monitoring function with you all.
>>
>> Best wishes,
>> LiuRui
>
Re: Monitor Module in IoTDB
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
> 1. How can you prevent user to create a timeseries named as
'root.stats.xxx', will conflict occur?
Maybe move `root.stats.xxx` to `root.system.stats` is better... I think
it is acceptable that set `root.system` as a reserved word...
> 2. Can you provide an user-guide to demonstrate what do these timeseries
mean and how can i query them?
> 3. Do i need to write some sql statements by jdbc to get these statistics
, or there are interfaces to directly get them?
Indeed, do not forget modify the user-manual. But because the PR's title is
[phase 1.1], maybe the author will submit it latter...
I have reviewed the PR, I think it is basically OK.....
BUT!
As now I am thinking how to refactor FileNodeManager and FileNodeProcessor,
I find that the IStatistic interface has too many functions to be
implemented and many works can be done by StatMonitor service in fact.
Now, Each IStatistic implementation handles its monitored data and provids
a function `getAllStatisticsValue()` (and 3 other functions) to let
StatMonitor service get data and then let StatMonitor service write data
on disk. It is ok... but lead to FileNodeProcessor and FileNodeManager (and
all other implementations of IStatistic) having too many logic codes that
they actually do not care.
So, how about just leave two functions in the interface: (1)
registerToStatMonitor(String name); and (2) reportToStatMonitor(TsRecord
record).
The StatMonitor just needs to provide a function to these monitored
instances: isRegistered(String name, String measurement), and provide
query interfaces to users by JMX.
That is, all monitored instances do not need to temporary store monitored
data in memory, they just push the data to the StatMonitor Service.
StatMonitor Service can handle that.
It also requires that,
(1) Given a instance name, StatMonitor knows how many measurements the name
has.
(2) If we use a Map<instance name, Map<measurement name, value>> or
Map<instance name, Map<measurement name, List<value>>> to save data in
memory in StatMonitor, we need to consider the loss caused by concurrency
control of the first level of the Map (which is no need to consider in
current implementation).
The new implementation is totally an event-driven pattern or Observer
Pattern (in design Pattern). StatMonitor can decide how to take action: put
a report into memory temporarily and then flush it on disk per 5 seconds,
or just discard some of them if the system is overhead...
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
徐毅 <xu...@126.com> 于2019年3月8日周五 下午8:57写道：
> Hi
>
>
> I have some questions：
> 1）what is the meaning of ' successfully ingested data point number'?
> 2）As you mentioned, statistics information is stored in root.stats,
> 1. How can you prevent user to create a timeseries named as
> 'root.stats.xxx', will conflict occur?
> 2. Can you provide an user-guide to demonstrate what do these
> timeseries mean and how can i query them?
> 3. Do i need to write some sql statements by jdbc to get these
> statistics , or there are interfaces to directly get them?
> 3) Have your tested your codes and make sure your statistics is correct?
>
>
> Thanks
> XuYi
>
>
> On 3/8/2019 13:15，刘睿<24...@qq.com> wrote：
> Hi all,
>
> I am LiuRui (second-year graduated student) from the IoTDB group. Recently
> am working on the Monitor Module in IoTDB which is to store some monitoring
> statistics like data file size and successfully ingested data point number
> into an internal storage group (root.stats). Monitor module is an important
> part of a TSDB system and other TSDBs like InfluxDB or KairosDB also have
> some internal metrics. The related JIRA issue is
> https://issues.apache.org/jira/browse/IOTDB-23 <
> https://issues.apache.org/jira/browse/IOTDB-23>. I would be very glad to
> discuss the monitoring function with you all.
>
> Best wishes,
> LiuRui
Re: Monitor Module in IoTDB
Posted by 徐毅 <xu...@126.com>.
Hi
I have some questions：
1）what is the meaning of ' successfully ingested data point number'?
2）As you mentioned, statistics information is stored in root.stats,
1. How can you prevent user to create a timeseries named as 'root.stats.xxx', will conflict occur?
2. Can you provide an user-guide to demonstrate what do these timeseries mean and how can i query them?
3. Do i need to write some sql statements by jdbc to get these statistics , or there are interfaces to directly get them?
3) Have your tested your codes and make sure your statistics is correct?
Thanks
XuYi
On 3/8/2019 13:15，刘睿<24...@qq.com> wrote：
Hi all,
I am LiuRui (second-year graduated student) from the IoTDB group. Recently am working on the Monitor Module in IoTDB which is to store some monitoring statistics like data file size and successfully ingested data point number into an internal storage group (root.stats). Monitor module is an important part of a TSDB system and other TSDBs like InfluxDB or KairosDB also have some internal metrics. The related JIRA issue is https://issues.apache.org/jira/browse/IOTDB-23 <https://issues.apache.org/jira/browse/IOTDB-23>. I would be very glad to discuss the monitoring function with you all.
Best wishes,
LiuRui