You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by gongning <27...@qq.com.INVALID> on 2022/05/07 03:38:58 UTC
IoTDB supports declarative subscription discussions
Hi all
1. Topic
Target: Let IoTDB support declarative subscription mode.
Implementation: First, improve the trigger module, then support trigger running in the new cluster architecture and independent subscription syntax later.
2. Participants:
Gong Ning, Steve Yurong Su, Wei Liu, Qijun Xie.
3. Current Situation and Problems
a) Trigger only supports full-path subscription now. However, in the prefix-path subscription, trigger cannot know the data of which measurement triggers, because the fire method in trigger is missing path parameter.
b) Trigger only supports subscripting single event currently. We could optimize it for batch writing.
c) Trigger fire interface currently only supports synchronous call interface, so it is only suitable for triggering lightweight calls. For subscription forwarding and other functions, users are required to implement a service to realize asynchronous call and resource management. This kind of service lacks an abstract mechanism in IoTDB, so there may be as many services as trigger classes.
d) Trigger does not support cluster mode. The main reason is that trigger is a kind of stateful computing.
4. General Idea
a) Optimize the current trigger interface definition and its function.
i. Each fire in trigger should contain the full-path information.
ii. Trigger handler should support processing batch events.
b) Support database extension (CREATE EXTENSION) and pluggable features of services.
i. Support attaching external services (class implements IService) to the database through dynamic agent.
ii. Make Influxdb protocol service, HTTP/MQTT service, UDF external library and others pluggable.
iii. Each plugin will be universal and can support the internal services of the database: REST Service, Metrics Reporting and the HTTP triggering in trigger don’t need to go through different HTTP service implementations.
iv. Each plugin can define its own calling method and resource management method to reduce the overall resource overhead.
v. Reduce the dependence on packaging (more suitable for edge deployment), allow users to dynamically select plugins during compilation, and improve the scalability of IoTDB.
c) Support trigger running in the new cluster architecture.
i. Metadata storage of trigger. The amount of trigger data is small, so all nodes can store a copy.
ii. Only the leader node triggers.
iii. Only the low reliability of trigger is guaranteed to users. The persistence and consistency of the state are not guaranteed. Trigger only guarantees at least one call for each trigger.
d) Support subscription function or subscription syntax
i. Using HTTP or MQTT protocal.
ii. Could refer to the implementation of the build-in forward trigger (ForwardTrigger). Link: https://github.com/iskytek/iotdb/tree/trigger_forward.
5. Action
a) Ideas 1 and 2 are expected to complete the design and development in three weeks.
b) After action 1 is released, determine the design and development completion time of ideas 3 and 4
Best,
------------------------------------
gongning
IoTDB supports declarative subscription discussions
Posted by gongning <27...@qq.com.INVALID>.
1. Topic
Target: Let IoTDB support declarative subscription mode.
Implementation: First, improve the trigger module, then support trigger running in the new cluster architecture and independent subscription syntax later.
2. Participants:
Gong Ning, Steve Yurong Su, Wei Liu, Qijun Xie.
3. Current Situation and Problems
a) Trigger only supports full-path subscription now. However, in the prefix-path subscription, trigger cannot know the data of which measurement triggers, because the fire method in trigger is missing path parameter.
b) Trigger only supports subscripting single event currently. We could optimize it for batch writing.
c) Trigger fire interface currently only supports synchronous call interface, so it is only suitable for triggering lightweight calls. For subscription forwarding and other functions, users are required to implement a service to realize asynchronous call and resource management. This kind of service lacks an abstract mechanism in IoTDB, so there may be as many services as trigger classes.
d) Trigger does not support cluster mode. The main reason is that trigger is a kind of stateful computing.
4. General Idea
a) Optimize the current trigger interface definition and its function.
i. Each fire in trigger should contain the full-path information.
ii. Trigger handler should support processing batch events.
b) Support database extension (CREATE EXTENSION) and pluggable features of services.
i. Support attaching external services (class implements IService) to the database through dynamic agent.
ii. Make Influxdb protocol service, HTTP/MQTT service, UDF external library and others pluggable.
iii. Each plugin will be universal and can support the internal services of the database: REST Service, Metrics Reporting and the HTTP triggering in trigger don’t need to go through different HTTP service implementations.
iv. Each plugin can define its own calling method and resource management method to reduce the overall resource overhead.
v. Reduce the dependence on packaging (more suitable for edge deployment), allow users to dynamically select plugins during compilation, and improve the scalability of IoTDB.
c) Support trigger running in the new cluster architecture.
i. Metadata storage of trigger. The amount of trigger data is small, so all nodes can store a copy.
ii. Only the leader node triggers.
iii. Only the low reliability of trigger is guaranteed to users. The persistence and consistency of the state are not guaranteed. Trigger only guarantees at least one call for each trigger.
d) Support subscription function or subscription syntax
i. Using HTTP or MQTT protocal.
ii. Could refer to the implementation of the build-in forward trigger (ForwardTrigger). Link: https://github.com/iskytek/iotdb/tree/trigger_forward.
5. Action
a) Ideas 1 and 2 are expected to complete the design and development in three weeks.
b) After action 1 is released, determine the design and development completion time of ideas 3 and 4