Re: Share a list of IoTDB integration tasks

[Xiangdong Huang <sa...@gmail.com>]

Hi Philipp,

I also like Java API.

Besides, I invite Steven Yurong ( @steveyurongsu@qq.com
<st...@qq.com> ) to join the discussion, who wrote the proposal.

Best,

-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Philipp Zehnder <ze...@apache.org> 于2020年4月1日周三 上午5:02写道：

> Hi Xiangdong,
>
> yes I think this feature is exactly what we need.
>
> I also had a look at the proposal in the comments and it looks very
> interesting.
>
> I have one question regarding the sinks. Do you also plan to integrate a
> programming API (e.g. in Java) or do you plan to rely on broker
> technologies?
> For us it would be good to have a Java API, which registers the query and
> a callback receiving the events.
> Alternatively, we could use a message broker to forward the events into
> the StreamPipes adapter, but then we would depend on another service.
> What do you think about that?
>
> Philipp
>
>
> > On 23. Mar 2020, at 15:17, Xiangdong Huang <sa...@gmail.com> wrote:
> >
> > Hi Philipp,
> >
> > That is really really big good news!
> >
> >> I think data set integration should be straightforward, but what about
> > the data stream adapter? Is it possible to subscribe to changes in IoTDB?
> >
> > Yes the data set adapter is straightforward.
> >
> > As for the data stream adapter....
> >
> >> Does this trigger have to be set in the database or can a client
> > subscribe to changes?
> >
> > We are doing that! Have a look about issue [1]. It is called trigger in
> > IoTDB.
> > In our design, if a user registers a trigger for a time series and claims
> > to send new data points to somewhere (e.g., Streampipes),
> > then, you can get the changes of data on that timeseries in stream way.
> > It is just a design and we have not implement it now.
> > How do you think about this?
> >
> > [1]
> >
> https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Philipp Zehnder <ze...@apache.org> 于2020年3月23日周一 下午8:17写道：
> >
> >> Hi Xiangdong,
> >>
> >> I come from the StreamPipes community and I also think it's a great idea
> >> to work together.
> >> We have already integrated IoTDB as a data sink in StreamPipes. Users
> can
> >> model their analytics pipelines and write the results directly into
> IoTDB.
> >> The next step would be to also get data from IoTDB, then we can read
> data,
> >> do stream calculations and write the results back.
> >>
> >> We have already integrated several adapters for databases (e.g.
> InfluxDB,
> >> mysql, ...) [1]. I would like to add such an adapter for IoTDB.
> >> Usually we distinguish between dataset and datastream adapters. With a
> >> data set adapter we poll the data once from the database and stream it
> >> through the pipeline.
> >> With data stream adapters we start the adapter and continuously read and
> >> process events.
> >> I think data set integration should be straightforward, but what about
> the
> >> data stream adapter? Is it possible to subscribe to changes in IoTDB?
> >>
> >> In [2] it is described that a user must add a trigger. Does this trigger
> >> have to be set in the database or can a client subscribe to changes?
> >>
> >> Philipp
> >>
> >> [1]
> >>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/streampipes-connect-adapter/src/main/java/org/apache/streampipes/connect/adapters
> >> <
> >>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/streampipes-connect-adapter/src/main/java/org/apache/streampipes/connect/adapters
> >>>
> >> [2]
> >>
> https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues
> >> <
> >>
> https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues
> >>>
> >>
> >>> On 20. Mar 2020, at 10:36, Julian Feinauer <
> j.feinauer@pragmaticminds.de>
> >> wrote:
> >>>
> >>> Hi Xiangdong,
> >>>
> >>> very nice to share it here!
> >>> Looking forward to Prometheus and Streampipes!
> >>> Andi f I can support with DBCP (or commons pool2 or whatever, I'm happy
> >> to help!).
> >>>
> >>> Julian
> >>>
> >>> Am 20.03.20, 02:55 schrieb "Xiangdong Huang" <sa...@gmail.com>:
> >>>
> >>>   Hi all,
> >>>
> >>>   I come from IoTDB community.
> >>>   Following Christofer's suggestion, I'd like to share the process of
> >> the
> >>>   integration work between IoTDB and some other Apache projects to let
> >> users
> >>>   manage time-series data easier.
> >>>
> >>>   1. PLC4X (done)
> >>>   Under the help of Julian and other guys, PLC4X has provided an
> >> example to
> >>>   write data directly to IoTDB [1].
> >>>
> >>>   2. Prometheus (just beginning)
> >>>   Prometheus is a popular data collection and event alert system for
> >> many
> >>>   applications (but maybe not so popular for IoT).
> >>>   We'd like to integrate Prometheus with IoTDB, by replacing
> >> Prometheus's
> >>>   data store with IoTDB [2].
> >>>
> >>>   3. Flink and RocketMQ (in-progress)
> >>>   Both Flink and rocketMQ are message queues and in many applications,
> >> they
> >>>   are the entrances of data and IoTDB is behind them [3] [4] [5].
> >>>
> >>>   4. MiniFi (just beginning)
> >>>   MiniFi is a dataflow management system. We'd like to integrate IoTDB
> >> with
> >>>   it to allow: write data from a processor to IoTDB, and consume data
> >> from
> >>>   IoTDB to other processors [6].
> >>>
> >>>   5. Streampipes (just beginning)
> >>>   We'd like to add trigger function in IoTDB to allow publish alerts,
> >> or do
> >>>   some stream calculation. A current possible solution is integrating
> >> with
> >>>   StreamPipes [8].
> >>>
> >>>   IoTDB also begin to integrate with some other projects (which are
> >> irrelated
> >>>   to IoT) but make it friendly to use IoTDB. like:
> >>>
> >>>   6. Calcite (almost done)
> >>>   Calcite provides a Standard SQL language to IoTDB, which make it easy
> >> to
> >>>   use. A pr is open and ask for code review [7].
> >>>
> >>>   7. Zeppelin (in-progress)
> >>>   Zeppelin gives us a web-based GUI to let users operate IoTDB
> >> interactively
> >>>   [9].
> >>>
> >>>   8. DataBase Connection Pool (just beginning)
> >>>   with a DBCP, developers do not need to write too many codes in their
> >>>   business logic codes.
> >>>
> >>>   Those are what we can consider.
> >>>   We also would like to hear more ideas to make it easier to use IoTDB
> >> in IoT
> >>>   applications.
> >>>   Welcome to join us if you are interested in some integration ideas.
> >>>
> >>>   [1]
> >>>
> >>
> https://github.com/apache/plc4x/tree/develop/plc4j/examples/hello-integration-iotdb
> >>>   [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-519
> >>>   [3] https://issues.apache.org/jira/browse/COMDEV-350
> >>>   [4] https://issues.apache.org/jira/browse/IOTDB-560
> >>>   [5] https://github.com/apache/incubator-iotdb/pull/817
> >>>   [6] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-518
> >>>   [7] https://github.com/apache/incubator-iotdb/pull/902
> >>>   [8] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516
> >>>   [9] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-515
> >>>
> >>>   Best,
> >>>   -----------------------------------
> >>>   Xiangdong Huang
> >>>   School of Software, Tsinghua University
> >>>
> >>>    黄向东
> >>>   清华大学 软件学院
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: iot-unsubscribe@apache.org
> >>> For additional commands, e-mail: iot-help@apache.org
> >>>
> >>
> >>
> >>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: iot-unsubscribe@apache.org
> For additional commands, e-mail: iot-help@apache.org
>
>