You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Lukas Ott <ot...@gmail.com> on 2020/01/11 12:36:02 UTC
Historical data from Ditto
Hi Iotdb developers,
First short introduction: I met Julian Feinhauer and saw the talk about
Apache IoTDB and then the following question came up: How can Apache IoTDB
support the Histical data from Eclipse Ditto?
This issue has been raised on Eclipse Ditto code repository:
https://github.com/eclipse/ditto/issues/545 and i was wondering if there
are plans to support this. As one of the developers already mentioned IoTDB
as potential solution it would be awesome if IoTDB would provide a way to
safe Historical Data of Ditto.
Additionally
https://stackoverflow.com/questions/56993884/how-to-get-historical-sensor-data-of-a-registered-hono-device
Bosch IoT Suite seems to have an own way to safe historical data.
Thanks in advance for your answers!
Kind regards
Lukas
Re: Historical data from Ditto
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Thanks for the very detailed analysis Xiangdong!
Indeed a full and proper integration may be hard, but one could start with smaller steps and e.g. leave out the access management yet.
Ditto even supports Access Management on each single point.
I also wanted to share the github issue with you that the ditto community is using to discuss that point: https://github.com/eclipse/ditto/issues/545
Julian
Am 11.01.20, 17:03 schrieb "Xiangdong Huang" <sa...@gmail.com>:
Hi,
I have quickly read the document of Ditto.
In my understanding, Ditto allow consume data from kafka or MQTT server to
get the latest status of a Things (or, a device), and provides API to let
users know the latest value of a device's feature (measurement concept in
IoTDB).
I think it is easy to persist the data into IoTDB meanwhile. But if we want
to provide high write performance (e.g., more than 10 million data points
per second), we may need to use batch_write api, which is a little complex.
Another thing that need to be discussed is that as ditto does not have api
for query historical data. we have to design and add one.
The third is, it sounds that Ditto does not require K8s and Keycloak
Authentication.
I quickly searched Keycloak and find it is a solution for Identity and
Access Management. Hmm... IoTDB has a user/role management module, but does
not support Keycloak...
I think currently the community lacks committers to develop this
feature..... But we can create an issue about that, and maybe some familiar
guys can do it.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Xiangdong Huang <sa...@gmail.com> 于2020年1月11日周六 下午10:05写道：
> Hi,
>
> Thanks Lukas. I also need to take some time to familiarize with Ditto.
> :-)
>
> > One important question for me would be the possibility to deploy Apache
> IoTDB on Kubernetes
>
> Now we can deploy IoTDB as a docker container.
> So I think it is ok to deploy it on K8S. But as it is a database,
> you may have to use a network filesystem or run IoTDB on a certain node by
> force.
>
> As for the Keycloak Authentication, I also need time to familiarize with
> it..
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Lukas Ott <ot...@gmail.com> 于2020年1月11日周六 下午8:46写道：
>
>> Hi Julian,
>>
>> Great to hear from you. Just for information I was active on
>> https://gitter.im/eclipse/ditto and raised same issues on stackoverflow
>> (ended up in a mini PR for Ditto concerning Kubernetes API break)
>>
>> So lets see how we can bring these two systems closer together. :)
>>
>> One important question for me would be the possibility to deploy Apache
>> IoTDB on Kubernetes (including Keycloak Authentication -> That is already
>> included in Eclipse Ditto)
>>
>> Lukas
>>
>> Am Sa., 11. Jan. 2020 um 13:41 Uhr schrieb Julian Feinauer <
>> j.feinauer@pragmaticminds.de>:
>>
>> > Hi Lukas,
>> >
>> > What a coincidence!
>> > Yesterday I met some of the ditto devs from Bosch and am currently
>> > thinking about exactly that!
>> >
>> > So let me familiarize a bit more with ditto then everything its
>> possible!
>> >
>> > Julian
>> >
>> > Von meinem Mobiltelefon gesendet
>> >
>> >
>> > -------- Ursprüngliche Nachricht --------
>> > Von: Lukas Ott <ot...@gmail.com>
>> > Datum: Sa., 11. Jan. 2020, 13:36
>> > An: dev@iotdb.apache.org
>> > Betreff: Historical data from Ditto
>> > Hi Iotdb developers,
>> >
>> > First short introduction: I met Julian Feinhauer and saw the talk about
>> > Apache IoTDB and then the following question came up: How can Apache
>> IoTDB
>> > support the Histical data from Eclipse Ditto?
>> >
>> > This issue has been raised on Eclipse Ditto code repository:
>> > https://github.com/eclipse/ditto/issues/545 and i was wondering if
>> there
>> > are plans to support this. As one of the developers already mentioned
>> IoTDB
>> > as potential solution it would be awesome if IoTDB would provide a way
>> to
>> > safe Historical Data of Ditto.
>> >
>> > Additionally
>> >
>> >
>> https://stackoverflow.com/questions/56993884/how-to-get-historical-sensor-data-of-a-registered-hono-device
>> > Bosch IoT Suite seems to have an own way to safe historical data.
>> >
>> > Thanks in advance for your answers!
>> >
>> > Kind regards
>> > Lukas
>> >
>>
>
Re: Historical data from Ditto
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
I have quickly read the document of Ditto.
In my understanding, Ditto allow consume data from kafka or MQTT server to
get the latest status of a Things (or, a device), and provides API to let
users know the latest value of a device's feature (measurement concept in
IoTDB).
I think it is easy to persist the data into IoTDB meanwhile. But if we want
to provide high write performance (e.g., more than 10 million data points
per second), we may need to use batch_write api, which is a little complex.
Another thing that need to be discussed is that as ditto does not have api
for query historical data. we have to design and add one.
The third is, it sounds that Ditto does not require K8s and Keycloak
Authentication.
I quickly searched Keycloak and find it is a solution for Identity and
Access Management. Hmm... IoTDB has a user/role management module, but does
not support Keycloak...
I think currently the community lacks committers to develop this
feature..... But we can create an issue about that, and maybe some familiar
guys can do it.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Xiangdong Huang <sa...@gmail.com> 于2020年1月11日周六 下午10:05写道：
> Hi,
>
> Thanks Lukas. I also need to take some time to familiarize with Ditto.
> :-)
>
> > One important question for me would be the possibility to deploy Apache
> IoTDB on Kubernetes
>
> Now we can deploy IoTDB as a docker container.
> So I think it is ok to deploy it on K8S. But as it is a database,
> you may have to use a network filesystem or run IoTDB on a certain node by
> force.
>
> As for the Keycloak Authentication, I also need time to familiarize with
> it..
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Lukas Ott <ot...@gmail.com> 于2020年1月11日周六 下午8:46写道：
>
>> Hi Julian,
>>
>> Great to hear from you. Just for information I was active on
>> https://gitter.im/eclipse/ditto and raised same issues on stackoverflow
>> (ended up in a mini PR for Ditto concerning Kubernetes API break)
>>
>> So lets see how we can bring these two systems closer together. :)
>>
>> One important question for me would be the possibility to deploy Apache
>> IoTDB on Kubernetes (including Keycloak Authentication -> That is already
>> included in Eclipse Ditto)
>>
>> Lukas
>>
>> Am Sa., 11. Jan. 2020 um 13:41 Uhr schrieb Julian Feinauer <
>> j.feinauer@pragmaticminds.de>:
>>
>> > Hi Lukas,
>> >
>> > What a coincidence!
>> > Yesterday I met some of the ditto devs from Bosch and am currently
>> > thinking about exactly that!
>> >
>> > So let me familiarize a bit more with ditto then everything its
>> possible!
>> >
>> > Julian
>> >
>> > Von meinem Mobiltelefon gesendet
>> >
>> >
>> > -------- Ursprüngliche Nachricht --------
>> > Von: Lukas Ott <ot...@gmail.com>
>> > Datum: Sa., 11. Jan. 2020, 13:36
>> > An: dev@iotdb.apache.org
>> > Betreff: Historical data from Ditto
>> > Hi Iotdb developers,
>> >
>> > First short introduction: I met Julian Feinhauer and saw the talk about
>> > Apache IoTDB and then the following question came up: How can Apache
>> IoTDB
>> > support the Histical data from Eclipse Ditto?
>> >
>> > This issue has been raised on Eclipse Ditto code repository:
>> > https://github.com/eclipse/ditto/issues/545 and i was wondering if
>> there
>> > are plans to support this. As one of the developers already mentioned
>> IoTDB
>> > as potential solution it would be awesome if IoTDB would provide a way
>> to
>> > safe Historical Data of Ditto.
>> >
>> > Additionally
>> >
>> >
>> https://stackoverflow.com/questions/56993884/how-to-get-historical-sensor-data-of-a-registered-hono-device
>> > Bosch IoT Suite seems to have an own way to safe historical data.
>> >
>> > Thanks in advance for your answers!
>> >
>> > Kind regards
>> > Lukas
>> >
>>
>
Re: Historical data from Ditto
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
Thanks Lukas. I also need to take some time to familiarize with Ditto. :-)
> One important question for me would be the possibility to deploy Apache
IoTDB on Kubernetes
Now we can deploy IoTDB as a docker container.
So I think it is ok to deploy it on K8S. But as it is a database,
you may have to use a network filesystem or run IoTDB on a certain node by
force.
As for the Keycloak Authentication, I also need time to familiarize with
it..
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Lukas Ott <ot...@gmail.com> 于2020年1月11日周六 下午8:46写道：
> Hi Julian,
>
> Great to hear from you. Just for information I was active on
> https://gitter.im/eclipse/ditto and raised same issues on stackoverflow
> (ended up in a mini PR for Ditto concerning Kubernetes API break)
>
> So lets see how we can bring these two systems closer together. :)
>
> One important question for me would be the possibility to deploy Apache
> IoTDB on Kubernetes (including Keycloak Authentication -> That is already
> included in Eclipse Ditto)
>
> Lukas
>
> Am Sa., 11. Jan. 2020 um 13:41 Uhr schrieb Julian Feinauer <
> j.feinauer@pragmaticminds.de>:
>
> > Hi Lukas,
> >
> > What a coincidence!
> > Yesterday I met some of the ditto devs from Bosch and am currently
> > thinking about exactly that!
> >
> > So let me familiarize a bit more with ditto then everything its possible!
> >
> > Julian
> >
> > Von meinem Mobiltelefon gesendet
> >
> >
> > -------- Ursprüngliche Nachricht --------
> > Von: Lukas Ott <ot...@gmail.com>
> > Datum: Sa., 11. Jan. 2020, 13:36
> > An: dev@iotdb.apache.org
> > Betreff: Historical data from Ditto
> > Hi Iotdb developers,
> >
> > First short introduction: I met Julian Feinhauer and saw the talk about
> > Apache IoTDB and then the following question came up: How can Apache
> IoTDB
> > support the Histical data from Eclipse Ditto?
> >
> > This issue has been raised on Eclipse Ditto code repository:
> > https://github.com/eclipse/ditto/issues/545 and i was wondering if there
> > are plans to support this. As one of the developers already mentioned
> IoTDB
> > as potential solution it would be awesome if IoTDB would provide a way to
> > safe Historical Data of Ditto.
> >
> > Additionally
> >
> >
> https://stackoverflow.com/questions/56993884/how-to-get-historical-sensor-data-of-a-registered-hono-device
> > Bosch IoT Suite seems to have an own way to safe historical data.
> >
> > Thanks in advance for your answers!
> >
> > Kind regards
> > Lukas
> >
>
Re: Historical data from Ditto
Posted by Lukas Ott <ot...@gmail.com>.
Hi Julian,
Great to hear from you. Just for information I was active on
https://gitter.im/eclipse/ditto and raised same issues on stackoverflow
(ended up in a mini PR for Ditto concerning Kubernetes API break)
So lets see how we can bring these two systems closer together. :)
One important question for me would be the possibility to deploy Apache
IoTDB on Kubernetes (including Keycloak Authentication -> That is already
included in Eclipse Ditto)
Lukas
Am Sa., 11. Jan. 2020 um 13:41 Uhr schrieb Julian Feinauer <
j.feinauer@pragmaticminds.de>:
> Hi Lukas,
>
> What a coincidence!
> Yesterday I met some of the ditto devs from Bosch and am currently
> thinking about exactly that!
>
> So let me familiarize a bit more with ditto then everything its possible!
>
> Julian
>
> Von meinem Mobiltelefon gesendet
>
>
> -------- Ursprüngliche Nachricht --------
> Von: Lukas Ott <ot...@gmail.com>
> Datum: Sa., 11. Jan. 2020, 13:36
> An: dev@iotdb.apache.org
> Betreff: Historical data from Ditto
> Hi Iotdb developers,
>
> First short introduction: I met Julian Feinhauer and saw the talk about
> Apache IoTDB and then the following question came up: How can Apache IoTDB
> support the Histical data from Eclipse Ditto?
>
> This issue has been raised on Eclipse Ditto code repository:
> https://github.com/eclipse/ditto/issues/545 and i was wondering if there
> are plans to support this. As one of the developers already mentioned IoTDB
> as potential solution it would be awesome if IoTDB would provide a way to
> safe Historical Data of Ditto.
>
> Additionally
>
> https://stackoverflow.com/questions/56993884/how-to-get-historical-sensor-data-of-a-registered-hono-device
> Bosch IoT Suite seems to have an own way to safe historical data.
>
> Thanks in advance for your answers!
>
> Kind regards
> Lukas
>
AW: Historical data from Ditto
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hi Lukas,
What a coincidence!
Yesterday I met some of the ditto devs from Bosch and am currently thinking about exactly that!
So let me familiarize a bit more with ditto then everything its possible!
Julian
Von meinem Mobiltelefon gesendet
-------- Ursprüngliche Nachricht --------
Von: Lukas Ott <ot...@gmail.com>
Datum: Sa., 11. Jan. 2020, 13:36
An: dev@iotdb.apache.org
Betreff: Historical data from Ditto
Hi Iotdb developers,
First short introduction: I met Julian Feinhauer and saw the talk about
Apache IoTDB and then the following question came up: How can Apache IoTDB
support the Histical data from Eclipse Ditto?
This issue has been raised on Eclipse Ditto code repository:
https://github.com/eclipse/ditto/issues/545 and i was wondering if there
are plans to support this. As one of the developers already mentioned IoTDB
as potential solution it would be awesome if IoTDB would provide a way to
safe Historical Data of Ditto.
Additionally
https://stackoverflow.com/questions/56993884/how-to-get-historical-sensor-data-of-a-registered-hono-device
Bosch IoT Suite seems to have an own way to safe historical data.
Thanks in advance for your answers!
Kind regards
Lukas