Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi all,

I am currently thinking, if it wouldn’t be a good idea to build something like an open-source Historian based on PLC4X and IoTDB.

For those of you, who don’t know what a Historian is. In the manufacturing industry they have these extremely expensive servers (usually you buy them as a bundle of hard- and software).
These servers are nothing else than a really crappy and brutally expensive Database for storing Time-Series data.

The thing is most commercial products are currently really having trouble to keep up with the increasing amount of Data being sent.

So instead of suggesting building an Historian at each customer’s site, I thought: Perhaps a ready-to-use solution based on open-source would be a good idea.
Not 100% sure where I’d locate such an initiative, but I would tend to see it more on the Database side.

I think IoTDB would be the perfect storage system, all we seem to need is some sort of rest-interface that matches the industry standards for querying the information and on the other side something like PLC4X to fill the database.

Here some examples:
https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf

What do you folks think?


Chris

Re: Building a PLC4X and IoTDB Historian?

[Otto Fowler <ot...@gmail.com>]

I would make it IOT Historian.
Historian is too generic at the apache level
And that name fits:  IOT DB, IOT Historian, IOT Visualize, IOT XXXX

On Mon, Nov 28, 2022 at 7:45 AM Christofer Dutz <ch...@c-ware.de>
wrote:

> And if we already agree on a name (I think “Apache Historian” would be a
> good name),
> we could even already work on code with that package-name and maven
> coordinates, so we don’t have to change much when coming to Apache?
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Monday, 28. November 2022 at 13:18
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> Indeed, your concerns are valid regarding creating an already abandoned
> TLP.
>
> In that case, I would opt for a private repo on GitHub which we invite
> everyone willing to help work on it and as soon as we have something
> workable, we make it public as part of a TLP?
>
> This avoids at least the fear of creating a new TLP that’s product doesn’t
> even see the light of day. And as soon as we have something we want to show
> the world, we join Apache?
>
> Would that be a compromise?
>
>
> Chris
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 13:14
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hey,
>
> as I said.. we could just do it as part of one of the TLPs like PLC4X or
> IoTDB easily or in a private repo as well, for sure.
> I just want to avoid to create a TLP which is soon abandoned, that’s my
> only concern.
> Regarding all your comments I totally agree and am with you.
>
> Julian
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 13:07
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> well … I would be hesitant to try something like this without the legal
> shield of the ASF … from what I’ve learned in the early days, Siemens and
> Co. continuously threatened to sue me for my work on PLC4X.
> The only thing that helped me then, was to say: I am doing everything
> legally, so you’d have trouble succeeding with that and you also can’t sue
> me but would have to sue the ASF and good luck on that not backfiring badly
> on you publicly.
>
> I know that in the sector of drivers in the past there were several
> occasions where open-source developers were annoyed so much by nuisance
> law-suites, that they gave up.
>
> I also know Historians are silly expensive systems. I haven’t seen one
> under 100k€ and the annual fees for certified solutions tend to start at
> 100k€/year/production-line … so there’s a lot of money for them to fear
> losing. So yes: I’m too scared to try something like this publicly without
> a legal shield.
>
> We could however work on something like this in a protected repo at GitHub
> etc. and go to TLP as soon as we have something.
>
> Would that be an option?
>
> Chris
>
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 12:57
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hey Chris,
>
> wrt testing I meant how „sustainable” the community is, that we intend to
> building.
> You know the effect pretty well if you ask “How wants to start this”
> everybody is like “me, me, me” and after 2 weeks you stand there alone.
> And one important aspect of a TLP for me is that the community is also
> “resilient” and will last “long” (whatever that means).
> And this is something I think is hard to guarantee in the current state.
>
> Technically I have no worries, this is something we would get to work
> rather easily, I see no “magics” hidden there.
>
> Julian
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 12:48
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> Well, the problem with starting at one Project as a subproject … We then
> need to make everyone involved or wanting to be involved in the other
> project to become committers in the other.
> I wouldn’t have problems with doing it under the hood of IoTDB, personally.
>
> A separate Project would have the benefit of giving the industry an
> impression that this is something built just for them instead of being a
> side-product of something that’s already there (I have given up expecting
> the Automation industry to use any form of common sense ;-) … guess that
> will be my definition of Industry 5.0 … “connected everything + common
> sense”)
>
> The incubator is intended for teaching the initial community how things
> work at Apache, I guess as we’d be starting with mainly people that have
> already brought projects from outside into the incubator and into TLPs,
> this teaching effort will not exist in this case. Even Justin mentioned
> that this would probably be a case for straight TLP.
>
> Regarding the testing … not sure I understand what you want to test? The
> idea of an open-source historian?
> I mean the need exists … I’ve seen it in multiple occasions and our
> friends from IoTDB have confirmed they have built or at least seen IoTDB
> serve as such a system.
> Not 100% sure what you want to try out.
>
> But I agree … we should probably start with an (online) workshop, as this
> time we’d probably be spread out a bit wider geographically, than with our
> PLC4X workshops in the past.
>
> Chris
>
>
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 12:37
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hey,
>
> I understand your point and also agree generally.
> But I personally dislike the straight to TLP route as this is something
> I’d like to try out first.
> And my point with the separate repo (or whatever) was just to make the
> community ready for the Incubator (where a codebase and community normally
> should exist).
>
> But the easiest way in any regard would be to start of as a subproject in
> an existing PMC. This would only require to get another repo and we could
> start (formally no need to even register it as a subproject).
> And the repo thing could be done by you alone I think as Chair.
>
> Next steps would then be more of brainstorming and workshoping and
> building something (whatever?).
> I think the timing is quite good before Christmas because if its
> intereting, I see myself contributing over the Holidays (better don’t show
> this mail to my wife…).
>
> Best
> Julian
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 09:27
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <
> hoeper@ltsoft.de>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> happy you like the idea ;-)
>
> We would probably outclass the established ones in point of price ;-)
> Right now, I think we should probably set up something that bundles IoTDB
> and PLC4X and then the “code of the project” is the bubble-wrap, that
> brings the two together and then provides an API that is equivalent to
> existing Historian products.
> I think the only way we can score points here, is if switching to it is as
> easy as possible and that only works if it’s sort of a drop-in replacement.
>
> Currently also discussing with other board members and the IPMC chair … he
> at least would not see us in the incubator. If there are enough PMC members
> of other projects and Apache Members on board, we could take the straight
> to TLP route.
>
> And I would really like to keep it at Apache and not set up something
> outside. This for multiple reasons:
>
>   *   Protection (The market for the established products is huge and the
> companies behind them are also huge with huge legal departments, I would
> fear the same as I did with PLC4X, when I started it … don’t want to be
> stuck in nuisance lawsuits, just filed for keeping us occupied)
>   *   The satement: we want Open-Source to be accepted by the industry and
> for me “Apache” is the form of Open-Source that I admire most. If it’s
> built up out of Apache stuff, it should be an Apache project (If we
> establish this in a way, that the industry notices Apache as a “vendor”,
> this will benefit many other projects)
>
> Chris
>
>
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 09:17
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <
> hoeper@ltsoft.de>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> first of all.. sorry for coming in so late.
> I think this is a great idea and there really is a product market fit for
> such a solution.
> I think technology wise its not "that complicated" to build such a system,
> I think many of the details that are necessary for a good adoption of such
> a software is in the areas around like documentation, marketing, more
> marketing, even more and even bolder marketing and sales (how to sell an
> open source project??).
>
> But technology wise I really like the idea and I think with the technology
> one has today it is very easy to "outclass" these established systems e.g.
> in performance, efficiency and als user friendliness.
> The only thing which might be challenging for the future are things like
> plugins or an extension system to allow users to customize their
> installation.
>
> Regarding the "separate project" approach I am not 100% certain.
> Personally, I would consider starting EITHER as a subproject in an
> existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere
> else and not directly go to the Incubator or something.
> Because I think we should really find out if there is enough developer
> interest to build and sustain such a system.
>
> But I'm totally in for such a system either way!
>
> Julian
>
> PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with
> historians
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 08:56
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Xiangdong,
>
> that’s perfect that some of you folks would be on board. I should probably
> check, if there are enough PLC4X folks on board too … would be a shame if
> it was just me ;-)
>
> And regarding the name … yeah … you are absolutely right: Historian might
> really be the ideal choice … as it instantly explains what it is … A
> Historian from the Apache Software Foundation and probably also no bias
> towards any regional culture.
>
> So (Speaking to my fellow PLC4X folks … who would be on board with this?)
>
>
> Chris
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Monday, 28. November 2022 at 04:24
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> > I guess I would sort of also vote for a separate Project. Would however
> only make sense if some people from both of our projects would join in.
>
> Agree, I think the iotdb developers in Timecho can join.
>
> > And name-wise … one of the names of Historic Historians (Ideally a Greek
> one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are
> “famous” Buttler names ;-)
>
> If we consider famous names (well, there will be culture bias), I'd
> like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
> "Records of the Grand Historian" is tooo famous (at least in China).
>
> BTW. naming "Apache Historian" directly may be another option.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>
> Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
> >
> > How about Apache Ephorous? ;-)
> >
> > https://en.wikipedia.org/wiki/Ephorus
> > Ephorus of Cyme (/ˈɛfərəs/<
> https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<
> https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros
> ho Kymaios; c. 400 – 330 BC) was an ancient Greek<
> https://en.wikipedia.org/wiki/Ancient_Greece> historian<
> https://en.wikipedia.org/wiki/Historian> known for his universal history<
> https://en.wikipedia.org/wiki/Universal_history>.
> >
> > Chris
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > Date: Sunday, 27. November 2022 at 15:28
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > I guess I would sort of also vote for a separate Project. Would however
> only make sense if some people from both of our projects would join in.
> >
> > And name-wise … one of the names of Historic Historians (Ideally a Greek
> one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are
> “famous” Buttler names ;-)
> > https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> > https://en.wikipedia.org/wiki/List_of_historians
> >
> > I think anything but a turnkey-ready solution will not be accepted by
> anyone in the Automation industry.
> >
> > Chris
> >
> >
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Sunday, 27. November 2022 at 14:05
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > If we just provide a solution, or demonstration, then either is ok.
> > If we want to provide an "one-box thing" (even without GUI), +1 for a
> > new project.
> >
> > > I don’t even think it would be bad if an entity like Timecho would add
> enterprise offerings, because I know, that the industry won’t “buy”
> something, if there’s no commercial support or anyone, they can throw money
> at, even if it’s free (That might even make them more skeptical).
> > Yes, indeed. :D
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> >
> > Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> > >
> > > Hi,
> > >
> > > I think it's a great idea.
> > >
> > > I'd vote for having a separate project for something like this, just so
> > > it's clear that it is a historian and people can build it separately. I
> > > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> > >
> > > Cake does go with Cafe.
> > >
> > > Ben
> > >
> > >
> > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <
> christofer.dutz@c-ware.de>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Trying to respond to all (except the naming question ;-) ):
> > > >
> > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we
> could be
> > > > collecting data on one node, writing that to TSFiles using the
> IoTDB-Client
> > > > (As Xaingdong mentioned).
> > > > These could either be online and transfer data to the server right
> away,
> > > > or operate in an offline-like mode and periodically fetch data in
> bursts
> > > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > > >
> > > > @Xaiangdong: In general, PLC4X operates in an active mode, but we
> also
> > > > have a proposed and prototypically proven “passive-mode”. Here the
> software
> > > > doesn’t actively participate in the data acquisition directly, but
> simply
> > > > listens to the data stream and makes data available. We also have
> something
> > > > we call a “Data-Diode”, which technically makes interference
> impossible as
> > > > it lets ethernet packets flow in one direction, but nothing in the
> other
> > > > (Firewalls usually have the problem of letting packets pass in both
> > > > directions for established connections). We invented this, because
> we knew
> > > > we will never have validated and audited open-source software, to a
> level
> > > > that it would be certified for some of these use cases.
> > > >
> > > > So, either we can live with everything the SCADA system is already
> > > > requesting, or we add an active PLC4X node in the secure network,
> that
> > > > requests data, and simply ignores it, and a second – passive mode –
> node
> > > > sits outside the secure network to capture the information.
> > > >
> > > > I agree that this sort of thing needs to be a “product”. The
> Automation
> > > > Industry just doesn’t know how to work with frameworks. Ideally a
> one-box
> > > > thing.
> > > > Right now, most tools I have seen don’t even need the calculation or
> the
> > > > visualization. This is usually done on another level. Important
> would be
> > > > that we could be somewhat API compatible with existing products. Them
> > > > usually having SQL or REST APIs, should make it relatively easy to
> sort of
> > > > produce frontends to our Apache Historian, that are API compatible
> with
> > > > some existing industry products, so they are replaceable.
> > > >
> > > > If we wanted to add a visualization layer or a data-curation layer
> later
> > > > on, we should have a chat with the NiFi or StreamPipes folks as
> that’s what
> > > > they already have.
> > > >
> > > > For now, I would be proposing to build something that uses PLC4X for
> the
> > > > data-acquisition, IoTDB for the storage and build a REST frontend
> for this,
> > > > that’s somewhat API compatible with one of the major established
> products
> > > > and to add more on a step-by-step basis.
> > > >
> > > > I don’t even think it would be bad if an entity like Timecho would
> add
> > > > enterprise offerings, because I know, that the industry won’t “buy”
> > > > something, if there’s no commercial support or anyone, they can
> throw money
> > > > at, even if it’s free (That might even make them more skeptical).
> > > >
> > > >
> > > > What do you all think? Does this make sense? If yes, where would it
> make
> > > > sense to start working on something like this? In the IoTDB project?
> In the
> > > > PLC4X project, in a separate (new) project?
> > > >
> > > >
> > > > Chris
> > > >
> > > >
> > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > Date: Sunday, 27. November 2022 at 06:02
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > Hi all,
> > > >
> > > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > > Actually, there are several users using IoTDB as their historian
> > > > solution...
> > > >
> > > > According to my knowledge, a historian software contains several
> features:
> > > >
> > > > - get data from OPC/modubs server and write to IoTDB
> > > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > > >
> > > > - (optional) there is a single-way network gateway for security.
> > > >   * If the gateway is deployed between device and PLC4x program, then
> > > > we need to check whether plc4x supports that.
> > > >   * If the gateway is deployed between IoTDB-client and IoTDB, then
> we
> > > > need to check wheter IoTDB-client supports that.
> > > >
> > > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > > collect A and B from a device (for example, C=A+B/2 )
> > > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > > programming GUI. (maybe integrating some open source projects, like
> > > > Red-node is a good idea)
> > > >
> > > > - get the latest data from IoTDB for visualization (usually users
> need
> > > > to draw a process flow diagram, and put the latest data on each
> device
> > > > icon on the diagram)
> > > >   * IoTDB supports getting latest data. but we need a new software
> for
> > > > drawing the diagram (I do not know if there is any open source
> project
> > > > for this. In our real applications, we ususally buy and integrate
> some
> > > > other commerical software)
> > > >
> > > > - send alert message if the real time data meets some rules
> > > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > > (IoTDB also supports Prometheus Alert Manager)
> > > >
> > > > - get the historical data
> > > >   * IoTDB supports that and grafana is good at visualization
> > > >
> > > > - last but not the least, if we provide all the features to
> industrial
> > > > users,  all the GUIs should be integrated into one entrance.
> > > >
> > > >
> > > > Best,
> > > > -----------------------------------
> > > > Xiangdong Huang
> > > > School of Software, Tsinghua University
> > > >
> > > >  黄向东
> > > > 清华大学 软件学院
> > > >
> > > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > > >
> > > > > Hi folks, long time lurker…
> > > > >
> > > > > A historian would be great. Does PLC4X queue data on the machine
> it’s
> > > > > running on already?
> > > > >
> > > > > A common pattern in historians is to queue data on the runtime
> server
> > > > until
> > > > > data can be extracted to a sql server, preventing data loss, and
> reducing
> > > > > the need for redundancy.
> > > > >
> > > > > There are a ton of edge cases to watch out for which I can
> elaborate on
> > > > > further.
> > > > >
> > > > > I’m a former Industrial Controls Engineer turned software dev and
> can
> > > > help
> > > > > out where needed.
> > > > >
> > > > > -Ryan
> > > > >
> > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <
> ottobackwards@gmail.com>
> > > > > wrote:
> > > > >
> > > > > >  Would you name it “Cake”?
> > > > > >
> > > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > > <ch...@c-ware.de>
> > > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > > dev@plc4x.apache.org>
> > > > > > Date: November 26, 2022 at 10:36:23
> > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> dev@plc4x.apache.org
> > > > >,
> > > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <
> dev@iotdb.apache.org>
> > > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > > something
> > > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > > >
> > > > > > For those of you, who don’t know what a Historian is. In the
> > > > manufacturing
> > > > > > industry they have these extremely expensive servers (usually
> you buy
> > > > them
> > > > > > as a bundle of hard- and software).
> > > > > > These servers are nothing else than a really crappy and brutally
> > > > expensive
> > > > > > Database for storing Time-Series data.
> > > > > >
> > > > > > The thing is most commercial products are currently really having
> > > > trouble
> > > > > > to keep up with the increasing amount of Data being sent.
> > > > > >
> > > > > > So instead of suggesting building an Historian at each customer’s
> > > > site, I
> > > > > > thought: Perhaps a ready-to-use solution based on open-source
> would be
> > > > a
> > > > > > good idea.
> > > > > > Not 100% sure where I’d locate such an initiative, but I would
> tend to
> > > > see
> > > > > > it more on the Database side.
> > > > > >
> > > > > > I think IoTDB would be the perfect storage system, all we seem
> to need
> > > > is
> > > > > > some sort of rest-interface that matches the industry standards
> for
> > > > > > querying the information and on the other side something like
> PLC4X to
> > > > fill
> > > > > > the database.
> > > > > >
> > > > > > Here some examples:
> > > > > >
> > > > > >
> > > >
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > > >
> > > >
> https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > > >
> > > > > > What do you folks think?
> > > > > >
> > > > > >
> > > > > > Chris
> > > > > >
> > > >
>

Re: AW: Building a PLC4X and IoTDB Historian?

[Ben Hutcheson <be...@gmail.com>]

I can commit to occasionally contributing.

On Tue, 29 Nov 2022 at 1:03 am, Christofer Dutz <ch...@c-ware.de>
wrote:

> So, I’ve upgraded my GitHub private account to a Pro account so we can
> also use the Wiki there and created a repository.
>
> Please send me the list of email addresses, that I should invite.
>
>
> Chris
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Tuesday, 29. November 2022 at 06:46
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: AW: Building a PLC4X and IoTDB Historian?
> > If we get something to work in private we could easily go TLP or through
> the Incubator whatever is decided then to do.
> +1
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> Niklas Merz <ni...@apache.org> 于2022年11月28日周一 21:08写道：
> >
> > Hello all,
> >
> > I think this is a good compromise to get started and then publish it
> > with a bit of fanfare (marketing) as an Apache project to the world once
> > it's ready.
> >
> > I am also interested in working with you on this project. I think PLC4X
> > and IoTDB need something like this to become popular in the automation
> > world.
> >
> > Regards
> > Niklas
> >
> > On November 28, 2022, Julian Feinauer
> > <j....@pragmaticindustries.de> wrote:
> > > I fully agree with that.
> > > If we get something to work in private we could easily go TLP or
> > > through the Incubator whatever is decided then to do.
> > >
> > > Julian
> > >
> > > PS.: Do you setup the repo and invite myself?
> > >
> > > Von: Christofer Dutz <ch...@c-ware.de>
> > > Datum: Montag, 28. November 2022 um 13:47
> > > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > > And if we already agree on a name (I think “Apache Historian” would be
> > > a good name),
> > > we could even already work on code with that package-name and maven
> > > coordinates, so we don’t have to change much when coming to Apache?
> > >
> > > Chris
> > >
> > > From: Christofer Dutz <ch...@c-ware.de>
> > > Date: Monday, 28. November 2022 at 13:18
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi Julian,
> > >
> > > Indeed, your concerns are valid regarding creating an already
> > > abandoned TLP.
> > >
> > > In that case, I would opt for a private repo on GitHub which we invite
> > > everyone willing to help work on it and as soon as we have something
> > > workable, we make it public as part of a TLP?
> > >
> > > This avoids at least the fear of creating a new TLP that’s product
> > > doesn’t even see the light of day. And as soon as we have something we
> > > want to show the world, we join Apache?
> > >
> > > Would that be a compromise?
> > >
> > >
> > > Chris
> > >
> > > From: Julian Feinauer <j....@pragmaticindustries.de>
> > > Date: Monday, 28. November 2022 at 13:14
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: AW: Building a PLC4X and IoTDB Historian?
> > > Hey,
> > >
> > > as I said.. we could just do it as part of one of the TLPs like PLC4X
> > > or IoTDB easily or in a private repo as well, for sure.
> > > I just want to avoid to create a TLP which is soon abandoned, that’s
> > > my only concern.
> > > Regarding all your comments I totally agree and am with you.
> > >
> > > Julian
> > >
> > > Von: Christofer Dutz <ch...@c-ware.de>
> > > Datum: Montag, 28. November 2022 um 13:07
> > > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > > Hi Julian,
> > >
> > > well … I would be hesitant to try something like this without the
> > > legal shield of the ASF … from what I’ve learned in the early days,
> > > Siemens and Co. continuously threatened to sue me for my work on
> > > PLC4X.
> > > The only thing that helped me then, was to say: I am doing everything
> > > legally, so you’d have trouble succeeding with that and you also can’t
> > > sue me but would have to sue the ASF and good luck on that not
> > > backfiring badly on you publicly.
> > >
> > > I know that in the sector of drivers in the past there were several
> > > occasions where open-source developers were annoyed so much by
> > > nuisance law-suites, that they gave up.
> > >
> > > I also know Historians are silly expensive systems. I haven’t seen one
> > > under 100k€ and the annual fees for certified solutions tend to start
> > > at 100k€/year/production-line … so there’s a lot of money for them to
> > > fear losing. So yes: I’m too scared to try something like this
> > > publicly without a legal shield.
> > >
> > > We could however work on something like this in a protected repo at
> > > GitHub etc. and go to TLP as soon as we have something.
> > >
> > > Would that be an option?
> > >
> > > Chris
> > >
> > >
> > > From: Julian Feinauer <j....@pragmaticindustries.de>
> > > Date: Monday, 28. November 2022 at 12:57
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: AW: Building a PLC4X and IoTDB Historian?
> > > Hey Chris,
> > >
> > > wrt testing I meant how „sustainable” the community is, that we intend
> > > to building.
> > > You know the effect pretty well if you ask “How wants to start this”
> > > everybody is like “me, me, me” and after 2 weeks you stand there
> > > alone.
> > > And one important aspect of a TLP for me is that the community is also
> > > “resilient” and will last “long” (whatever that means).
> > > And this is something I think is hard to guarantee in the current
> > > state.
> > >
> > > Technically I have no worries, this is something we would get to work
> > > rather easily, I see no “magics” hidden there.
> > >
> > > Julian
> > >
> > > Von: Christofer Dutz <ch...@c-ware.de>
> > > Datum: Montag, 28. November 2022 um 12:48
> > > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > > Hi Julian,
> > >
> > > Well, the problem with starting at one Project as a subproject … We
> > > then need to make everyone involved or wanting to be involved in the
> > > other project to become committers in the other.
> > > I wouldn’t have problems with doing it under the hood of IoTDB,
> > > personally.
> > >
> > > A separate Project would have the benefit of giving the industry an
> > > impression that this is something built just for them instead of being
> > > a side-product of something that’s already there (I have given up
> > > expecting the Automation industry to use any form of common sense ;-)
> > > … guess that will be my definition of Industry 5.0 … “connected
> > > everything + common sense”)
> > >
> > > The incubator is intended for teaching the initial community how
> > > things work at Apache, I guess as we’d be starting with mainly people
> > > that have already brought projects from outside into the incubator and
> > > into TLPs, this teaching effort will not exist in this case. Even
> > > Justin mentioned that this would probably be a case for straight TLP.
> > >
> > > Regarding the testing … not sure I understand what you want to test?
> > > The idea of an open-source historian?
> > > I mean the need exists … I’ve seen it in multiple occasions and our
> > > friends from IoTDB have confirmed they have built or at least seen
> > > IoTDB serve as such a system.
> > > Not 100% sure what you want to try out.
> > >
> > > But I agree … we should probably start with an (online) workshop, as
> > > this time we’d probably be spread out a bit wider geographically, than
> > > with our PLC4X workshops in the past.
> > >
> > > Chris
> > >
> > >
> > >
> > > From: Julian Feinauer <j....@pragmaticindustries.de>
> > > Date: Monday, 28. November 2022 at 12:37
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: AW: Building a PLC4X and IoTDB Historian?
> > > Hey,
> > >
> > > I understand your point and also agree generally.
> > > But I personally dislike the straight to TLP route as this is
> > > something I’d like to try out first.
> > > And my point with the separate repo (or whatever) was just to make the
> > > community ready for the Incubator (where a codebase and community
> > > normally should exist).
> > >
> > > But the easiest way in any regard would be to start of as a subproject
> > > in an existing PMC. This would only require to get another repo and we
> > > could start (formally no need to even register it as a subproject).
> > > And the repo thing could be done by you alone I think as Chair.
> > >
> > > Next steps would then be more of brainstorming and workshoping and
> > > building something (whatever?).
> > > I think the timing is quite good before Christmas because if its
> > > intereting, I see myself contributing over the Holidays (better don’t
> > > show this mail to my wife…).
> > >
> > > Best
> > > Julian
> > >
> > > Von: Christofer Dutz <ch...@c-ware.de>
> > > Datum: Montag, 28. November 2022 um 09:27
> > > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> > > <ho...@ltsoft.de>
> > > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > > Hi Julian,
> > >
> > > happy you like the idea ;-)
> > >
> > > We would probably outclass the established ones in point of price ;-)
> > > Right now, I think we should probably set up something that bundles
> > > IoTDB and PLC4X and then the “code of the project” is the bubble-wrap,
> > > that brings the two together and then provides an API that is
> > > equivalent to existing Historian products.
> > > I think the only way we can score points here, is if switching to it
> > > is as easy as possible and that only works if it’s sort of a drop-in
> > > replacement.
> > >
> > > Currently also discussing with other board members and the IPMC chair
> > > … he at least would not see us in the incubator. If there are enough
> > > PMC members of other projects and Apache Members on board, we could
> > > take the straight to TLP route.
> > >
> > > And I would really like to keep it at Apache and not set up something
> > > outside. This for multiple reasons:
> > >
> > >  * Protection (The market for the established products is huge and the
> > > companies behind them are also huge with huge legal departments, I
> > > would fear the same as I did with PLC4X, when I started it … don’t
> > > want to be stuck in nuisance lawsuits, just filed for keeping us
> > > occupied)
> > >  * The satement: we want Open-Source to be accepted by the industry
> > > and for me “Apache” is the form of Open-Source that I admire most. If
> > > it’s built up out of Apache stuff, it should be an Apache project (If
> > > we establish this in a way, that the industry notices Apache as a
> > > “vendor”, this will benefit many other projects)
> > >
> > > Chris
> > >
> > >
> > >
> > > From: Julian Feinauer <j....@pragmaticindustries.de>
> > > Date: Monday, 28. November 2022 at 09:17
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> > > <ho...@ltsoft.de>
> > > Subject: AW: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > first of all.. sorry for coming in so late.
> > > I think this is a great idea and there really is a product market fit
> > > for such a solution.
> > > I think technology wise its not "that complicated" to build such a
> > > system, I think many of the details that are necessary for a good
> > > adoption of such a software is in the areas around like documentation,
> > > marketing, more marketing, even more and even bolder marketing and
> > > sales (how to sell an open source project??).
> > >
> > > But technology wise I really like the idea and I think with the
> > > technology one has today it is very easy to "outclass" these
> > > established systems e.g. in performance, efficiency and als user
> > > friendliness.
> > > The only thing which might be challenging for the future are things
> > > like plugins or an extension system to allow users to customize their
> > > installation.
> > >
> > > Regarding the "separate project" approach I am not 100% certain.
> > > Personally, I would consider starting EITHER as a subproject in an
> > > existing PMC (PLC4X?) or as a separate undertaking on GitHub or
> > > somewhere else and not directly go to the Incubator or something.
> > > Because I think we should really find out if there is enough developer
> > > interest to build and sustain such a system.
> > >
> > > But I'm totally in for such a system either way!
> > >
> > > Julian
> > >
> > > PS.: Also forwarding the email to Björn and Tim who work(ed) a lot
> > > with historians
> > >
> > > Von: Christofer Dutz <ch...@c-ware.de>
> > > Datum: Montag, 28. November 2022 um 08:56
> > > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > > Hi Xiangdong,
> > >
> > > that’s perfect that some of you folks would be on board. I should
> > > probably check, if there are enough PLC4X folks on board too … would
> > > be a shame if it was just me ;-)
> > >
> > > And regarding the name … yeah … you are absolutely right: Historian
> > > might really be the ideal choice … as it instantly explains what it is
> > > … A Historian from the Apache Software Foundation and probably also no
> > > bias towards any regional culture.
> > >
> > > So (Speaking to my fellow PLC4X folks … who would be on board with
> > > this?)
> > >
> > >
> > > Chris
> > >
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Monday, 28. November 2022 at 04:24
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > I guess I would sort of also vote for a separate Project. Would
> > > however only make sense if some people from both of our projects would
> > > join in.
> > >
> > > Agree, I think the iotdb developers in Timecho can join.
> > >
> > > > And name-wise … one of the names of Historic Historians (Ideally a
> > > Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> > > Jenkins are “famous” Buttler names ;-)
> > >
> > > If we consider famous names (well, there will be culture bias), I'd
> > > like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
> > > "Records of the Grand Historian" is tooo famous (at least in China).
> > >
> > > BTW. naming "Apache Historian" directly may be another option.
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >
> > > Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
> > > >
> > > > How about Apache Ephorous? ;-)
> > > >
> > > > https://en.wikipedia.org/wiki/Ephorus
> > > > Ephorus of Cyme
> > > (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>;
> > > Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος,
> > > Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient
> > > Greek<https://en.wikipedia.org/wiki/Ancient_Greece>
> > > historian<https://en.wikipedia.org/wiki/Historian> known for his
> > > universal history<https://en.wikipedia.org/wiki/Universal_history>.
> > > >
> > > > Chris
> > > >
> > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > Date: Sunday, 27. November 2022 at 15:28
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > Hi all,
> > > >
> > > > I guess I would sort of also vote for a separate Project. Would
> > > however only make sense if some people from both of our projects would
> > > join in.
> > > >
> > > > And name-wise … one of the names of Historic Historians (Ideally a
> > > Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> > > Jenkins are “famous” Buttler names ;-)
> > > > https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> > > > https://en.wikipedia.org/wiki/List_of_historians
> > > >
> > > > I think anything but a turnkey-ready solution will not be accepted
> > > by anyone in the Automation industry.
> > > >
> > > > Chris
> > > >
> > > >
> > > >
> > > >
> > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > Date: Sunday, 27. November 2022 at 14:05
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > If we just provide a solution, or demonstration, then either is ok.
> > > > If we want to provide an "one-box thing" (even without GUI), +1 for
> > > a
> > > > new project.
> > > >
> > > > > I don’t even think it would be bad if an entity like Timecho would
> > > add enterprise offerings, because I know, that the industry won’t
> > > “buy” something, if there’s no commercial support or anyone, they can
> > > throw money at, even if it’s free (That might even make them more
> > > skeptical).
> > > > Yes, indeed. :D
> > > >
> > > > Best,
> > > > -----------------------------------
> > > > Xiangdong Huang
> > > >
> > > > Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> > > > >
> > > > > Hi,
> > > > >
> > > > > I think it's a great idea.
> > > > >
> > > > > I'd vote for having a separate project for something like this,
> > > just so
> > > > > it's clear that it is a historian and people can build it
> > > separately. I
> > > > > wouldn't want it hidden away in an obscure folder in the PLC4X
> > > repo.
> > > > >
> > > > > Cake does go with Cafe.
> > > > >
> > > > > Ben
> > > > >
> > > > >
> > > > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz
> > > <ch...@c-ware.de>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Trying to respond to all (except the naming question ;-) ):
> > > > > >
> > > > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we
> > > could be
> > > > > > collecting data on one node, writing that to TSFiles using the
> > > IoTDB-Client
> > > > > > (As Xaingdong mentioned).
> > > > > > These could either be online and transfer data to the server
> > > right away,
> > > > > > or operate in an offline-like mode and periodically fetch data
> > > in bursts
> > > > > > from the client (Hope that’s correct, please correct me if I’m
> > > wrong)
> > > > > >
> > > > > > @Xaiangdong: In general, PLC4X operates in an active mode, but
> > > we also
> > > > > > have a proposed and prototypically proven “passive-mode”. Here
> > > the software
> > > > > > doesn’t actively participate in the data acquisition directly,
> > > but simply
> > > > > > listens to the data stream and makes data available. We also
> > > have something
> > > > > > we call a “Data-Diode”, which technically makes interference
> > > impossible as
> > > > > > it lets ethernet packets flow in one direction, but nothing in
> > > the other
> > > > > > (Firewalls usually have the problem of letting packets pass in
> > > both
> > > > > > directions for established connections). We invented this,
> > > because we knew
> > > > > > we will never have validated and audited open-source software,
> > > to a level
> > > > > > that it would be certified for some of these use cases.
> > > > > >
> > > > > > So, either we can live with everything the SCADA system is
> > > already
> > > > > > requesting, or we add an active PLC4X node in the secure
> > > network, that
> > > > > > requests data, and simply ignores it, and a second – passive
> > > mode – node
> > > > > > sits outside the secure network to capture the information.
> > > > > >
> > > > > > I agree that this sort of thing needs to be a “product”. The
> > > Automation
> > > > > > Industry just doesn’t know how to work with frameworks. Ideally
> > > a one-box
> > > > > > thing.
> > > > > > Right now, most tools I have seen don’t even need the
> > > calculation or the
> > > > > > visualization. This is usually done on another level. Important
> > > would be
> > > > > > that we could be somewhat API compatible with existing products.
> > > Them
> > > > > > usually having SQL or REST APIs, should make it relatively easy
> > > to sort of
> > > > > > produce frontends to our Apache Historian, that are API
> > > compatible with
> > > > > > some existing industry products, so they are replaceable.
> > > > > >
> > > > > > If we wanted to add a visualization layer or a data-curation
> > > layer later
> > > > > > on, we should have a chat with the NiFi or StreamPipes folks as
> > > that’s what
> > > > > > they already have.
> > > > > >
> > > > > > For now, I would be proposing to build something that uses PLC4X
> > > for the
> > > > > > data-acquisition, IoTDB for the storage and build a REST
> > > frontend for this,
> > > > > > that’s somewhat API compatible with one of the major established
> > > products
> > > > > > and to add more on a step-by-step basis.
> > > > > >
> > > > > > I don’t even think it would be bad if an entity like Timecho
> > > would add
> > > > > > enterprise offerings, because I know, that the industry won’t
> > > “buy”
> > > > > > something, if there’s no commercial support or anyone, they can
> > > throw money
> > > > > > at, even if it’s free (That might even make them more
> > > skeptical).
> > > > > >
> > > > > >
> > > > > > What do you all think? Does this make sense? If yes, where would
> > > it make
> > > > > > sense to start working on something like this? In the IoTDB
> > > project? In the
> > > > > > PLC4X project, in a separate (new) project?
> > > > > >
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > > > Date: Sunday, 27. November 2022 at 06:02
> > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > > > Hi all,
> > > > > >
> > > > > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > > > > Actually, there are several users using IoTDB as their historian
> > > > > > solution...
> > > > > >
> > > > > > According to my knowledge, a historian software contains several
> > > features:
> > > > > >
> > > > > > - get data from OPC/modubs server and write to IoTDB
> > > > > > * use PLC4x + IoTDB-client. (we call this data collector)
> > > > > >
> > > > > > - (optional) there is a single-way network gateway for security.
> > > > > > * If the gateway is deployed between device and PLC4x program,
> > > then
> > > > > > we need to check whether plc4x supports that.
> > > > > > * If the gateway is deployed between IoTDB-client and IoTDB,
> > > then we
> > > > > > need to check wheter IoTDB-client supports that.
> > > > > >
> > > > > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > > > > collect A and B from a device (for example, C=A+B/2 )
> > > > > > * IoTDB's trigger supports that but we need a Drag-and-drop
> > > > > > programming GUI. (maybe integrating some open source projects,
> > > like
> > > > > > Red-node is a good idea)
> > > > > >
> > > > > > - get the latest data from IoTDB for visualization (usually
> > > users need
> > > > > > to draw a process flow diagram, and put the latest data on each
> > > device
> > > > > > icon on the diagram)
> > > > > > * IoTDB supports getting latest data. but we need a new software
> > > for
> > > > > > drawing the diagram (I do not know if there is any open source
> > > project
> > > > > > for this. In our real applications, we ususally buy and
> > > integrate some
> > > > > > other commerical software)
> > > > > >
> > > > > > - send alert message if the real time data meets some rules
> > > > > > * IoTDB trigger supports the rule. but currently we have no a
> > > GUI.
> > > > > > (IoTDB also supports Prometheus Alert Manager)
> > > > > >
> > > > > > - get the historical data
> > > > > > * IoTDB supports that and grafana is good at visualization
> > > > > >
> > > > > > - last but not the least, if we provide all the features to
> > > industrial
> > > > > > users, all the GUIs should be integrated into one entrance.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > -----------------------------------
> > > > > > Xiangdong Huang
> > > > > > School of Software, Tsinghua University
> > > > > >
> > > > > > 黄向东
> > > > > > 清华大学 软件学院
> > > > > >
> > > > > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > > > > >
> > > > > > > Hi folks, long time lurker…
> > > > > > >
> > > > > > > A historian would be great. Does PLC4X queue data on the
> > > machine it’s
> > > > > > > running on already?
> > > > > > >
> > > > > > > A common pattern in historians is to queue data on the runtime
> > > server
> > > > > > until
> > > > > > > data can be extracted to a sql server, preventing data loss,
> > > and reducing
> > > > > > > the need for redundancy.
> > > > > > >
> > > > > > > There are a ton of edge cases to watch out for which I can
> > > elaborate on
> > > > > > > further.
> > > > > > >
> > > > > > > I’m a former Industrial Controls Engineer turned software dev
> > > and can
> > > > > > help
> > > > > > > out where needed.
> > > > > > >
> > > > > > > -Ryan
> > > > > > >
> > > > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler
> > > <ot...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Would you name it “Cake”?
> > > > > > > >
> > > > > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > > > > <ch...@c-ware.de>
> > > > > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > > > > dev@plc4x.apache.org>
> > > > > > > > Date: November 26, 2022 at 10:36:23
> > > > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > <dev@plc4x.apache.org
> > > > > > >,
> > > > > > > > dev@iotdb.apache.org <de...@iotdb.apache.org>
> > > <de...@iotdb.apache.org>
> > > > > > > > Subject: Building a PLC4X and IoTDB Historian?
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I am currently thinking, if it wouldn’t be a good idea to
> > > build
> > > > > > something
> > > > > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > > > > >
> > > > > > > > For those of you, who don’t know what a Historian is. In the
> > > > > > manufacturing
> > > > > > > > industry they have these extremely expensive servers
> > > (usually you buy
> > > > > > them
> > > > > > > > as a bundle of hard- and software).
> > > > > > > > These servers are nothing else than a really crappy and
> > > brutally
> > > > > > expensive
> > > > > > > > Database for storing Time-Series data.
> > > > > > > >
> > > > > > > > The thing is most commercial products are currently really
> > > having
> > > > > > trouble
> > > > > > > > to keep up with the increasing amount of Data being sent.
> > > > > > > >
> > > > > > > > So instead of suggesting building an Historian at each
> > > customer’s
> > > > > > site, I
> > > > > > > > thought: Perhaps a ready-to-use solution based on open-
> > > source would be
> > > > > > a
> > > > > > > > good idea.
> > > > > > > > Not 100% sure where I’d locate such an initiative, but I
> > > would tend to
> > > > > > see
> > > > > > > > it more on the Database side.
> > > > > > > >
> > > > > > > > I think IoTDB would be the perfect storage system, all we
> > > seem to need
> > > > > > is
> > > > > > > > some sort of rest-interface that matches the industry
> > > standards for
> > > > > > > > querying the information and on the other side something
> > > like PLC4X to
> > > > > > fill
> > > > > > > > the database.
> > > > > > > >
> > > > > > > > Here some examples:
> > > > > > > >
> > > > > > > >
> > > > > >
> > >
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > > > > >
> > > > > > https://cdn.logic-
> > > control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > > > > >
> > > > > > > > What do you folks think?
> > > > > > > >
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > >
>

Re: AW: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

So, I’ve upgraded my GitHub private account to a Pro account so we can also use the Wiki there and created a repository.

Please send me the list of email addresses, that I should invite.


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Tuesday, 29. November 2022 at 06:46
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: AW: Building a PLC4X and IoTDB Historian?
> If we get something to work in private we could easily go TLP or through the Incubator whatever is decided then to do.
+1
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

Niklas Merz <ni...@apache.org> 于2022年11月28日周一 21:08写道：
>
> Hello all,
>
> I think this is a good compromise to get started and then publish it
> with a bit of fanfare (marketing) as an Apache project to the world once
> it's ready.
>
> I am also interested in working with you on this project. I think PLC4X
> and IoTDB need something like this to become popular in the automation
> world.
>
> Regards
> Niklas
>
> On November 28, 2022, Julian Feinauer
> <j....@pragmaticindustries.de> wrote:
> > I fully agree with that.
> > If we get something to work in private we could easily go TLP or
> > through the Incubator whatever is decided then to do.
> >
> > Julian
> >
> > PS.: Do you setup the repo and invite myself?
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 13:47
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > And if we already agree on a name (I think “Apache Historian” would be
> > a good name),
> > we could even already work on code with that package-name and maven
> > coordinates, so we don’t have to change much when coming to Apache?
> >
> > Chris
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > Date: Monday, 28. November 2022 at 13:18
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > Indeed, your concerns are valid regarding creating an already
> > abandoned TLP.
> >
> > In that case, I would opt for a private repo on GitHub which we invite
> > everyone willing to help work on it and as soon as we have something
> > workable, we make it public as part of a TLP?
> >
> > This avoids at least the fear of creating a new TLP that’s product
> > doesn’t even see the light of day. And as soon as we have something we
> > want to show the world, we join Apache?
> >
> > Would that be a compromise?
> >
> >
> > Chris
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 13:14
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hey,
> >
> > as I said.. we could just do it as part of one of the TLPs like PLC4X
> > or IoTDB easily or in a private repo as well, for sure.
> > I just want to avoid to create a TLP which is soon abandoned, that’s
> > my only concern.
> > Regarding all your comments I totally agree and am with you.
> >
> > Julian
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 13:07
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > well … I would be hesitant to try something like this without the
> > legal shield of the ASF … from what I’ve learned in the early days,
> > Siemens and Co. continuously threatened to sue me for my work on
> > PLC4X.
> > The only thing that helped me then, was to say: I am doing everything
> > legally, so you’d have trouble succeeding with that and you also can’t
> > sue me but would have to sue the ASF and good luck on that not
> > backfiring badly on you publicly.
> >
> > I know that in the sector of drivers in the past there were several
> > occasions where open-source developers were annoyed so much by
> > nuisance law-suites, that they gave up.
> >
> > I also know Historians are silly expensive systems. I haven’t seen one
> > under 100k€ and the annual fees for certified solutions tend to start
> > at 100k€/year/production-line … so there’s a lot of money for them to
> > fear losing. So yes: I’m too scared to try something like this
> > publicly without a legal shield.
> >
> > We could however work on something like this in a protected repo at
> > GitHub etc. and go to TLP as soon as we have something.
> >
> > Would that be an option?
> >
> > Chris
> >
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 12:57
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hey Chris,
> >
> > wrt testing I meant how „sustainable” the community is, that we intend
> > to building.
> > You know the effect pretty well if you ask “How wants to start this”
> > everybody is like “me, me, me” and after 2 weeks you stand there
> > alone.
> > And one important aspect of a TLP for me is that the community is also
> > “resilient” and will last “long” (whatever that means).
> > And this is something I think is hard to guarantee in the current
> > state.
> >
> > Technically I have no worries, this is something we would get to work
> > rather easily, I see no “magics” hidden there.
> >
> > Julian
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 12:48
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > Well, the problem with starting at one Project as a subproject … We
> > then need to make everyone involved or wanting to be involved in the
> > other project to become committers in the other.
> > I wouldn’t have problems with doing it under the hood of IoTDB,
> > personally.
> >
> > A separate Project would have the benefit of giving the industry an
> > impression that this is something built just for them instead of being
> > a side-product of something that’s already there (I have given up
> > expecting the Automation industry to use any form of common sense ;-)
> > … guess that will be my definition of Industry 5.0 … “connected
> > everything + common sense”)
> >
> > The incubator is intended for teaching the initial community how
> > things work at Apache, I guess as we’d be starting with mainly people
> > that have already brought projects from outside into the incubator and
> > into TLPs, this teaching effort will not exist in this case. Even
> > Justin mentioned that this would probably be a case for straight TLP.
> >
> > Regarding the testing … not sure I understand what you want to test?
> > The idea of an open-source historian?
> > I mean the need exists … I’ve seen it in multiple occasions and our
> > friends from IoTDB have confirmed they have built or at least seen
> > IoTDB serve as such a system.
> > Not 100% sure what you want to try out.
> >
> > But I agree … we should probably start with an (online) workshop, as
> > this time we’d probably be spread out a bit wider geographically, than
> > with our PLC4X workshops in the past.
> >
> > Chris
> >
> >
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 12:37
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hey,
> >
> > I understand your point and also agree generally.
> > But I personally dislike the straight to TLP route as this is
> > something I’d like to try out first.
> > And my point with the separate repo (or whatever) was just to make the
> > community ready for the Incubator (where a codebase and community
> > normally should exist).
> >
> > But the easiest way in any regard would be to start of as a subproject
> > in an existing PMC. This would only require to get another repo and we
> > could start (formally no need to even register it as a subproject).
> > And the repo thing could be done by you alone I think as Chair.
> >
> > Next steps would then be more of brainstorming and workshoping and
> > building something (whatever?).
> > I think the timing is quite good before Christmas because if its
> > intereting, I see myself contributing over the Holidays (better don’t
> > show this mail to my wife…).
> >
> > Best
> > Julian
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 09:27
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> > <ho...@ltsoft.de>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > happy you like the idea ;-)
> >
> > We would probably outclass the established ones in point of price ;-)
> > Right now, I think we should probably set up something that bundles
> > IoTDB and PLC4X and then the “code of the project” is the bubble-wrap,
> > that brings the two together and then provides an API that is
> > equivalent to existing Historian products.
> > I think the only way we can score points here, is if switching to it
> > is as easy as possible and that only works if it’s sort of a drop-in
> > replacement.
> >
> > Currently also discussing with other board members and the IPMC chair
> > … he at least would not see us in the incubator. If there are enough
> > PMC members of other projects and Apache Members on board, we could
> > take the straight to TLP route.
> >
> > And I would really like to keep it at Apache and not set up something
> > outside. This for multiple reasons:
> >
> >  * Protection (The market for the established products is huge and the
> > companies behind them are also huge with huge legal departments, I
> > would fear the same as I did with PLC4X, when I started it … don’t
> > want to be stuck in nuisance lawsuits, just filed for keeping us
> > occupied)
> >  * The satement: we want Open-Source to be accepted by the industry
> > and for me “Apache” is the form of Open-Source that I admire most. If
> > it’s built up out of Apache stuff, it should be an Apache project (If
> > we establish this in a way, that the industry notices Apache as a
> > “vendor”, this will benefit many other projects)
> >
> > Chris
> >
> >
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 09:17
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> > <ho...@ltsoft.de>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > first of all.. sorry for coming in so late.
> > I think this is a great idea and there really is a product market fit
> > for such a solution.
> > I think technology wise its not "that complicated" to build such a
> > system, I think many of the details that are necessary for a good
> > adoption of such a software is in the areas around like documentation,
> > marketing, more marketing, even more and even bolder marketing and
> > sales (how to sell an open source project??).
> >
> > But technology wise I really like the idea and I think with the
> > technology one has today it is very easy to "outclass" these
> > established systems e.g. in performance, efficiency and als user
> > friendliness.
> > The only thing which might be challenging for the future are things
> > like plugins or an extension system to allow users to customize their
> > installation.
> >
> > Regarding the "separate project" approach I am not 100% certain.
> > Personally, I would consider starting EITHER as a subproject in an
> > existing PMC (PLC4X?) or as a separate undertaking on GitHub or
> > somewhere else and not directly go to the Incubator or something.
> > Because I think we should really find out if there is enough developer
> > interest to build and sustain such a system.
> >
> > But I'm totally in for such a system either way!
> >
> > Julian
> >
> > PS.: Also forwarding the email to Björn and Tim who work(ed) a lot
> > with historians
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 08:56
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Xiangdong,
> >
> > that’s perfect that some of you folks would be on board. I should
> > probably check, if there are enough PLC4X folks on board too … would
> > be a shame if it was just me ;-)
> >
> > And regarding the name … yeah … you are absolutely right: Historian
> > might really be the ideal choice … as it instantly explains what it is
> > … A Historian from the Apache Software Foundation and probably also no
> > bias towards any regional culture.
> >
> > So (Speaking to my fellow PLC4X folks … who would be on board with
> > this?)
> >
> >
> > Chris
> >
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Monday, 28. November 2022 at 04:24
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > I guess I would sort of also vote for a separate Project. Would
> > however only make sense if some people from both of our projects would
> > join in.
> >
> > Agree, I think the iotdb developers in Timecho can join.
> >
> > > And name-wise … one of the names of Historic Historians (Ideally a
> > Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> > Jenkins are “famous” Buttler names ;-)
> >
> > If we consider famous names (well, there will be culture bias), I'd
> > like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
> > "Records of the Grand Historian" is tooo famous (at least in China).
> >
> > BTW. naming "Apache Historian" directly may be another option.
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >
> > Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
> > >
> > > How about Apache Ephorous? ;-)
> > >
> > > https://en.wikipedia.org/wiki/Ephorus
> > > Ephorus of Cyme
> > (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>;
> > Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος,
> > Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient
> > Greek<https://en.wikipedia.org/wiki/Ancient_Greece>
> > historian<https://en.wikipedia.org/wiki/Historian> known for his
> > universal history<https://en.wikipedia.org/wiki/Universal_history>.
> > >
> > > Chris
> > >
> > > From: Christofer Dutz <ch...@c-ware.de>
> > > Date: Sunday, 27. November 2022 at 15:28
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > I guess I would sort of also vote for a separate Project. Would
> > however only make sense if some people from both of our projects would
> > join in.
> > >
> > > And name-wise … one of the names of Historic Historians (Ideally a
> > Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> > Jenkins are “famous” Buttler names ;-)
> > > https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> > > https://en.wikipedia.org/wiki/List_of_historians
> > >
> > > I think anything but a turnkey-ready solution will not be accepted
> > by anyone in the Automation industry.
> > >
> > > Chris
> > >
> > >
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 14:05
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > If we just provide a solution, or demonstration, then either is ok.
> > > If we want to provide an "one-box thing" (even without GUI), +1 for
> > a
> > > new project.
> > >
> > > > I don’t even think it would be bad if an entity like Timecho would
> > add enterprise offerings, because I know, that the industry won’t
> > “buy” something, if there’s no commercial support or anyone, they can
> > throw money at, even if it’s free (That might even make them more
> > skeptical).
> > > Yes, indeed. :D
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > >
> > > Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> > > >
> > > > Hi,
> > > >
> > > > I think it's a great idea.
> > > >
> > > > I'd vote for having a separate project for something like this,
> > just so
> > > > it's clear that it is a historian and people can build it
> > separately. I
> > > > wouldn't want it hidden away in an obscure folder in the PLC4X
> > repo.
> > > >
> > > > Cake does go with Cafe.
> > > >
> > > > Ben
> > > >
> > > >
> > > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz
> > <ch...@c-ware.de>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Trying to respond to all (except the naming question ;-) ):
> > > > >
> > > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we
> > could be
> > > > > collecting data on one node, writing that to TSFiles using the
> > IoTDB-Client
> > > > > (As Xaingdong mentioned).
> > > > > These could either be online and transfer data to the server
> > right away,
> > > > > or operate in an offline-like mode and periodically fetch data
> > in bursts
> > > > > from the client (Hope that’s correct, please correct me if I’m
> > wrong)
> > > > >
> > > > > @Xaiangdong: In general, PLC4X operates in an active mode, but
> > we also
> > > > > have a proposed and prototypically proven “passive-mode”. Here
> > the software
> > > > > doesn’t actively participate in the data acquisition directly,
> > but simply
> > > > > listens to the data stream and makes data available. We also
> > have something
> > > > > we call a “Data-Diode”, which technically makes interference
> > impossible as
> > > > > it lets ethernet packets flow in one direction, but nothing in
> > the other
> > > > > (Firewalls usually have the problem of letting packets pass in
> > both
> > > > > directions for established connections). We invented this,
> > because we knew
> > > > > we will never have validated and audited open-source software,
> > to a level
> > > > > that it would be certified for some of these use cases.
> > > > >
> > > > > So, either we can live with everything the SCADA system is
> > already
> > > > > requesting, or we add an active PLC4X node in the secure
> > network, that
> > > > > requests data, and simply ignores it, and a second – passive
> > mode – node
> > > > > sits outside the secure network to capture the information.
> > > > >
> > > > > I agree that this sort of thing needs to be a “product”. The
> > Automation
> > > > > Industry just doesn’t know how to work with frameworks. Ideally
> > a one-box
> > > > > thing.
> > > > > Right now, most tools I have seen don’t even need the
> > calculation or the
> > > > > visualization. This is usually done on another level. Important
> > would be
> > > > > that we could be somewhat API compatible with existing products.
> > Them
> > > > > usually having SQL or REST APIs, should make it relatively easy
> > to sort of
> > > > > produce frontends to our Apache Historian, that are API
> > compatible with
> > > > > some existing industry products, so they are replaceable.
> > > > >
> > > > > If we wanted to add a visualization layer or a data-curation
> > layer later
> > > > > on, we should have a chat with the NiFi or StreamPipes folks as
> > that’s what
> > > > > they already have.
> > > > >
> > > > > For now, I would be proposing to build something that uses PLC4X
> > for the
> > > > > data-acquisition, IoTDB for the storage and build a REST
> > frontend for this,
> > > > > that’s somewhat API compatible with one of the major established
> > products
> > > > > and to add more on a step-by-step basis.
> > > > >
> > > > > I don’t even think it would be bad if an entity like Timecho
> > would add
> > > > > enterprise offerings, because I know, that the industry won’t
> > “buy”
> > > > > something, if there’s no commercial support or anyone, they can
> > throw money
> > > > > at, even if it’s free (That might even make them more
> > skeptical).
> > > > >
> > > > >
> > > > > What do you all think? Does this make sense? If yes, where would
> > it make
> > > > > sense to start working on something like this? In the IoTDB
> > project? In the
> > > > > PLC4X project, in a separate (new) project?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > > > >
> > > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > > Date: Sunday, 27. November 2022 at 06:02
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > > Hi all,
> > > > >
> > > > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > > > Actually, there are several users using IoTDB as their historian
> > > > > solution...
> > > > >
> > > > > According to my knowledge, a historian software contains several
> > features:
> > > > >
> > > > > - get data from OPC/modubs server and write to IoTDB
> > > > > * use PLC4x + IoTDB-client. (we call this data collector)
> > > > >
> > > > > - (optional) there is a single-way network gateway for security.
> > > > > * If the gateway is deployed between device and PLC4x program,
> > then
> > > > > we need to check whether plc4x supports that.
> > > > > * If the gateway is deployed between IoTDB-client and IoTDB,
> > then we
> > > > > need to check wheter IoTDB-client supports that.
> > > > >
> > > > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > > > collect A and B from a device (for example, C=A+B/2 )
> > > > > * IoTDB's trigger supports that but we need a Drag-and-drop
> > > > > programming GUI. (maybe integrating some open source projects,
> > like
> > > > > Red-node is a good idea)
> > > > >
> > > > > - get the latest data from IoTDB for visualization (usually
> > users need
> > > > > to draw a process flow diagram, and put the latest data on each
> > device
> > > > > icon on the diagram)
> > > > > * IoTDB supports getting latest data. but we need a new software
> > for
> > > > > drawing the diagram (I do not know if there is any open source
> > project
> > > > > for this. In our real applications, we ususally buy and
> > integrate some
> > > > > other commerical software)
> > > > >
> > > > > - send alert message if the real time data meets some rules
> > > > > * IoTDB trigger supports the rule. but currently we have no a
> > GUI.
> > > > > (IoTDB also supports Prometheus Alert Manager)
> > > > >
> > > > > - get the historical data
> > > > > * IoTDB supports that and grafana is good at visualization
> > > > >
> > > > > - last but not the least, if we provide all the features to
> > industrial
> > > > > users, all the GUIs should be integrated into one entrance.
> > > > >
> > > > >
> > > > > Best,
> > > > > -----------------------------------
> > > > > Xiangdong Huang
> > > > > School of Software, Tsinghua University
> > > > >
> > > > > 黄向东
> > > > > 清华大学 软件学院
> > > > >
> > > > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > > > >
> > > > > > Hi folks, long time lurker…
> > > > > >
> > > > > > A historian would be great. Does PLC4X queue data on the
> > machine it’s
> > > > > > running on already?
> > > > > >
> > > > > > A common pattern in historians is to queue data on the runtime
> > server
> > > > > until
> > > > > > data can be extracted to a sql server, preventing data loss,
> > and reducing
> > > > > > the need for redundancy.
> > > > > >
> > > > > > There are a ton of edge cases to watch out for which I can
> > elaborate on
> > > > > > further.
> > > > > >
> > > > > > I’m a former Industrial Controls Engineer turned software dev
> > and can
> > > > > help
> > > > > > out where needed.
> > > > > >
> > > > > > -Ryan
> > > > > >
> > > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler
> > <ot...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Would you name it “Cake”?
> > > > > > >
> > > > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > > > <ch...@c-ware.de>
> > > > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > > > dev@plc4x.apache.org>
> > > > > > > Date: November 26, 2022 at 10:36:23
> > > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > <dev@plc4x.apache.org
> > > > > >,
> > > > > > > dev@iotdb.apache.org <de...@iotdb.apache.org>
> > <de...@iotdb.apache.org>
> > > > > > > Subject: Building a PLC4X and IoTDB Historian?
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I am currently thinking, if it wouldn’t be a good idea to
> > build
> > > > > something
> > > > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > > > >
> > > > > > > For those of you, who don’t know what a Historian is. In the
> > > > > manufacturing
> > > > > > > industry they have these extremely expensive servers
> > (usually you buy
> > > > > them
> > > > > > > as a bundle of hard- and software).
> > > > > > > These servers are nothing else than a really crappy and
> > brutally
> > > > > expensive
> > > > > > > Database for storing Time-Series data.
> > > > > > >
> > > > > > > The thing is most commercial products are currently really
> > having
> > > > > trouble
> > > > > > > to keep up with the increasing amount of Data being sent.
> > > > > > >
> > > > > > > So instead of suggesting building an Historian at each
> > customer’s
> > > > > site, I
> > > > > > > thought: Perhaps a ready-to-use solution based on open-
> > source would be
> > > > > a
> > > > > > > good idea.
> > > > > > > Not 100% sure where I’d locate such an initiative, but I
> > would tend to
> > > > > see
> > > > > > > it more on the Database side.
> > > > > > >
> > > > > > > I think IoTDB would be the perfect storage system, all we
> > seem to need
> > > > > is
> > > > > > > some sort of rest-interface that matches the industry
> > standards for
> > > > > > > querying the information and on the other side something
> > like PLC4X to
> > > > > fill
> > > > > > > the database.
> > > > > > >
> > > > > > > Here some examples:
> > > > > > >
> > > > > > >
> > > > >
> > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > > > >
> > > > > https://cdn.logic-
> > control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > > > >
> > > > > > > What do you folks think?
> > > > > > >
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > >

Re: AW: Building a PLC4X and IoTDB Historian?

[Xiangdong Huang <sa...@gmail.com>]

> If we get something to work in private we could easily go TLP or through the Incubator whatever is decided then to do.
+1
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

Niklas Merz <ni...@apache.org> 于2022年11月28日周一 21:08写道：
>
> Hello all,
>
> I think this is a good compromise to get started and then publish it
> with a bit of fanfare (marketing) as an Apache project to the world once
> it's ready.
>
> I am also interested in working with you on this project. I think PLC4X
> and IoTDB need something like this to become popular in the automation
> world.
>
> Regards
> Niklas
>
> On November 28, 2022, Julian Feinauer
> <j....@pragmaticindustries.de> wrote:
> > I fully agree with that.
> > If we get something to work in private we could easily go TLP or
> > through the Incubator whatever is decided then to do.
> >
> > Julian
> >
> > PS.: Do you setup the repo and invite myself?
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 13:47
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > And if we already agree on a name (I think “Apache Historian” would be
> > a good name),
> > we could even already work on code with that package-name and maven
> > coordinates, so we don’t have to change much when coming to Apache?
> >
> > Chris
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > Date: Monday, 28. November 2022 at 13:18
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > Indeed, your concerns are valid regarding creating an already
> > abandoned TLP.
> >
> > In that case, I would opt for a private repo on GitHub which we invite
> > everyone willing to help work on it and as soon as we have something
> > workable, we make it public as part of a TLP?
> >
> > This avoids at least the fear of creating a new TLP that’s product
> > doesn’t even see the light of day. And as soon as we have something we
> > want to show the world, we join Apache?
> >
> > Would that be a compromise?
> >
> >
> > Chris
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 13:14
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hey,
> >
> > as I said.. we could just do it as part of one of the TLPs like PLC4X
> > or IoTDB easily or in a private repo as well, for sure.
> > I just want to avoid to create a TLP which is soon abandoned, that’s
> > my only concern.
> > Regarding all your comments I totally agree and am with you.
> >
> > Julian
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 13:07
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > well … I would be hesitant to try something like this without the
> > legal shield of the ASF … from what I’ve learned in the early days,
> > Siemens and Co. continuously threatened to sue me for my work on
> > PLC4X.
> > The only thing that helped me then, was to say: I am doing everything
> > legally, so you’d have trouble succeeding with that and you also can’t
> > sue me but would have to sue the ASF and good luck on that not
> > backfiring badly on you publicly.
> >
> > I know that in the sector of drivers in the past there were several
> > occasions where open-source developers were annoyed so much by
> > nuisance law-suites, that they gave up.
> >
> > I also know Historians are silly expensive systems. I haven’t seen one
> > under 100k€ and the annual fees for certified solutions tend to start
> > at 100k€/year/production-line … so there’s a lot of money for them to
> > fear losing. So yes: I’m too scared to try something like this
> > publicly without a legal shield.
> >
> > We could however work on something like this in a protected repo at
> > GitHub etc. and go to TLP as soon as we have something.
> >
> > Would that be an option?
> >
> > Chris
> >
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 12:57
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hey Chris,
> >
> > wrt testing I meant how „sustainable” the community is, that we intend
> > to building.
> > You know the effect pretty well if you ask “How wants to start this”
> > everybody is like “me, me, me” and after 2 weeks you stand there
> > alone.
> > And one important aspect of a TLP for me is that the community is also
> > “resilient” and will last “long” (whatever that means).
> > And this is something I think is hard to guarantee in the current
> > state.
> >
> > Technically I have no worries, this is something we would get to work
> > rather easily, I see no “magics” hidden there.
> >
> > Julian
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 12:48
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > Well, the problem with starting at one Project as a subproject … We
> > then need to make everyone involved or wanting to be involved in the
> > other project to become committers in the other.
> > I wouldn’t have problems with doing it under the hood of IoTDB,
> > personally.
> >
> > A separate Project would have the benefit of giving the industry an
> > impression that this is something built just for them instead of being
> > a side-product of something that’s already there (I have given up
> > expecting the Automation industry to use any form of common sense ;-)
> > … guess that will be my definition of Industry 5.0 … “connected
> > everything + common sense”)
> >
> > The incubator is intended for teaching the initial community how
> > things work at Apache, I guess as we’d be starting with mainly people
> > that have already brought projects from outside into the incubator and
> > into TLPs, this teaching effort will not exist in this case. Even
> > Justin mentioned that this would probably be a case for straight TLP.
> >
> > Regarding the testing … not sure I understand what you want to test?
> > The idea of an open-source historian?
> > I mean the need exists … I’ve seen it in multiple occasions and our
> > friends from IoTDB have confirmed they have built or at least seen
> > IoTDB serve as such a system.
> > Not 100% sure what you want to try out.
> >
> > But I agree … we should probably start with an (online) workshop, as
> > this time we’d probably be spread out a bit wider geographically, than
> > with our PLC4X workshops in the past.
> >
> > Chris
> >
> >
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 12:37
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hey,
> >
> > I understand your point and also agree generally.
> > But I personally dislike the straight to TLP route as this is
> > something I’d like to try out first.
> > And my point with the separate repo (or whatever) was just to make the
> > community ready for the Incubator (where a codebase and community
> > normally should exist).
> >
> > But the easiest way in any regard would be to start of as a subproject
> > in an existing PMC. This would only require to get another repo and we
> > could start (formally no need to even register it as a subproject).
> > And the repo thing could be done by you alone I think as Chair.
> >
> > Next steps would then be more of brainstorming and workshoping and
> > building something (whatever?).
> > I think the timing is quite good before Christmas because if its
> > intereting, I see myself contributing over the Holidays (better don’t
> > show this mail to my wife…).
> >
> > Best
> > Julian
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 09:27
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> > <ho...@ltsoft.de>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Julian,
> >
> > happy you like the idea ;-)
> >
> > We would probably outclass the established ones in point of price ;-)
> > Right now, I think we should probably set up something that bundles
> > IoTDB and PLC4X and then the “code of the project” is the bubble-wrap,
> > that brings the two together and then provides an API that is
> > equivalent to existing Historian products.
> > I think the only way we can score points here, is if switching to it
> > is as easy as possible and that only works if it’s sort of a drop-in
> > replacement.
> >
> > Currently also discussing with other board members and the IPMC chair
> > … he at least would not see us in the incubator. If there are enough
> > PMC members of other projects and Apache Members on board, we could
> > take the straight to TLP route.
> >
> > And I would really like to keep it at Apache and not set up something
> > outside. This for multiple reasons:
> >
> >  * Protection (The market for the established products is huge and the
> > companies behind them are also huge with huge legal departments, I
> > would fear the same as I did with PLC4X, when I started it … don’t
> > want to be stuck in nuisance lawsuits, just filed for keeping us
> > occupied)
> >  * The satement: we want Open-Source to be accepted by the industry
> > and for me “Apache” is the form of Open-Source that I admire most. If
> > it’s built up out of Apache stuff, it should be an Apache project (If
> > we establish this in a way, that the industry notices Apache as a
> > “vendor”, this will benefit many other projects)
> >
> > Chris
> >
> >
> >
> > From: Julian Feinauer <j....@pragmaticindustries.de>
> > Date: Monday, 28. November 2022 at 09:17
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> > <ho...@ltsoft.de>
> > Subject: AW: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > first of all.. sorry for coming in so late.
> > I think this is a great idea and there really is a product market fit
> > for such a solution.
> > I think technology wise its not "that complicated" to build such a
> > system, I think many of the details that are necessary for a good
> > adoption of such a software is in the areas around like documentation,
> > marketing, more marketing, even more and even bolder marketing and
> > sales (how to sell an open source project??).
> >
> > But technology wise I really like the idea and I think with the
> > technology one has today it is very easy to "outclass" these
> > established systems e.g. in performance, efficiency and als user
> > friendliness.
> > The only thing which might be challenging for the future are things
> > like plugins or an extension system to allow users to customize their
> > installation.
> >
> > Regarding the "separate project" approach I am not 100% certain.
> > Personally, I would consider starting EITHER as a subproject in an
> > existing PMC (PLC4X?) or as a separate undertaking on GitHub or
> > somewhere else and not directly go to the Incubator or something.
> > Because I think we should really find out if there is enough developer
> > interest to build and sustain such a system.
> >
> > But I'm totally in for such a system either way!
> >
> > Julian
> >
> > PS.: Also forwarding the email to Björn and Tim who work(ed) a lot
> > with historians
> >
> > Von: Christofer Dutz <ch...@c-ware.de>
> > Datum: Montag, 28. November 2022 um 08:56
> > An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Betreff: Re: Building a PLC4X and IoTDB Historian?
> > Hi Xiangdong,
> >
> > that’s perfect that some of you folks would be on board. I should
> > probably check, if there are enough PLC4X folks on board too … would
> > be a shame if it was just me ;-)
> >
> > And regarding the name … yeah … you are absolutely right: Historian
> > might really be the ideal choice … as it instantly explains what it is
> > … A Historian from the Apache Software Foundation and probably also no
> > bias towards any regional culture.
> >
> > So (Speaking to my fellow PLC4X folks … who would be on board with
> > this?)
> >
> >
> > Chris
> >
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Monday, 28. November 2022 at 04:24
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > I guess I would sort of also vote for a separate Project. Would
> > however only make sense if some people from both of our projects would
> > join in.
> >
> > Agree, I think the iotdb developers in Timecho can join.
> >
> > > And name-wise … one of the names of Historic Historians (Ideally a
> > Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> > Jenkins are “famous” Buttler names ;-)
> >
> > If we consider famous names (well, there will be culture bias), I'd
> > like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
> > "Records of the Grand Historian" is tooo famous (at least in China).
> >
> > BTW. naming "Apache Historian" directly may be another option.
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >
> > Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
> > >
> > > How about Apache Ephorous? ;-)
> > >
> > > https://en.wikipedia.org/wiki/Ephorus
> > > Ephorus of Cyme
> > (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>;
> > Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος,
> > Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient
> > Greek<https://en.wikipedia.org/wiki/Ancient_Greece>
> > historian<https://en.wikipedia.org/wiki/Historian> known for his
> > universal history<https://en.wikipedia.org/wiki/Universal_history>.
> > >
> > > Chris
> > >
> > > From: Christofer Dutz <ch...@c-ware.de>
> > > Date: Sunday, 27. November 2022 at 15:28
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > I guess I would sort of also vote for a separate Project. Would
> > however only make sense if some people from both of our projects would
> > join in.
> > >
> > > And name-wise … one of the names of Historic Historians (Ideally a
> > Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> > Jenkins are “famous” Buttler names ;-)
> > > https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> > > https://en.wikipedia.org/wiki/List_of_historians
> > >
> > > I think anything but a turnkey-ready solution will not be accepted
> > by anyone in the Automation industry.
> > >
> > > Chris
> > >
> > >
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 14:05
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > If we just provide a solution, or demonstration, then either is ok.
> > > If we want to provide an "one-box thing" (even without GUI), +1 for
> > a
> > > new project.
> > >
> > > > I don’t even think it would be bad if an entity like Timecho would
> > add enterprise offerings, because I know, that the industry won’t
> > “buy” something, if there’s no commercial support or anyone, they can
> > throw money at, even if it’s free (That might even make them more
> > skeptical).
> > > Yes, indeed. :D
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > >
> > > Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> > > >
> > > > Hi,
> > > >
> > > > I think it's a great idea.
> > > >
> > > > I'd vote for having a separate project for something like this,
> > just so
> > > > it's clear that it is a historian and people can build it
> > separately. I
> > > > wouldn't want it hidden away in an obscure folder in the PLC4X
> > repo.
> > > >
> > > > Cake does go with Cafe.
> > > >
> > > > Ben
> > > >
> > > >
> > > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz
> > <ch...@c-ware.de>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Trying to respond to all (except the naming question ;-) ):
> > > > >
> > > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we
> > could be
> > > > > collecting data on one node, writing that to TSFiles using the
> > IoTDB-Client
> > > > > (As Xaingdong mentioned).
> > > > > These could either be online and transfer data to the server
> > right away,
> > > > > or operate in an offline-like mode and periodically fetch data
> > in bursts
> > > > > from the client (Hope that’s correct, please correct me if I’m
> > wrong)
> > > > >
> > > > > @Xaiangdong: In general, PLC4X operates in an active mode, but
> > we also
> > > > > have a proposed and prototypically proven “passive-mode”. Here
> > the software
> > > > > doesn’t actively participate in the data acquisition directly,
> > but simply
> > > > > listens to the data stream and makes data available. We also
> > have something
> > > > > we call a “Data-Diode”, which technically makes interference
> > impossible as
> > > > > it lets ethernet packets flow in one direction, but nothing in
> > the other
> > > > > (Firewalls usually have the problem of letting packets pass in
> > both
> > > > > directions for established connections). We invented this,
> > because we knew
> > > > > we will never have validated and audited open-source software,
> > to a level
> > > > > that it would be certified for some of these use cases.
> > > > >
> > > > > So, either we can live with everything the SCADA system is
> > already
> > > > > requesting, or we add an active PLC4X node in the secure
> > network, that
> > > > > requests data, and simply ignores it, and a second – passive
> > mode – node
> > > > > sits outside the secure network to capture the information.
> > > > >
> > > > > I agree that this sort of thing needs to be a “product”. The
> > Automation
> > > > > Industry just doesn’t know how to work with frameworks. Ideally
> > a one-box
> > > > > thing.
> > > > > Right now, most tools I have seen don’t even need the
> > calculation or the
> > > > > visualization. This is usually done on another level. Important
> > would be
> > > > > that we could be somewhat API compatible with existing products.
> > Them
> > > > > usually having SQL or REST APIs, should make it relatively easy
> > to sort of
> > > > > produce frontends to our Apache Historian, that are API
> > compatible with
> > > > > some existing industry products, so they are replaceable.
> > > > >
> > > > > If we wanted to add a visualization layer or a data-curation
> > layer later
> > > > > on, we should have a chat with the NiFi or StreamPipes folks as
> > that’s what
> > > > > they already have.
> > > > >
> > > > > For now, I would be proposing to build something that uses PLC4X
> > for the
> > > > > data-acquisition, IoTDB for the storage and build a REST
> > frontend for this,
> > > > > that’s somewhat API compatible with one of the major established
> > products
> > > > > and to add more on a step-by-step basis.
> > > > >
> > > > > I don’t even think it would be bad if an entity like Timecho
> > would add
> > > > > enterprise offerings, because I know, that the industry won’t
> > “buy”
> > > > > something, if there’s no commercial support or anyone, they can
> > throw money
> > > > > at, even if it’s free (That might even make them more
> > skeptical).
> > > > >
> > > > >
> > > > > What do you all think? Does this make sense? If yes, where would
> > it make
> > > > > sense to start working on something like this? In the IoTDB
> > project? In the
> > > > > PLC4X project, in a separate (new) project?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > > > >
> > > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > > Date: Sunday, 27. November 2022 at 06:02
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > > Hi all,
> > > > >
> > > > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > > > Actually, there are several users using IoTDB as their historian
> > > > > solution...
> > > > >
> > > > > According to my knowledge, a historian software contains several
> > features:
> > > > >
> > > > > - get data from OPC/modubs server and write to IoTDB
> > > > > * use PLC4x + IoTDB-client. (we call this data collector)
> > > > >
> > > > > - (optional) there is a single-way network gateway for security.
> > > > > * If the gateway is deployed between device and PLC4x program,
> > then
> > > > > we need to check whether plc4x supports that.
> > > > > * If the gateway is deployed between IoTDB-client and IoTDB,
> > then we
> > > > > need to check wheter IoTDB-client supports that.
> > > > >
> > > > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > > > collect A and B from a device (for example, C=A+B/2 )
> > > > > * IoTDB's trigger supports that but we need a Drag-and-drop
> > > > > programming GUI. (maybe integrating some open source projects,
> > like
> > > > > Red-node is a good idea)
> > > > >
> > > > > - get the latest data from IoTDB for visualization (usually
> > users need
> > > > > to draw a process flow diagram, and put the latest data on each
> > device
> > > > > icon on the diagram)
> > > > > * IoTDB supports getting latest data. but we need a new software
> > for
> > > > > drawing the diagram (I do not know if there is any open source
> > project
> > > > > for this. In our real applications, we ususally buy and
> > integrate some
> > > > > other commerical software)
> > > > >
> > > > > - send alert message if the real time data meets some rules
> > > > > * IoTDB trigger supports the rule. but currently we have no a
> > GUI.
> > > > > (IoTDB also supports Prometheus Alert Manager)
> > > > >
> > > > > - get the historical data
> > > > > * IoTDB supports that and grafana is good at visualization
> > > > >
> > > > > - last but not the least, if we provide all the features to
> > industrial
> > > > > users, all the GUIs should be integrated into one entrance.
> > > > >
> > > > >
> > > > > Best,
> > > > > -----------------------------------
> > > > > Xiangdong Huang
> > > > > School of Software, Tsinghua University
> > > > >
> > > > > 黄向东
> > > > > 清华大学 软件学院
> > > > >
> > > > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > > > >
> > > > > > Hi folks, long time lurker…
> > > > > >
> > > > > > A historian would be great. Does PLC4X queue data on the
> > machine it’s
> > > > > > running on already?
> > > > > >
> > > > > > A common pattern in historians is to queue data on the runtime
> > server
> > > > > until
> > > > > > data can be extracted to a sql server, preventing data loss,
> > and reducing
> > > > > > the need for redundancy.
> > > > > >
> > > > > > There are a ton of edge cases to watch out for which I can
> > elaborate on
> > > > > > further.
> > > > > >
> > > > > > I’m a former Industrial Controls Engineer turned software dev
> > and can
> > > > > help
> > > > > > out where needed.
> > > > > >
> > > > > > -Ryan
> > > > > >
> > > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler
> > <ot...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Would you name it “Cake”?
> > > > > > >
> > > > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > > > <ch...@c-ware.de>
> > > > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > > > dev@plc4x.apache.org>
> > > > > > > Date: November 26, 2022 at 10:36:23
> > > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > <dev@plc4x.apache.org
> > > > > >,
> > > > > > > dev@iotdb.apache.org <de...@iotdb.apache.org>
> > <de...@iotdb.apache.org>
> > > > > > > Subject: Building a PLC4X and IoTDB Historian?
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I am currently thinking, if it wouldn’t be a good idea to
> > build
> > > > > something
> > > > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > > > >
> > > > > > > For those of you, who don’t know what a Historian is. In the
> > > > > manufacturing
> > > > > > > industry they have these extremely expensive servers
> > (usually you buy
> > > > > them
> > > > > > > as a bundle of hard- and software).
> > > > > > > These servers are nothing else than a really crappy and
> > brutally
> > > > > expensive
> > > > > > > Database for storing Time-Series data.
> > > > > > >
> > > > > > > The thing is most commercial products are currently really
> > having
> > > > > trouble
> > > > > > > to keep up with the increasing amount of Data being sent.
> > > > > > >
> > > > > > > So instead of suggesting building an Historian at each
> > customer’s
> > > > > site, I
> > > > > > > thought: Perhaps a ready-to-use solution based on open-
> > source would be
> > > > > a
> > > > > > > good idea.
> > > > > > > Not 100% sure where I’d locate such an initiative, but I
> > would tend to
> > > > > see
> > > > > > > it more on the Database side.
> > > > > > >
> > > > > > > I think IoTDB would be the perfect storage system, all we
> > seem to need
> > > > > is
> > > > > > > some sort of rest-interface that matches the industry
> > standards for
> > > > > > > querying the information and on the other side something
> > like PLC4X to
> > > > > fill
> > > > > > > the database.
> > > > > > >
> > > > > > > Here some examples:
> > > > > > >
> > > > > > >
> > > > >
> > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > > > >
> > > > > https://cdn.logic-
> > control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > > > >
> > > > > > > What do you folks think?
> > > > > > >
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > >

Re: AW: Building a PLC4X and IoTDB Historian?

[Niklas Merz <ni...@apache.org>]

Hello all,

I think this is a good compromise to get started and then publish it
with a bit of fanfare (marketing) as an Apache project to the world once
it's ready.

I am also interested in working with you on this project. I think PLC4X
and IoTDB need something like this to become popular in the automation
world.

Regards
Niklas

On November 28, 2022, Julian Feinauer
<j....@pragmaticindustries.de> wrote:
> I fully agree with that.
> If we get something to work in private we could easily go TLP or
> through the Incubator whatever is decided then to do.
>
> Julian
>
> PS.: Do you setup the repo and invite myself?
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 13:47
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> And if we already agree on a name (I think “Apache Historian” would be
> a good name),
> we could even already work on code with that package-name and maven
> coordinates, so we don’t have to change much when coming to Apache?
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Monday, 28. November 2022 at 13:18
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> Indeed, your concerns are valid regarding creating an already
> abandoned TLP.
>
> In that case, I would opt for a private repo on GitHub which we invite
> everyone willing to help work on it and as soon as we have something
> workable, we make it public as part of a TLP?
>
> This avoids at least the fear of creating a new TLP that’s product
> doesn’t even see the light of day. And as soon as we have something we
> want to show the world, we join Apache?
>
> Would that be a compromise?
>
>
> Chris
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 13:14
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hey,
>
> as I said.. we could just do it as part of one of the TLPs like PLC4X
> or IoTDB easily or in a private repo as well, for sure.
> I just want to avoid to create a TLP which is soon abandoned, that’s
> my only concern.
> Regarding all your comments I totally agree and am with you.
>
> Julian
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 13:07
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> well … I would be hesitant to try something like this without the
> legal shield of the ASF … from what I’ve learned in the early days,
> Siemens and Co. continuously threatened to sue me for my work on
> PLC4X.
> The only thing that helped me then, was to say: I am doing everything
> legally, so you’d have trouble succeeding with that and you also can’t
> sue me but would have to sue the ASF and good luck on that not
> backfiring badly on you publicly.
>
> I know that in the sector of drivers in the past there were several
> occasions where open-source developers were annoyed so much by
> nuisance law-suites, that they gave up.
>
> I also know Historians are silly expensive systems. I haven’t seen one
> under 100k€ and the annual fees for certified solutions tend to start
> at 100k€/year/production-line … so there’s a lot of money for them to
> fear losing. So yes: I’m too scared to try something like this
> publicly without a legal shield.
>
> We could however work on something like this in a protected repo at
> GitHub etc. and go to TLP as soon as we have something.
>
> Would that be an option?
>
> Chris
>
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 12:57
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hey Chris,
>
> wrt testing I meant how „sustainable” the community is, that we intend
> to building.
> You know the effect pretty well if you ask “How wants to start this”
> everybody is like “me, me, me” and after 2 weeks you stand there
> alone.
> And one important aspect of a TLP for me is that the community is also
> “resilient” and will last “long” (whatever that means).
> And this is something I think is hard to guarantee in the current
> state.
>
> Technically I have no worries, this is something we would get to work
> rather easily, I see no “magics” hidden there.
>
> Julian
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 12:48
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> Well, the problem with starting at one Project as a subproject … We
> then need to make everyone involved or wanting to be involved in the
> other project to become committers in the other.
> I wouldn’t have problems with doing it under the hood of IoTDB,
> personally.
>
> A separate Project would have the benefit of giving the industry an
> impression that this is something built just for them instead of being
> a side-product of something that’s already there (I have given up
> expecting the Automation industry to use any form of common sense ;-)
> … guess that will be my definition of Industry 5.0 … “connected
> everything + common sense”)
>
> The incubator is intended for teaching the initial community how
> things work at Apache, I guess as we’d be starting with mainly people
> that have already brought projects from outside into the incubator and
> into TLPs, this teaching effort will not exist in this case. Even
> Justin mentioned that this would probably be a case for straight TLP.
>
> Regarding the testing … not sure I understand what you want to test?
> The idea of an open-source historian?
> I mean the need exists … I’ve seen it in multiple occasions and our
> friends from IoTDB have confirmed they have built or at least seen
> IoTDB serve as such a system.
> Not 100% sure what you want to try out.
>
> But I agree … we should probably start with an (online) workshop, as
> this time we’d probably be spread out a bit wider geographically, than
> with our PLC4X workshops in the past.
>
> Chris
>
>
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 12:37
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hey,
>
> I understand your point and also agree generally.
> But I personally dislike the straight to TLP route as this is
> something I’d like to try out first.
> And my point with the separate repo (or whatever) was just to make the
> community ready for the Incubator (where a codebase and community
> normally should exist).
>
> But the easiest way in any regard would be to start of as a subproject
> in an existing PMC. This would only require to get another repo and we
> could start (formally no need to even register it as a subproject).
> And the repo thing could be done by you alone I think as Chair.
>
> Next steps would then be more of brainstorming and workshoping and
> building something (whatever?).
> I think the timing is quite good before Christmas because if its
> intereting, I see myself contributing over the Holidays (better don’t
> show this mail to my wife…).
>
> Best
> Julian
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 09:27
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> <ho...@ltsoft.de>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Julian,
>
> happy you like the idea ;-)
>
> We would probably outclass the established ones in point of price ;-)
> Right now, I think we should probably set up something that bundles
> IoTDB and PLC4X and then the “code of the project” is the bubble-wrap,
> that brings the two together and then provides an API that is
> equivalent to existing Historian products.
> I think the only way we can score points here, is if switching to it
> is as easy as possible and that only works if it’s sort of a drop-in
> replacement.
>
> Currently also discussing with other board members and the IPMC chair
> … he at least would not see us in the incubator. If there are enough
> PMC members of other projects and Apache Members on board, we could
> take the straight to TLP route.
>
> And I would really like to keep it at Apache and not set up something
> outside. This for multiple reasons:
>
>  * Protection (The market for the established products is huge and the
> companies behind them are also huge with huge legal departments, I
> would fear the same as I did with PLC4X, when I started it … don’t
> want to be stuck in nuisance lawsuits, just filed for keeping us
> occupied)
>  * The satement: we want Open-Source to be accepted by the industry
> and for me “Apache” is the form of Open-Source that I admire most. If
> it’s built up out of Apache stuff, it should be an Apache project (If
> we establish this in a way, that the industry notices Apache as a
> “vendor”, this will benefit many other projects)
>
> Chris
>
>
>
> From: Julian Feinauer <j....@pragmaticindustries.de>
> Date: Monday, 28. November 2022 at 09:17
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de
> <ho...@ltsoft.de>
> Subject: AW: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> first of all.. sorry for coming in so late.
> I think this is a great idea and there really is a product market fit
> for such a solution.
> I think technology wise its not "that complicated" to build such a
> system, I think many of the details that are necessary for a good
> adoption of such a software is in the areas around like documentation,
> marketing, more marketing, even more and even bolder marketing and
> sales (how to sell an open source project??).
>
> But technology wise I really like the idea and I think with the
> technology one has today it is very easy to "outclass" these
> established systems e.g. in performance, efficiency and als user
> friendliness.
> The only thing which might be challenging for the future are things
> like plugins or an extension system to allow users to customize their
> installation.
>
> Regarding the "separate project" approach I am not 100% certain.
> Personally, I would consider starting EITHER as a subproject in an
> existing PMC (PLC4X?) or as a separate undertaking on GitHub or
> somewhere else and not directly go to the Incubator or something.
> Because I think we should really find out if there is enough developer
> interest to build and sustain such a system.
>
> But I'm totally in for such a system either way!
>
> Julian
>
> PS.: Also forwarding the email to Björn and Tim who work(ed) a lot
> with historians
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 08:56
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Xiangdong,
>
> that’s perfect that some of you folks would be on board. I should
> probably check, if there are enough PLC4X folks on board too … would
> be a shame if it was just me ;-)
>
> And regarding the name … yeah … you are absolutely right: Historian
> might really be the ideal choice … as it instantly explains what it is
> … A Historian from the Apache Software Foundation and probably also no
> bias towards any regional culture.
>
> So (Speaking to my fellow PLC4X folks … who would be on board with
> this?)
>
>
> Chris
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Monday, 28. November 2022 at 04:24
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> > I guess I would sort of also vote for a separate Project. Would
> however only make sense if some people from both of our projects would
> join in.
>
> Agree, I think the iotdb developers in Timecho can join.
>
> > And name-wise … one of the names of Historic Historians (Ideally a
> Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> Jenkins are “famous” Buttler names ;-)
>
> If we consider famous names (well, there will be culture bias), I'd
> like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
> "Records of the Grand Historian" is tooo famous (at least in China).
>
> BTW. naming "Apache Historian" directly may be another option.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>
> Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
> >
> > How about Apache Ephorous? ;-)
> >
> > https://en.wikipedia.org/wiki/Ephorus
> > Ephorus of Cyme
> (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>;
> Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος,
> Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient
> Greek<https://en.wikipedia.org/wiki/Ancient_Greece>
> historian<https://en.wikipedia.org/wiki/Historian> known for his
> universal history<https://en.wikipedia.org/wiki/Universal_history>.
> >
> > Chris
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > Date: Sunday, 27. November 2022 at 15:28
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > I guess I would sort of also vote for a separate Project. Would
> however only make sense if some people from both of our projects would
> join in.
> >
> > And name-wise … one of the names of Historic Historians (Ideally a
> Greek one) would make it into my top 10 ;-) … sort of like Hudson or
> Jenkins are “famous” Buttler names ;-)
> > https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> > https://en.wikipedia.org/wiki/List_of_historians
> >
> > I think anything but a turnkey-ready solution will not be accepted
> by anyone in the Automation industry.
> >
> > Chris
> >
> >
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Sunday, 27. November 2022 at 14:05
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > If we just provide a solution, or demonstration, then either is ok.
> > If we want to provide an "one-box thing" (even without GUI), +1 for
> a
> > new project.
> >
> > > I don’t even think it would be bad if an entity like Timecho would
> add enterprise offerings, because I know, that the industry won’t
> “buy” something, if there’s no commercial support or anyone, they can
> throw money at, even if it’s free (That might even make them more
> skeptical).
> > Yes, indeed. :D
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> >
> > Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> > >
> > > Hi,
> > >
> > > I think it's a great idea.
> > >
> > > I'd vote for having a separate project for something like this,
> just so
> > > it's clear that it is a historian and people can build it
> separately. I
> > > wouldn't want it hidden away in an obscure folder in the PLC4X
> repo.
> > >
> > > Cake does go with Cafe.
> > >
> > > Ben
> > >
> > >
> > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz
> <ch...@c-ware.de>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Trying to respond to all (except the naming question ;-) ):
> > > >
> > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we
> could be
> > > > collecting data on one node, writing that to TSFiles using the
> IoTDB-Client
> > > > (As Xaingdong mentioned).
> > > > These could either be online and transfer data to the server
> right away,
> > > > or operate in an offline-like mode and periodically fetch data
> in bursts
> > > > from the client (Hope that’s correct, please correct me if I’m
> wrong)
> > > >
> > > > @Xaiangdong: In general, PLC4X operates in an active mode, but
> we also
> > > > have a proposed and prototypically proven “passive-mode”. Here
> the software
> > > > doesn’t actively participate in the data acquisition directly,
> but simply
> > > > listens to the data stream and makes data available. We also
> have something
> > > > we call a “Data-Diode”, which technically makes interference
> impossible as
> > > > it lets ethernet packets flow in one direction, but nothing in
> the other
> > > > (Firewalls usually have the problem of letting packets pass in
> both
> > > > directions for established connections). We invented this,
> because we knew
> > > > we will never have validated and audited open-source software,
> to a level
> > > > that it would be certified for some of these use cases.
> > > >
> > > > So, either we can live with everything the SCADA system is
> already
> > > > requesting, or we add an active PLC4X node in the secure
> network, that
> > > > requests data, and simply ignores it, and a second – passive
> mode – node
> > > > sits outside the secure network to capture the information.
> > > >
> > > > I agree that this sort of thing needs to be a “product”. The
> Automation
> > > > Industry just doesn’t know how to work with frameworks. Ideally
> a one-box
> > > > thing.
> > > > Right now, most tools I have seen don’t even need the
> calculation or the
> > > > visualization. This is usually done on another level. Important
> would be
> > > > that we could be somewhat API compatible with existing products.
> Them
> > > > usually having SQL or REST APIs, should make it relatively easy
> to sort of
> > > > produce frontends to our Apache Historian, that are API
> compatible with
> > > > some existing industry products, so they are replaceable.
> > > >
> > > > If we wanted to add a visualization layer or a data-curation
> layer later
> > > > on, we should have a chat with the NiFi or StreamPipes folks as
> that’s what
> > > > they already have.
> > > >
> > > > For now, I would be proposing to build something that uses PLC4X
> for the
> > > > data-acquisition, IoTDB for the storage and build a REST
> frontend for this,
> > > > that’s somewhat API compatible with one of the major established
> products
> > > > and to add more on a step-by-step basis.
> > > >
> > > > I don’t even think it would be bad if an entity like Timecho
> would add
> > > > enterprise offerings, because I know, that the industry won’t
> “buy”
> > > > something, if there’s no commercial support or anyone, they can
> throw money
> > > > at, even if it’s free (That might even make them more
> skeptical).
> > > >
> > > >
> > > > What do you all think? Does this make sense? If yes, where would
> it make
> > > > sense to start working on something like this? In the IoTDB
> project? In the
> > > > PLC4X project, in a separate (new) project?
> > > >
> > > >
> > > > Chris
> > > >
> > > >
> > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > Date: Sunday, 27. November 2022 at 06:02
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > Hi all,
> > > >
> > > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > > Actually, there are several users using IoTDB as their historian
> > > > solution...
> > > >
> > > > According to my knowledge, a historian software contains several
> features:
> > > >
> > > > - get data from OPC/modubs server and write to IoTDB
> > > > * use PLC4x + IoTDB-client. (we call this data collector)
> > > >
> > > > - (optional) there is a single-way network gateway for security.
> > > > * If the gateway is deployed between device and PLC4x program,
> then
> > > > we need to check whether plc4x supports that.
> > > > * If the gateway is deployed between IoTDB-client and IoTDB,
> then we
> > > > need to check wheter IoTDB-client supports that.
> > > >
> > > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > > collect A and B from a device (for example, C=A+B/2 )
> > > > * IoTDB's trigger supports that but we need a Drag-and-drop
> > > > programming GUI. (maybe integrating some open source projects,
> like
> > > > Red-node is a good idea)
> > > >
> > > > - get the latest data from IoTDB for visualization (usually
> users need
> > > > to draw a process flow diagram, and put the latest data on each
> device
> > > > icon on the diagram)
> > > > * IoTDB supports getting latest data. but we need a new software
> for
> > > > drawing the diagram (I do not know if there is any open source
> project
> > > > for this. In our real applications, we ususally buy and
> integrate some
> > > > other commerical software)
> > > >
> > > > - send alert message if the real time data meets some rules
> > > > * IoTDB trigger supports the rule. but currently we have no a
> GUI.
> > > > (IoTDB also supports Prometheus Alert Manager)
> > > >
> > > > - get the historical data
> > > > * IoTDB supports that and grafana is good at visualization
> > > >
> > > > - last but not the least, if we provide all the features to
> industrial
> > > > users, all the GUIs should be integrated into one entrance.
> > > >
> > > >
> > > > Best,
> > > > -----------------------------------
> > > > Xiangdong Huang
> > > > School of Software, Tsinghua University
> > > >
> > > > 黄向东
> > > > 清华大学 软件学院
> > > >
> > > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > > >
> > > > > Hi folks, long time lurker…
> > > > >
> > > > > A historian would be great. Does PLC4X queue data on the
> machine it’s
> > > > > running on already?
> > > > >
> > > > > A common pattern in historians is to queue data on the runtime
> server
> > > > until
> > > > > data can be extracted to a sql server, preventing data loss,
> and reducing
> > > > > the need for redundancy.
> > > > >
> > > > > There are a ton of edge cases to watch out for which I can
> elaborate on
> > > > > further.
> > > > >
> > > > > I’m a former Industrial Controls Engineer turned software dev
> and can
> > > > help
> > > > > out where needed.
> > > > >
> > > > > -Ryan
> > > > >
> > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler
> <ot...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Would you name it “Cake”?
> > > > > >
> > > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > > <ch...@c-ware.de>
> > > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > > dev@plc4x.apache.org>
> > > > > > Date: November 26, 2022 at 10:36:23
> > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> <dev@plc4x.apache.org
> > > > >,
> > > > > > dev@iotdb.apache.org <de...@iotdb.apache.org>
> <de...@iotdb.apache.org>
> > > > > > Subject: Building a PLC4X and IoTDB Historian?
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I am currently thinking, if it wouldn’t be a good idea to
> build
> > > > something
> > > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > > >
> > > > > > For those of you, who don’t know what a Historian is. In the
> > > > manufacturing
> > > > > > industry they have these extremely expensive servers
> (usually you buy
> > > > them
> > > > > > as a bundle of hard- and software).
> > > > > > These servers are nothing else than a really crappy and
> brutally
> > > > expensive
> > > > > > Database for storing Time-Series data.
> > > > > >
> > > > > > The thing is most commercial products are currently really
> having
> > > > trouble
> > > > > > to keep up with the increasing amount of Data being sent.
> > > > > >
> > > > > > So instead of suggesting building an Historian at each
> customer’s
> > > > site, I
> > > > > > thought: Perhaps a ready-to-use solution based on open-
> source would be
> > > > a
> > > > > > good idea.
> > > > > > Not 100% sure where I’d locate such an initiative, but I
> would tend to
> > > > see
> > > > > > it more on the Database side.
> > > > > >
> > > > > > I think IoTDB would be the perfect storage system, all we
> seem to need
> > > > is
> > > > > > some sort of rest-interface that matches the industry
> standards for
> > > > > > querying the information and on the other side something
> like PLC4X to
> > > > fill
> > > > > > the database.
> > > > > >
> > > > > > Here some examples:
> > > > > >
> > > > > >
> > > >
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > > >
> > > > https://cdn.logic-
> control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > > >
> > > > > > What do you folks think?
> > > > > >
> > > > > >
> > > > > > Chris
> > > > > >
> > > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

I can do that …

if the others agree that this the best way to go?

Chris

From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 13:50
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
I fully agree with that.
If we get something to work in private we could easily go TLP or through the Incubator whatever is decided then to do.

Julian

PS.: Do you setup the repo and invite myself?

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 13:47
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
And if we already agree on a name (I think “Apache Historian” would be a good name),
we could even already work on code with that package-name and maven coordinates, so we don’t have to change much when coming to Apache?

Chris

From: Christofer Dutz <ch...@c-ware.de>
Date: Monday, 28. November 2022 at 13:18
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Indeed, your concerns are valid regarding creating an already abandoned TLP.

In that case, I would opt for a private repo on GitHub which we invite everyone willing to help work on it and as soon as we have something workable, we make it public as part of a TLP?

This avoids at least the fear of creating a new TLP that’s product doesn’t even see the light of day. And as soon as we have something we want to show the world, we join Apache?

Would that be a compromise?


Chris

From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 13:14
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

as I said.. we could just do it as part of one of the TLPs like PLC4X or IoTDB easily or in a private repo as well, for sure.
I just want to avoid to create a TLP which is soon abandoned, that’s my only concern.
Regarding all your comments I totally agree and am with you.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 13:07
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

well … I would be hesitant to try something like this without the legal shield of the ASF … from what I’ve learned in the early days, Siemens and Co. continuously threatened to sue me for my work on PLC4X.
The only thing that helped me then, was to say: I am doing everything legally, so you’d have trouble succeeding with that and you also can’t sue me but would have to sue the ASF and good luck on that not backfiring badly on you publicly.

I know that in the sector of drivers in the past there were several occasions where open-source developers were annoyed so much by nuisance law-suites, that they gave up.

I also know Historians are silly expensive systems. I haven’t seen one under 100k€ and the annual fees for certified solutions tend to start at 100k€/year/production-line … so there’s a lot of money for them to fear losing. So yes: I’m too scared to try something like this publicly without a legal shield.

We could however work on something like this in a protected repo at GitHub etc. and go to TLP as soon as we have something.

Would that be an option?

Chris


From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:57
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey Chris,

wrt testing I meant how „sustainable” the community is, that we intend to building.
You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone.
And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means).
And this is something I think is hard to guarantee in the current state.

Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 12:48
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

AW: Building a PLC4X and IoTDB Historian?

[Julian Feinauer <j....@pragmaticindustries.de>]

I fully agree with that.
If we get something to work in private we could easily go TLP or through the Incubator whatever is decided then to do.

Julian

PS.: Do you setup the repo and invite myself?

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 13:47
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
And if we already agree on a name (I think “Apache Historian” would be a good name),
we could even already work on code with that package-name and maven coordinates, so we don’t have to change much when coming to Apache?

Chris

From: Christofer Dutz <ch...@c-ware.de>
Date: Monday, 28. November 2022 at 13:18
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Indeed, your concerns are valid regarding creating an already abandoned TLP.

In that case, I would opt for a private repo on GitHub which we invite everyone willing to help work on it and as soon as we have something workable, we make it public as part of a TLP?

This avoids at least the fear of creating a new TLP that’s product doesn’t even see the light of day. And as soon as we have something we want to show the world, we join Apache?

Would that be a compromise?


Chris

From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 13:14
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

as I said.. we could just do it as part of one of the TLPs like PLC4X or IoTDB easily or in a private repo as well, for sure.
I just want to avoid to create a TLP which is soon abandoned, that’s my only concern.
Regarding all your comments I totally agree and am with you.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 13:07
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

well … I would be hesitant to try something like this without the legal shield of the ASF … from what I’ve learned in the early days, Siemens and Co. continuously threatened to sue me for my work on PLC4X.
The only thing that helped me then, was to say: I am doing everything legally, so you’d have trouble succeeding with that and you also can’t sue me but would have to sue the ASF and good luck on that not backfiring badly on you publicly.

I know that in the sector of drivers in the past there were several occasions where open-source developers were annoyed so much by nuisance law-suites, that they gave up.

I also know Historians are silly expensive systems. I haven’t seen one under 100k€ and the annual fees for certified solutions tend to start at 100k€/year/production-line … so there’s a lot of money for them to fear losing. So yes: I’m too scared to try something like this publicly without a legal shield.

We could however work on something like this in a protected repo at GitHub etc. and go to TLP as soon as we have something.

Would that be an option?

Chris


From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:57
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey Chris,

wrt testing I meant how „sustainable” the community is, that we intend to building.
You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone.
And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means).
And this is something I think is hard to guarantee in the current state.

Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 12:48
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

And if we already agree on a name (I think “Apache Historian” would be a good name),
we could even already work on code with that package-name and maven coordinates, so we don’t have to change much when coming to Apache?

Chris

From: Christofer Dutz <ch...@c-ware.de>
Date: Monday, 28. November 2022 at 13:18
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Indeed, your concerns are valid regarding creating an already abandoned TLP.

In that case, I would opt for a private repo on GitHub which we invite everyone willing to help work on it and as soon as we have something workable, we make it public as part of a TLP?

This avoids at least the fear of creating a new TLP that’s product doesn’t even see the light of day. And as soon as we have something we want to show the world, we join Apache?

Would that be a compromise?


Chris

From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 13:14
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

as I said.. we could just do it as part of one of the TLPs like PLC4X or IoTDB easily or in a private repo as well, for sure.
I just want to avoid to create a TLP which is soon abandoned, that’s my only concern.
Regarding all your comments I totally agree and am with you.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 13:07
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

well … I would be hesitant to try something like this without the legal shield of the ASF … from what I’ve learned in the early days, Siemens and Co. continuously threatened to sue me for my work on PLC4X.
The only thing that helped me then, was to say: I am doing everything legally, so you’d have trouble succeeding with that and you also can’t sue me but would have to sue the ASF and good luck on that not backfiring badly on you publicly.

I know that in the sector of drivers in the past there were several occasions where open-source developers were annoyed so much by nuisance law-suites, that they gave up.

I also know Historians are silly expensive systems. I haven’t seen one under 100k€ and the annual fees for certified solutions tend to start at 100k€/year/production-line … so there’s a lot of money for them to fear losing. So yes: I’m too scared to try something like this publicly without a legal shield.

We could however work on something like this in a protected repo at GitHub etc. and go to TLP as soon as we have something.

Would that be an option?

Chris


From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:57
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey Chris,

wrt testing I meant how „sustainable” the community is, that we intend to building.
You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone.
And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means).
And this is something I think is hard to guarantee in the current state.

Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 12:48
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi Julian,

Indeed, your concerns are valid regarding creating an already abandoned TLP.

In that case, I would opt for a private repo on GitHub which we invite everyone willing to help work on it and as soon as we have something workable, we make it public as part of a TLP?

This avoids at least the fear of creating a new TLP that’s product doesn’t even see the light of day. And as soon as we have something we want to show the world, we join Apache?

Would that be a compromise?


Chris

From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 13:14
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

as I said.. we could just do it as part of one of the TLPs like PLC4X or IoTDB easily or in a private repo as well, for sure.
I just want to avoid to create a TLP which is soon abandoned, that’s my only concern.
Regarding all your comments I totally agree and am with you.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 13:07
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

well … I would be hesitant to try something like this without the legal shield of the ASF … from what I’ve learned in the early days, Siemens and Co. continuously threatened to sue me for my work on PLC4X.
The only thing that helped me then, was to say: I am doing everything legally, so you’d have trouble succeeding with that and you also can’t sue me but would have to sue the ASF and good luck on that not backfiring badly on you publicly.

I know that in the sector of drivers in the past there were several occasions where open-source developers were annoyed so much by nuisance law-suites, that they gave up.

I also know Historians are silly expensive systems. I haven’t seen one under 100k€ and the annual fees for certified solutions tend to start at 100k€/year/production-line … so there’s a lot of money for them to fear losing. So yes: I’m too scared to try something like this publicly without a legal shield.

We could however work on something like this in a protected repo at GitHub etc. and go to TLP as soon as we have something.

Would that be an option?

Chris


From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:57
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey Chris,

wrt testing I meant how „sustainable” the community is, that we intend to building.
You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone.
And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means).
And this is something I think is hard to guarantee in the current state.

Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 12:48
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

AW: Building a PLC4X and IoTDB Historian?

[Julian Feinauer <j....@pragmaticindustries.de>]

Hey,

as I said.. we could just do it as part of one of the TLPs like PLC4X or IoTDB easily or in a private repo as well, for sure.
I just want to avoid to create a TLP which is soon abandoned, that’s my only concern.
Regarding all your comments I totally agree and am with you.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 13:07
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

well … I would be hesitant to try something like this without the legal shield of the ASF … from what I’ve learned in the early days, Siemens and Co. continuously threatened to sue me for my work on PLC4X.
The only thing that helped me then, was to say: I am doing everything legally, so you’d have trouble succeeding with that and you also can’t sue me but would have to sue the ASF and good luck on that not backfiring badly on you publicly.

I know that in the sector of drivers in the past there were several occasions where open-source developers were annoyed so much by nuisance law-suites, that they gave up.

I also know Historians are silly expensive systems. I haven’t seen one under 100k€ and the annual fees for certified solutions tend to start at 100k€/year/production-line … so there’s a lot of money for them to fear losing. So yes: I’m too scared to try something like this publicly without a legal shield.

We could however work on something like this in a protected repo at GitHub etc. and go to TLP as soon as we have something.

Would that be an option?

Chris


From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:57
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey Chris,

wrt testing I meant how „sustainable” the community is, that we intend to building.
You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone.
And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means).
And this is something I think is hard to guarantee in the current state.

Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 12:48
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi Julian,

well … I would be hesitant to try something like this without the legal shield of the ASF … from what I’ve learned in the early days, Siemens and Co. continuously threatened to sue me for my work on PLC4X.
The only thing that helped me then, was to say: I am doing everything legally, so you’d have trouble succeeding with that and you also can’t sue me but would have to sue the ASF and good luck on that not backfiring badly on you publicly.

I know that in the sector of drivers in the past there were several occasions where open-source developers were annoyed so much by nuisance law-suites, that they gave up.

I also know Historians are silly expensive systems. I haven’t seen one under 100k€ and the annual fees for certified solutions tend to start at 100k€/year/production-line … so there’s a lot of money for them to fear losing. So yes: I’m too scared to try something like this publicly without a legal shield.

We could however work on something like this in a protected repo at GitHub etc. and go to TLP as soon as we have something.

Would that be an option?

Chris


From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:57
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey Chris,

wrt testing I meant how „sustainable” the community is, that we intend to building.
You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone.
And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means).
And this is something I think is hard to guarantee in the current state.

Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 12:48
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

AW: Building a PLC4X and IoTDB Historian?

[Julian Feinauer <j....@pragmaticindustries.de>]

Hey Chris,

wrt testing I meant how „sustainable” the community is, that we intend to building.
You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone.
And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means).
And this is something I think is hard to guarantee in the current state.

Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there.

Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 12:48
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi Julian,

Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other.
I wouldn’t have problems with doing it under the hood of IoTDB, personally.

A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”)

The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP.

Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian?
I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system.
Not 100% sure what you want to try out.

But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past.

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 12:37
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

AW: Building a PLC4X and IoTDB Historian?

[Julian Feinauer <j....@pragmaticindustries.de>]

Hey,

I understand your point and also agree generally.
But I personally dislike the straight to TLP route as this is something I’d like to try out first.
And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist).

But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject).
And the repo thing could be done by you alone I think as Chair.

Next steps would then be more of brainstorming and workshoping and building something (whatever?).
I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…).

Best
Julian

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 09:27
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi Julian,

happy you like the idea ;-)

We would probably outclass the established ones in point of price ;-)
Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products.
I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement.

Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route.

And I would really like to keep it at Apache and not set up something outside. This for multiple reasons:

  *   Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied)
  *   The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects)

Chris



From: Julian Feinauer <j....@pragmaticindustries.de>
Date: Monday, 28. November 2022 at 09:17
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Cc: Tim Mitsch <t....@pragmaticindustries.de>, hoeper@ltsoft.de <ho...@ltsoft.de>
Subject: AW: Building a PLC4X and IoTDB Historian?
Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Otto Fowler <ot...@gmail.com>]

have you used OSI PI?  "I think with the technology one has today it is
very easy to "outclass" these established systems e.g. in performance,
efficiency and als user friendliness."   It would not be easy to replicate
or outclass what they have done in decades of work.

On Mon, Nov 28, 2022 at 3:17 AM Julian Feinauer <
j.feinauer@pragmaticindustries.de> wrote:

> Hi all,
>
> first of all.. sorry for coming in so late.
> I think this is a great idea and there really is a product market fit for
> such a solution.
> I think technology wise its not "that complicated" to build such a system,
> I think many of the details that are necessary for a good adoption of such
> a software is in the areas around like documentation, marketing, more
> marketing, even more and even bolder marketing and sales (how to sell an
> open source project??).
>
> But technology wise I really like the idea and I think with the technology
> one has today it is very easy to "outclass" these established systems e.g.
> in performance, efficiency and als user friendliness.
> The only thing which might be challenging for the future are things like
> plugins or an extension system to allow users to customize their
> installation.
>
> Regarding the "separate project" approach I am not 100% certain.
> Personally, I would consider starting EITHER as a subproject in an
> existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere
> else and not directly go to the Incubator or something.
> Because I think we should really find out if there is enough developer
> interest to build and sustain such a system.
>
> But I'm totally in for such a system either way!
>
> Julian
>
> PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with
> historians
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 08:56
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Xiangdong,
>
> that’s perfect that some of you folks would be on board. I should probably
> check, if there are enough PLC4X folks on board too … would be a shame if
> it was just me ;-)
>
> And regarding the name … yeah … you are absolutely right: Historian might
> really be the ideal choice … as it instantly explains what it is … A
> Historian from the Apache Software Foundation and probably also no bias
> towards any regional culture.
>
> So (Speaking to my fellow PLC4X folks … who would be on board with this?)
>
>
> Chris
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Monday, 28. November 2022 at 04:24
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> > I guess I would sort of also vote for a separate Project. Would however
> only make sense if some people from both of our projects would join in.
>
> Agree, I think the iotdb developers in Timecho can join.
>
> > And name-wise … one of the names of Historic Historians (Ideally a Greek
> one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are
> “famous” Buttler names ;-)
>
> If we consider famous names (well, there will be culture bias), I'd
> like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
> "Records of the Grand Historian" is tooo famous (at least in China).
>
> BTW. naming "Apache Historian" directly may be another option.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>
> Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
> >
> > How about Apache Ephorous? ;-)
> >
> > https://en.wikipedia.org/wiki/Ephorus
> > Ephorus of Cyme (/ˈɛfərəs/<
> https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<
> https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros
> ho Kymaios; c. 400 – 330 BC) was an ancient Greek<
> https://en.wikipedia.org/wiki/Ancient_Greece> historian<
> https://en.wikipedia.org/wiki/Historian> known for his universal history<
> https://en.wikipedia.org/wiki/Universal_history>.
> >
> > Chris
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > Date: Sunday, 27. November 2022 at 15:28
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > I guess I would sort of also vote for a separate Project. Would however
> only make sense if some people from both of our projects would join in.
> >
> > And name-wise … one of the names of Historic Historians (Ideally a Greek
> one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are
> “famous” Buttler names ;-)
> > https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> > https://en.wikipedia.org/wiki/List_of_historians
> >
> > I think anything but a turnkey-ready solution will not be accepted by
> anyone in the Automation industry.
> >
> > Chris
> >
> >
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Sunday, 27. November 2022 at 14:05
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > If we just provide a solution, or demonstration, then either is ok.
> > If we want to provide an "one-box thing" (even without GUI), +1 for a
> > new project.
> >
> > > I don’t even think it would be bad if an entity like Timecho would add
> enterprise offerings, because I know, that the industry won’t “buy”
> something, if there’s no commercial support or anyone, they can throw money
> at, even if it’s free (That might even make them more skeptical).
> > Yes, indeed. :D
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> >
> > Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> > >
> > > Hi,
> > >
> > > I think it's a great idea.
> > >
> > > I'd vote for having a separate project for something like this, just so
> > > it's clear that it is a historian and people can build it separately. I
> > > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> > >
> > > Cake does go with Cafe.
> > >
> > > Ben
> > >
> > >
> > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <
> christofer.dutz@c-ware.de>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Trying to respond to all (except the naming question ;-) ):
> > > >
> > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we
> could be
> > > > collecting data on one node, writing that to TSFiles using the
> IoTDB-Client
> > > > (As Xaingdong mentioned).
> > > > These could either be online and transfer data to the server right
> away,
> > > > or operate in an offline-like mode and periodically fetch data in
> bursts
> > > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > > >
> > > > @Xaiangdong: In general, PLC4X operates in an active mode, but we
> also
> > > > have a proposed and prototypically proven “passive-mode”. Here the
> software
> > > > doesn’t actively participate in the data acquisition directly, but
> simply
> > > > listens to the data stream and makes data available. We also have
> something
> > > > we call a “Data-Diode”, which technically makes interference
> impossible as
> > > > it lets ethernet packets flow in one direction, but nothing in the
> other
> > > > (Firewalls usually have the problem of letting packets pass in both
> > > > directions for established connections). We invented this, because
> we knew
> > > > we will never have validated and audited open-source software, to a
> level
> > > > that it would be certified for some of these use cases.
> > > >
> > > > So, either we can live with everything the SCADA system is already
> > > > requesting, or we add an active PLC4X node in the secure network,
> that
> > > > requests data, and simply ignores it, and a second – passive mode –
> node
> > > > sits outside the secure network to capture the information.
> > > >
> > > > I agree that this sort of thing needs to be a “product”. The
> Automation
> > > > Industry just doesn’t know how to work with frameworks. Ideally a
> one-box
> > > > thing.
> > > > Right now, most tools I have seen don’t even need the calculation or
> the
> > > > visualization. This is usually done on another level. Important
> would be
> > > > that we could be somewhat API compatible with existing products. Them
> > > > usually having SQL or REST APIs, should make it relatively easy to
> sort of
> > > > produce frontends to our Apache Historian, that are API compatible
> with
> > > > some existing industry products, so they are replaceable.
> > > >
> > > > If we wanted to add a visualization layer or a data-curation layer
> later
> > > > on, we should have a chat with the NiFi or StreamPipes folks as
> that’s what
> > > > they already have.
> > > >
> > > > For now, I would be proposing to build something that uses PLC4X for
> the
> > > > data-acquisition, IoTDB for the storage and build a REST frontend
> for this,
> > > > that’s somewhat API compatible with one of the major established
> products
> > > > and to add more on a step-by-step basis.
> > > >
> > > > I don’t even think it would be bad if an entity like Timecho would
> add
> > > > enterprise offerings, because I know, that the industry won’t “buy”
> > > > something, if there’s no commercial support or anyone, they can
> throw money
> > > > at, even if it’s free (That might even make them more skeptical).
> > > >
> > > >
> > > > What do you all think? Does this make sense? If yes, where would it
> make
> > > > sense to start working on something like this? In the IoTDB project?
> In the
> > > > PLC4X project, in a separate (new) project?
> > > >
> > > >
> > > > Chris
> > > >
> > > >
> > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > Date: Sunday, 27. November 2022 at 06:02
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > Hi all,
> > > >
> > > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > > Actually, there are several users using IoTDB as their historian
> > > > solution...
> > > >
> > > > According to my knowledge, a historian software contains several
> features:
> > > >
> > > > - get data from OPC/modubs server and write to IoTDB
> > > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > > >
> > > > - (optional) there is a single-way network gateway for security.
> > > >   * If the gateway is deployed between device and PLC4x program, then
> > > > we need to check whether plc4x supports that.
> > > >   * If the gateway is deployed between IoTDB-client and IoTDB, then
> we
> > > > need to check wheter IoTDB-client supports that.
> > > >
> > > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > > collect A and B from a device (for example, C=A+B/2 )
> > > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > > programming GUI. (maybe integrating some open source projects, like
> > > > Red-node is a good idea)
> > > >
> > > > - get the latest data from IoTDB for visualization (usually users
> need
> > > > to draw a process flow diagram, and put the latest data on each
> device
> > > > icon on the diagram)
> > > >   * IoTDB supports getting latest data. but we need a new software
> for
> > > > drawing the diagram (I do not know if there is any open source
> project
> > > > for this. In our real applications, we ususally buy and integrate
> some
> > > > other commerical software)
> > > >
> > > > - send alert message if the real time data meets some rules
> > > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > > (IoTDB also supports Prometheus Alert Manager)
> > > >
> > > > - get the historical data
> > > >   * IoTDB supports that and grafana is good at visualization
> > > >
> > > > - last but not the least, if we provide all the features to
> industrial
> > > > users,  all the GUIs should be integrated into one entrance.
> > > >
> > > >
> > > > Best,
> > > > -----------------------------------
> > > > Xiangdong Huang
> > > > School of Software, Tsinghua University
> > > >
> > > >  黄向东
> > > > 清华大学 软件学院
> > > >
> > > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > > >
> > > > > Hi folks, long time lurker…
> > > > >
> > > > > A historian would be great. Does PLC4X queue data on the machine
> it’s
> > > > > running on already?
> > > > >
> > > > > A common pattern in historians is to queue data on the runtime
> server
> > > > until
> > > > > data can be extracted to a sql server, preventing data loss, and
> reducing
> > > > > the need for redundancy.
> > > > >
> > > > > There are a ton of edge cases to watch out for which I can
> elaborate on
> > > > > further.
> > > > >
> > > > > I’m a former Industrial Controls Engineer turned software dev and
> can
> > > > help
> > > > > out where needed.
> > > > >
> > > > > -Ryan
> > > > >
> > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <
> ottobackwards@gmail.com>
> > > > > wrote:
> > > > >
> > > > > >  Would you name it “Cake”?
> > > > > >
> > > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > > <ch...@c-ware.de>
> > > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > > dev@plc4x.apache.org>
> > > > > > Date: November 26, 2022 at 10:36:23
> > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> dev@plc4x.apache.org
> > > > >,
> > > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <
> dev@iotdb.apache.org>
> > > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > > something
> > > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > > >
> > > > > > For those of you, who don’t know what a Historian is. In the
> > > > manufacturing
> > > > > > industry they have these extremely expensive servers (usually
> you buy
> > > > them
> > > > > > as a bundle of hard- and software).
> > > > > > These servers are nothing else than a really crappy and brutally
> > > > expensive
> > > > > > Database for storing Time-Series data.
> > > > > >
> > > > > > The thing is most commercial products are currently really having
> > > > trouble
> > > > > > to keep up with the increasing amount of Data being sent.
> > > > > >
> > > > > > So instead of suggesting building an Historian at each customer’s
> > > > site, I
> > > > > > thought: Perhaps a ready-to-use solution based on open-source
> would be
> > > > a
> > > > > > good idea.
> > > > > > Not 100% sure where I’d locate such an initiative, but I would
> tend to
> > > > see
> > > > > > it more on the Database side.
> > > > > >
> > > > > > I think IoTDB would be the perfect storage system, all we seem
> to need
> > > > is
> > > > > > some sort of rest-interface that matches the industry standards
> for
> > > > > > querying the information and on the other side something like
> PLC4X to
> > > > fill
> > > > > > the database.
> > > > > >
> > > > > > Here some examples:
> > > > > >
> > > > > >
> > > >
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > > >
> > > >
> https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > > >
> > > > > > What do you folks think?
> > > > > >
> > > > > >
> > > > > > Chris
> > > > > >
> > > >
>

AW: Building a PLC4X and IoTDB Historian?

[Julian Feinauer <j....@pragmaticindustries.de>]

Hi all,

first of all.. sorry for coming in so late.
I think this is a great idea and there really is a product market fit for such a solution.
I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??).

But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness.
The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation.

Regarding the "separate project" approach I am not 100% certain.
Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something.
Because I think we should really find out if there is enough developer interest to build and sustain such a system.

But I'm totally in for such a system either way!

Julian

PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians

Von: Christofer Dutz <ch...@c-ware.de>
Datum: Montag, 28. November 2022 um 08:56
An: dev@plc4x.apache.org <de...@plc4x.apache.org>
Betreff: Re: Building a PLC4X and IoTDB Historian?
Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi Xiangdong,

that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-)

And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture.

So (Speaking to my fellow PLC4X folks … who would be on board with this?)


Chris



From: Xiangdong Huang <sa...@gmail.com>
Date: Monday, 28. November 2022 at 04:24
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Xiangdong Huang <sa...@gmail.com>]

> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

Agree, I think the iotdb developers in Timecho can join.

> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)

If we consider famous names (well, there will be culture bias), I'd
like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
"Records of the Grand Historian" is tooo famous (at least in China).

BTW. naming "Apache Historian" directly may be another option.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University


Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
>
> How about Apache Ephorous? ;-)
>
> https://en.wikipedia.org/wiki/Ephorus
> Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.
>
> Chris
>
> From: Christofer Dutz <ch...@c-ware.de>
> Date: Sunday, 27. November 2022 at 15:28
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.
>
> And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
> https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> https://en.wikipedia.org/wiki/List_of_historians
>
> I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.
>
> Chris
>
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 14:05
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> If we just provide a solution, or demonstration, then either is ok.
> If we want to provide an "one-box thing" (even without GUI), +1 for a
> new project.
>
> > I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
> Yes, indeed. :D
>
> Best,
> -----------------------------------
> Xiangdong Huang
>
> Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> >
> > Hi,
> >
> > I think it's a great idea.
> >
> > I'd vote for having a separate project for something like this, just so
> > it's clear that it is a historian and people can build it separately. I
> > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> >
> > Cake does go with Cafe.
> >
> > Ben
> >
> >
> > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> > wrote:
> >
> > > Hi all,
> > >
> > > Trying to respond to all (except the naming question ;-) ):
> > >
> > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > > (As Xaingdong mentioned).
> > > These could either be online and transfer data to the server right away,
> > > or operate in an offline-like mode and periodically fetch data in bursts
> > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > >
> > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > > have a proposed and prototypically proven “passive-mode”. Here the software
> > > doesn’t actively participate in the data acquisition directly, but simply
> > > listens to the data stream and makes data available. We also have something
> > > we call a “Data-Diode”, which technically makes interference impossible as
> > > it lets ethernet packets flow in one direction, but nothing in the other
> > > (Firewalls usually have the problem of letting packets pass in both
> > > directions for established connections). We invented this, because we knew
> > > we will never have validated and audited open-source software, to a level
> > > that it would be certified for some of these use cases.
> > >
> > > So, either we can live with everything the SCADA system is already
> > > requesting, or we add an active PLC4X node in the secure network, that
> > > requests data, and simply ignores it, and a second – passive mode – node
> > > sits outside the secure network to capture the information.
> > >
> > > I agree that this sort of thing needs to be a “product”. The Automation
> > > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > > thing.
> > > Right now, most tools I have seen don’t even need the calculation or the
> > > visualization. This is usually done on another level. Important would be
> > > that we could be somewhat API compatible with existing products. Them
> > > usually having SQL or REST APIs, should make it relatively easy to sort of
> > > produce frontends to our Apache Historian, that are API compatible with
> > > some existing industry products, so they are replaceable.
> > >
> > > If we wanted to add a visualization layer or a data-curation layer later
> > > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > > they already have.
> > >
> > > For now, I would be proposing to build something that uses PLC4X for the
> > > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > > that’s somewhat API compatible with one of the major established products
> > > and to add more on a step-by-step basis.
> > >
> > > I don’t even think it would be bad if an entity like Timecho would add
> > > enterprise offerings, because I know, that the industry won’t “buy”
> > > something, if there’s no commercial support or anyone, they can throw money
> > > at, even if it’s free (That might even make them more skeptical).
> > >
> > >
> > > What do you all think? Does this make sense? If yes, where would it make
> > > sense to start working on something like this? In the IoTDB project? In the
> > > PLC4X project, in a separate (new) project?
> > >
> > >
> > > Chris
> > >
> > >
> > > From: Xiangdong Huang <sa...@gmail.com>
> > > Date: Sunday, 27. November 2022 at 06:02
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > Hi all,
> > >
> > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > Actually, there are several users using IoTDB as their historian
> > > solution...
> > >
> > > According to my knowledge, a historian software contains several features:
> > >
> > > - get data from OPC/modubs server and write to IoTDB
> > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > >
> > > - (optional) there is a single-way network gateway for security.
> > >   * If the gateway is deployed between device and PLC4x program, then
> > > we need to check whether plc4x supports that.
> > >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > > need to check wheter IoTDB-client supports that.
> > >
> > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > collect A and B from a device (for example, C=A+B/2 )
> > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > programming GUI. (maybe integrating some open source projects, like
> > > Red-node is a good idea)
> > >
> > > - get the latest data from IoTDB for visualization (usually users need
> > > to draw a process flow diagram, and put the latest data on each device
> > > icon on the diagram)
> > >   * IoTDB supports getting latest data. but we need a new software for
> > > drawing the diagram (I do not know if there is any open source project
> > > for this. In our real applications, we ususally buy and integrate some
> > > other commerical software)
> > >
> > > - send alert message if the real time data meets some rules
> > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > (IoTDB also supports Prometheus Alert Manager)
> > >
> > > - get the historical data
> > >   * IoTDB supports that and grafana is good at visualization
> > >
> > > - last but not the least, if we provide all the features to industrial
> > > users,  all the GUIs should be integrated into one entrance.
> > >
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > >
> > > > Hi folks, long time lurker…
> > > >
> > > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > > running on already?
> > > >
> > > > A common pattern in historians is to queue data on the runtime server
> > > until
> > > > data can be extracted to a sql server, preventing data loss, and reducing
> > > > the need for redundancy.
> > > >
> > > > There are a ton of edge cases to watch out for which I can elaborate on
> > > > further.
> > > >
> > > > I’m a former Industrial Controls Engineer turned software dev and can
> > > help
> > > > out where needed.
> > > >
> > > > -Ryan
> > > >
> > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Would you name it “Cake”?
> > > > >
> > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > <ch...@c-ware.de>
> > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > dev@plc4x.apache.org>
> > > > > Date: November 26, 2022 at 10:36:23
> > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > > >,
> > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > something
> > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > >
> > > > > For those of you, who don’t know what a Historian is. In the
> > > manufacturing
> > > > > industry they have these extremely expensive servers (usually you buy
> > > them
> > > > > as a bundle of hard- and software).
> > > > > These servers are nothing else than a really crappy and brutally
> > > expensive
> > > > > Database for storing Time-Series data.
> > > > >
> > > > > The thing is most commercial products are currently really having
> > > trouble
> > > > > to keep up with the increasing amount of Data being sent.
> > > > >
> > > > > So instead of suggesting building an Historian at each customer’s
> > > site, I
> > > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > > a
> > > > > good idea.
> > > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > > see
> > > > > it more on the Database side.
> > > > >
> > > > > I think IoTDB would be the perfect storage system, all we seem to need
> > > is
> > > > > some sort of rest-interface that matches the industry standards for
> > > > > querying the information and on the other side something like PLC4X to
> > > fill
> > > > > the database.
> > > > >
> > > > > Here some examples:
> > > > >
> > > > >
> > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > >
> > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > >
> > > > > What do you folks think?
> > > > >
> > > > >
> > > > > Chris
> > > > >
> > >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

How about Apache Ephorous? ;-)

https://en.wikipedia.org/wiki/Ephorus
Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient Greek<https://en.wikipedia.org/wiki/Ancient_Greece> historian<https://en.wikipedia.org/wiki/Historian> known for his universal history<https://en.wikipedia.org/wiki/Universal_history>.

Chris

From: Christofer Dutz <ch...@c-ware.de>
Date: Sunday, 27. November 2022 at 15:28
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
Hi all,

I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

And name-wise … one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are “famous” Buttler names ;-)
https://en.wikipedia.org/wiki/List_of_Greek_historiographers
https://en.wikipedia.org/wiki/List_of_historians

I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.

Chris




From: Xiangdong Huang <sa...@gmail.com>
Date: Sunday, 27. November 2022 at 14:05
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
If we just provide a solution, or demonstration, then either is ok.
If we want to provide an "one-box thing" (even without GUI), +1 for a
new project.

> I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
Yes, indeed. :D

Best,
-----------------------------------
Xiangdong Huang

Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
>
> Hi,
>
> I think it's a great idea.
>
> I'd vote for having a separate project for something like this, just so
> it's clear that it is a historian and people can build it separately. I
> wouldn't want it hidden away in an obscure folder in the PLC4X repo.
>
> Cake does go with Cafe.
>
> Ben
>
>
> On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> wrote:
>
> > Hi all,
> >
> > Trying to respond to all (except the naming question ;-) ):
> >
> > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > (As Xaingdong mentioned).
> > These could either be online and transfer data to the server right away,
> > or operate in an offline-like mode and periodically fetch data in bursts
> > from the client (Hope that’s correct, please correct me if I’m wrong)
> >
> > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > have a proposed and prototypically proven “passive-mode”. Here the software
> > doesn’t actively participate in the data acquisition directly, but simply
> > listens to the data stream and makes data available. We also have something
> > we call a “Data-Diode”, which technically makes interference impossible as
> > it lets ethernet packets flow in one direction, but nothing in the other
> > (Firewalls usually have the problem of letting packets pass in both
> > directions for established connections). We invented this, because we knew
> > we will never have validated and audited open-source software, to a level
> > that it would be certified for some of these use cases.
> >
> > So, either we can live with everything the SCADA system is already
> > requesting, or we add an active PLC4X node in the secure network, that
> > requests data, and simply ignores it, and a second – passive mode – node
> > sits outside the secure network to capture the information.
> >
> > I agree that this sort of thing needs to be a “product”. The Automation
> > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > thing.
> > Right now, most tools I have seen don’t even need the calculation or the
> > visualization. This is usually done on another level. Important would be
> > that we could be somewhat API compatible with existing products. Them
> > usually having SQL or REST APIs, should make it relatively easy to sort of
> > produce frontends to our Apache Historian, that are API compatible with
> > some existing industry products, so they are replaceable.
> >
> > If we wanted to add a visualization layer or a data-curation layer later
> > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > they already have.
> >
> > For now, I would be proposing to build something that uses PLC4X for the
> > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > that’s somewhat API compatible with one of the major established products
> > and to add more on a step-by-step basis.
> >
> > I don’t even think it would be bad if an entity like Timecho would add
> > enterprise offerings, because I know, that the industry won’t “buy”
> > something, if there’s no commercial support or anyone, they can throw money
> > at, even if it’s free (That might even make them more skeptical).
> >
> >
> > What do you all think? Does this make sense? If yes, where would it make
> > sense to start working on something like this? In the IoTDB project? In the
> > PLC4X project, in a separate (new) project?
> >
> >
> > Chris
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Sunday, 27. November 2022 at 06:02
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > It makes sense to use PLC4X+IoTDB to build a historian software.
> > Actually, there are several users using IoTDB as their historian
> > solution...
> >
> > According to my knowledge, a historian software contains several features:
> >
> > - get data from OPC/modubs server and write to IoTDB
> >   * use PLC4x + IoTDB-client. (we call this data collector)
> >
> > - (optional) there is a single-way network gateway for security.
> >   * If the gateway is deployed between device and PLC4x program, then
> > we need to check whether plc4x supports that.
> >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > need to check wheter IoTDB-client supports that.
> >
> > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > collect A and B from a device (for example, C=A+B/2 )
> >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > programming GUI. (maybe integrating some open source projects, like
> > Red-node is a good idea)
> >
> > - get the latest data from IoTDB for visualization (usually users need
> > to draw a process flow diagram, and put the latest data on each device
> > icon on the diagram)
> >   * IoTDB supports getting latest data. but we need a new software for
> > drawing the diagram (I do not know if there is any open source project
> > for this. In our real applications, we ususally buy and integrate some
> > other commerical software)
> >
> > - send alert message if the real time data meets some rules
> >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > (IoTDB also supports Prometheus Alert Manager)
> >
> > - get the historical data
> >   * IoTDB supports that and grafana is good at visualization
> >
> > - last but not the least, if we provide all the features to industrial
> > users,  all the GUIs should be integrated into one entrance.
> >
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
> >
> > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > >
> > > Hi folks, long time lurker…
> > >
> > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > running on already?
> > >
> > > A common pattern in historians is to queue data on the runtime server
> > until
> > > data can be extracted to a sql server, preventing data loss, and reducing
> > > the need for redundancy.
> > >
> > > There are a ton of edge cases to watch out for which I can elaborate on
> > > further.
> > >
> > > I’m a former Industrial Controls Engineer turned software dev and can
> > help
> > > out where needed.
> > >
> > > -Ryan
> > >
> > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > >
> > > >  Would you name it “Cake”?
> > > >
> > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > <ch...@c-ware.de>
> > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > dev@plc4x.apache.org>
> > > > Date: November 26, 2022 at 10:36:23
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > >,
> > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > >
> > > > Hi all,
> > > >
> > > > I am currently thinking, if it wouldn’t be a good idea to build
> > something
> > > > like an open-source Historian based on PLC4X and IoTDB.
> > > >
> > > > For those of you, who don’t know what a Historian is. In the
> > manufacturing
> > > > industry they have these extremely expensive servers (usually you buy
> > them
> > > > as a bundle of hard- and software).
> > > > These servers are nothing else than a really crappy and brutally
> > expensive
> > > > Database for storing Time-Series data.
> > > >
> > > > The thing is most commercial products are currently really having
> > trouble
> > > > to keep up with the increasing amount of Data being sent.
> > > >
> > > > So instead of suggesting building an Historian at each customer’s
> > site, I
> > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > a
> > > > good idea.
> > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > see
> > > > it more on the Database side.
> > > >
> > > > I think IoTDB would be the perfect storage system, all we seem to need
> > is
> > > > some sort of rest-interface that matches the industry standards for
> > > > querying the information and on the other side something like PLC4X to
> > fill
> > > > the database.
> > > >
> > > > Here some examples:
> > > >
> > > >
> > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > >
> > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > >
> > > > What do you folks think?
> > > >
> > > >
> > > > Chris
> > > >
> >

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi all,

I guess I would sort of also vote for a separate Project. Would however only make sense if some people from both of our projects would join in.

And name-wise �� one of the names of Historic Historians (Ideally a Greek one) would make it into my top 10 ;-) �� sort of like Hudson or Jenkins are ��famous�� Buttler names ;-)
https://en.wikipedia.org/wiki/List_of_Greek_historiographers
https://en.wikipedia.org/wiki/List_of_historians

I think anything but a turnkey-ready solution will not be accepted by anyone in the Automation industry.

Chris




From: Xiangdong Huang <sa...@gmail.com>
Date: Sunday, 27. November 2022 at 14:05
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
If we just provide a solution, or demonstration, then either is ok.
If we want to provide an "one-box thing" (even without GUI), +1 for a
new project.

> I don��t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won��t ��buy�� something, if there��s no commercial support or anyone, they can throw money at, even if it��s free (That might even make them more skeptical).
Yes, indeed. :D

Best,
-----------------------------------
Xiangdong Huang

Ben Hutcheson <be...@gmail.com> ��2022��11��27������ 18:38���
>
> Hi,
>
> I think it's a great idea.
>
> I'd vote for having a separate project for something like this, just so
> it's clear that it is a historian and people can build it separately. I
> wouldn't want it hidden away in an obscure folder in the PLC4X repo.
>
> Cake does go with Cafe.
>
> Ben
>
>
> On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> wrote:
>
> > Hi all,
> >
> > Trying to respond to all (except the naming question ;-) ):
> >
> > @Ryan: PLC4X Doesn��t have such a queueing, but IoTDB has �� so we could be
> > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > (As Xaingdong mentioned).
> > These could either be online and transfer data to the server right away,
> > or operate in an offline-like mode and periodically fetch data in bursts
> > from the client (Hope that��s correct, please correct me if I��m wrong)
> >
> > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > have a proposed and prototypically proven ��passive-mode��. Here the software
> > doesn��t actively participate in the data acquisition directly, but simply
> > listens to the data stream and makes data available. We also have something
> > we call a ��Data-Diode��, which technically makes interference impossible as
> > it lets ethernet packets flow in one direction, but nothing in the other
> > (Firewalls usually have the problem of letting packets pass in both
> > directions for established connections). We invented this, because we knew
> > we will never have validated and audited open-source software, to a level
> > that it would be certified for some of these use cases.
> >
> > So, either we can live with everything the SCADA system is already
> > requesting, or we add an active PLC4X node in the secure network, that
> > requests data, and simply ignores it, and a second �C passive mode �C node
> > sits outside the secure network to capture the information.
> >
> > I agree that this sort of thing needs to be a ��product��. The Automation
> > Industry just doesn��t know how to work with frameworks. Ideally a one-box
> > thing.
> > Right now, most tools I have seen don��t even need the calculation or the
> > visualization. This is usually done on another level. Important would be
> > that we could be somewhat API compatible with existing products. Them
> > usually having SQL or REST APIs, should make it relatively easy to sort of
> > produce frontends to our Apache Historian, that are API compatible with
> > some existing industry products, so they are replaceable.
> >
> > If we wanted to add a visualization layer or a data-curation layer later
> > on, we should have a chat with the NiFi or StreamPipes folks as that��s what
> > they already have.
> >
> > For now, I would be proposing to build something that uses PLC4X for the
> > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > that��s somewhat API compatible with one of the major established products
> > and to add more on a step-by-step basis.
> >
> > I don��t even think it would be bad if an entity like Timecho would add
> > enterprise offerings, because I know, that the industry won��t ��buy��
> > something, if there��s no commercial support or anyone, they can throw money
> > at, even if it��s free (That might even make them more skeptical).
> >
> >
> > What do you all think? Does this make sense? If yes, where would it make
> > sense to start working on something like this? In the IoTDB project? In the
> > PLC4X project, in a separate (new) project?
> >
> >
> > Chris
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Sunday, 27. November 2022 at 06:02
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > It makes sense to use PLC4X+IoTDB to build a historian software.
> > Actually, there are several users using IoTDB as their historian
> > solution...
> >
> > According to my knowledge, a historian software contains several features:
> >
> > - get data from OPC/modubs server and write to IoTDB
> >   * use PLC4x + IoTDB-client. (we call this data collector)
> >
> > - (optional) there is a single-way network gateway for security.
> >   * If the gateway is deployed between device and PLC4x program, then
> > we need to check whether plc4x supports that.
> >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > need to check wheter IoTDB-client supports that.
> >
> > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > collect A and B from a device (for example, C=A+B/2 )
> >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > programming GUI. (maybe integrating some open source projects, like
> > Red-node is a good idea)
> >
> > - get the latest data from IoTDB for visualization (usually users need
> > to draw a process flow diagram, and put the latest data on each device
> > icon on the diagram)
> >   * IoTDB supports getting latest data. but we need a new software for
> > drawing the diagram (I do not know if there is any open source project
> > for this. In our real applications, we ususally buy and integrate some
> > other commerical software)
> >
> > - send alert message if the real time data meets some rules
> >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > (IoTDB also supports Prometheus Alert Manager)
> >
> > - get the historical data
> >   * IoTDB supports that and grafana is good at visualization
> >
> > - last but not the least, if we provide all the features to industrial
> > users,  all the GUIs should be integrated into one entrance.
> >
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  ����
> > �廪��ѧ ����ѧԺ
> >
> > Ryan Truran <ry...@gmail.com> ��2022��11��27������ 08:18���
> > >
> > > Hi folks, long time lurker��
> > >
> > > A historian would be great. Does PLC4X queue data on the machine it��s
> > > running on already?
> > >
> > > A common pattern in historians is to queue data on the runtime server
> > until
> > > data can be extracted to a sql server, preventing data loss, and reducing
> > > the need for redundancy.
> > >
> > > There are a ton of edge cases to watch out for which I can elaborate on
> > > further.
> > >
> > > I��m a former Industrial Controls Engineer turned software dev and can
> > help
> > > out where needed.
> > >
> > > -Ryan
> > >
> > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > >
> > > >  Would you name it ��Cake��?
> > > >
> > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > <ch...@c-ware.de>
> > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > dev@plc4x.apache.org>
> > > > Date: November 26, 2022 at 10:36:23
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > >,
> > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > >
> > > > Hi all,
> > > >
> > > > I am currently thinking, if it wouldn��t be a good idea to build
> > something
> > > > like an open-source Historian based on PLC4X and IoTDB.
> > > >
> > > > For those of you, who don��t know what a Historian is. In the
> > manufacturing
> > > > industry they have these extremely expensive servers (usually you buy
> > them
> > > > as a bundle of hard- and software).
> > > > These servers are nothing else than a really crappy and brutally
> > expensive
> > > > Database for storing Time-Series data.
> > > >
> > > > The thing is most commercial products are currently really having
> > trouble
> > > > to keep up with the increasing amount of Data being sent.
> > > >
> > > > So instead of suggesting building an Historian at each customer��s
> > site, I
> > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > a
> > > > good idea.
> > > > Not 100% sure where I��d locate such an initiative, but I would tend to
> > see
> > > > it more on the Database side.
> > > >
> > > > I think IoTDB would be the perfect storage system, all we seem to need
> > is
> > > > some sort of rest-interface that matches the industry standards for
> > > > querying the information and on the other side something like PLC4X to
> > fill
> > > > the database.
> > > >
> > > > Here some examples:
> > > >
> > > >
> > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > >
> > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > >
> > > > What do you folks think?
> > > >
> > > >
> > > > Chris
> > > >
> >

Re: Building a PLC4X and IoTDB Historian?

[Xiangdong Huang <sa...@gmail.com>]

If we just provide a solution, or demonstration, then either is ok.
If we want to provide an "one-box thing" (even without GUI), +1 for a
new project.

> I don’t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won’t “buy” something, if there’s no commercial support or anyone, they can throw money at, even if it’s free (That might even make them more skeptical).
Yes, indeed. :D

Best,
-----------------------------------
Xiangdong Huang

Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
>
> Hi,
>
> I think it's a great idea.
>
> I'd vote for having a separate project for something like this, just so
> it's clear that it is a historian and people can build it separately. I
> wouldn't want it hidden away in an obscure folder in the PLC4X repo.
>
> Cake does go with Cafe.
>
> Ben
>
>
> On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
> wrote:
>
> > Hi all,
> >
> > Trying to respond to all (except the naming question ;-) ):
> >
> > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> > collecting data on one node, writing that to TSFiles using the IoTDB-Client
> > (As Xaingdong mentioned).
> > These could either be online and transfer data to the server right away,
> > or operate in an offline-like mode and periodically fetch data in bursts
> > from the client (Hope that’s correct, please correct me if I’m wrong)
> >
> > @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> > have a proposed and prototypically proven “passive-mode”. Here the software
> > doesn’t actively participate in the data acquisition directly, but simply
> > listens to the data stream and makes data available. We also have something
> > we call a “Data-Diode”, which technically makes interference impossible as
> > it lets ethernet packets flow in one direction, but nothing in the other
> > (Firewalls usually have the problem of letting packets pass in both
> > directions for established connections). We invented this, because we knew
> > we will never have validated and audited open-source software, to a level
> > that it would be certified for some of these use cases.
> >
> > So, either we can live with everything the SCADA system is already
> > requesting, or we add an active PLC4X node in the secure network, that
> > requests data, and simply ignores it, and a second – passive mode – node
> > sits outside the secure network to capture the information.
> >
> > I agree that this sort of thing needs to be a “product”. The Automation
> > Industry just doesn’t know how to work with frameworks. Ideally a one-box
> > thing.
> > Right now, most tools I have seen don’t even need the calculation or the
> > visualization. This is usually done on another level. Important would be
> > that we could be somewhat API compatible with existing products. Them
> > usually having SQL or REST APIs, should make it relatively easy to sort of
> > produce frontends to our Apache Historian, that are API compatible with
> > some existing industry products, so they are replaceable.
> >
> > If we wanted to add a visualization layer or a data-curation layer later
> > on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> > they already have.
> >
> > For now, I would be proposing to build something that uses PLC4X for the
> > data-acquisition, IoTDB for the storage and build a REST frontend for this,
> > that’s somewhat API compatible with one of the major established products
> > and to add more on a step-by-step basis.
> >
> > I don’t even think it would be bad if an entity like Timecho would add
> > enterprise offerings, because I know, that the industry won’t “buy”
> > something, if there’s no commercial support or anyone, they can throw money
> > at, even if it’s free (That might even make them more skeptical).
> >
> >
> > What do you all think? Does this make sense? If yes, where would it make
> > sense to start working on something like this? In the IoTDB project? In the
> > PLC4X project, in a separate (new) project?
> >
> >
> > Chris
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Sunday, 27. November 2022 at 06:02
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > It makes sense to use PLC4X+IoTDB to build a historian software.
> > Actually, there are several users using IoTDB as their historian
> > solution...
> >
> > According to my knowledge, a historian software contains several features:
> >
> > - get data from OPC/modubs server and write to IoTDB
> >   * use PLC4x + IoTDB-client. (we call this data collector)
> >
> > - (optional) there is a single-way network gateway for security.
> >   * If the gateway is deployed between device and PLC4x program, then
> > we need to check whether plc4x supports that.
> >   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> > need to check wheter IoTDB-client supports that.
> >
> > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > collect A and B from a device (for example, C=A+B/2 )
> >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > programming GUI. (maybe integrating some open source projects, like
> > Red-node is a good idea)
> >
> > - get the latest data from IoTDB for visualization (usually users need
> > to draw a process flow diagram, and put the latest data on each device
> > icon on the diagram)
> >   * IoTDB supports getting latest data. but we need a new software for
> > drawing the diagram (I do not know if there is any open source project
> > for this. In our real applications, we ususally buy and integrate some
> > other commerical software)
> >
> > - send alert message if the real time data meets some rules
> >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > (IoTDB also supports Prometheus Alert Manager)
> >
> > - get the historical data
> >   * IoTDB supports that and grafana is good at visualization
> >
> > - last but not the least, if we provide all the features to industrial
> > users,  all the GUIs should be integrated into one entrance.
> >
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
> >
> > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > >
> > > Hi folks, long time lurker…
> > >
> > > A historian would be great. Does PLC4X queue data on the machine it’s
> > > running on already?
> > >
> > > A common pattern in historians is to queue data on the runtime server
> > until
> > > data can be extracted to a sql server, preventing data loss, and reducing
> > > the need for redundancy.
> > >
> > > There are a ton of edge cases to watch out for which I can elaborate on
> > > further.
> > >
> > > I’m a former Industrial Controls Engineer turned software dev and can
> > help
> > > out where needed.
> > >
> > > -Ryan
> > >
> > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > >
> > > >  Would you name it “Cake”?
> > > >
> > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > <ch...@c-ware.de>
> > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > dev@plc4x.apache.org>
> > > > Date: November 26, 2022 at 10:36:23
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> > >,
> > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > >
> > > > Hi all,
> > > >
> > > > I am currently thinking, if it wouldn’t be a good idea to build
> > something
> > > > like an open-source Historian based on PLC4X and IoTDB.
> > > >
> > > > For those of you, who don’t know what a Historian is. In the
> > manufacturing
> > > > industry they have these extremely expensive servers (usually you buy
> > them
> > > > as a bundle of hard- and software).
> > > > These servers are nothing else than a really crappy and brutally
> > expensive
> > > > Database for storing Time-Series data.
> > > >
> > > > The thing is most commercial products are currently really having
> > trouble
> > > > to keep up with the increasing amount of Data being sent.
> > > >
> > > > So instead of suggesting building an Historian at each customer’s
> > site, I
> > > > thought: Perhaps a ready-to-use solution based on open-source would be
> > a
> > > > good idea.
> > > > Not 100% sure where I’d locate such an initiative, but I would tend to
> > see
> > > > it more on the Database side.
> > > >
> > > > I think IoTDB would be the perfect storage system, all we seem to need
> > is
> > > > some sort of rest-interface that matches the industry standards for
> > > > querying the information and on the other side something like PLC4X to
> > fill
> > > > the database.
> > > >
> > > > Here some examples:
> > > >
> > > >
> > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > >
> > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > >
> > > > What do you folks think?
> > > >
> > > >
> > > > Chris
> > > >
> >

Re: Building a PLC4X and IoTDB Historian?

[Ben Hutcheson <be...@gmail.com>]

Hi,

I think it's a great idea.

I'd vote for having a separate project for something like this, just so
it's clear that it is a historian and people can build it separately. I
wouldn't want it hidden away in an obscure folder in the PLC4X repo.

Cake does go with Cafe.

Ben


On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <ch...@c-ware.de>
wrote:

> Hi all,
>
> Trying to respond to all (except the naming question ;-) ):
>
> @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be
> collecting data on one node, writing that to TSFiles using the IoTDB-Client
> (As Xaingdong mentioned).
> These could either be online and transfer data to the server right away,
> or operate in an offline-like mode and periodically fetch data in bursts
> from the client (Hope that’s correct, please correct me if I’m wrong)
>
> @Xaiangdong: In general, PLC4X operates in an active mode, but we also
> have a proposed and prototypically proven “passive-mode”. Here the software
> doesn’t actively participate in the data acquisition directly, but simply
> listens to the data stream and makes data available. We also have something
> we call a “Data-Diode”, which technically makes interference impossible as
> it lets ethernet packets flow in one direction, but nothing in the other
> (Firewalls usually have the problem of letting packets pass in both
> directions for established connections). We invented this, because we knew
> we will never have validated and audited open-source software, to a level
> that it would be certified for some of these use cases.
>
> So, either we can live with everything the SCADA system is already
> requesting, or we add an active PLC4X node in the secure network, that
> requests data, and simply ignores it, and a second – passive mode – node
> sits outside the secure network to capture the information.
>
> I agree that this sort of thing needs to be a “product”. The Automation
> Industry just doesn’t know how to work with frameworks. Ideally a one-box
> thing.
> Right now, most tools I have seen don’t even need the calculation or the
> visualization. This is usually done on another level. Important would be
> that we could be somewhat API compatible with existing products. Them
> usually having SQL or REST APIs, should make it relatively easy to sort of
> produce frontends to our Apache Historian, that are API compatible with
> some existing industry products, so they are replaceable.
>
> If we wanted to add a visualization layer or a data-curation layer later
> on, we should have a chat with the NiFi or StreamPipes folks as that’s what
> they already have.
>
> For now, I would be proposing to build something that uses PLC4X for the
> data-acquisition, IoTDB for the storage and build a REST frontend for this,
> that’s somewhat API compatible with one of the major established products
> and to add more on a step-by-step basis.
>
> I don’t even think it would be bad if an entity like Timecho would add
> enterprise offerings, because I know, that the industry won’t “buy”
> something, if there’s no commercial support or anyone, they can throw money
> at, even if it’s free (That might even make them more skeptical).
>
>
> What do you all think? Does this make sense? If yes, where would it make
> sense to start working on something like this? In the IoTDB project? In the
> PLC4X project, in a separate (new) project?
>
>
> Chris
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Sunday, 27. November 2022 at 06:02
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> Hi all,
>
> It makes sense to use PLC4X+IoTDB to build a historian software.
> Actually, there are several users using IoTDB as their historian
> solution...
>
> According to my knowledge, a historian software contains several features:
>
> - get data from OPC/modubs server and write to IoTDB
>   * use PLC4x + IoTDB-client. (we call this data collector)
>
> - (optional) there is a single-way network gateway for security.
>   * If the gateway is deployed between device and PLC4x program, then
> we need to check whether plc4x supports that.
>   * If the gateway is deployed between IoTDB-client and IoTDB, then we
> need to check wheter IoTDB-client supports that.
>
> - Calculation Engine, e.g., we want to write C to IoTDB when we
> collect A and B from a device (for example, C=A+B/2 )
>   *  IoTDB's trigger supports that but we need a Drag-and-drop
> programming GUI. (maybe integrating some open source projects, like
> Red-node is a good idea)
>
> - get the latest data from IoTDB for visualization (usually users need
> to draw a process flow diagram, and put the latest data on each device
> icon on the diagram)
>   * IoTDB supports getting latest data. but we need a new software for
> drawing the diagram (I do not know if there is any open source project
> for this. In our real applications, we ususally buy and integrate some
> other commerical software)
>
> - send alert message if the real time data meets some rules
>   * IoTDB trigger supports the rule. but currently we have no a GUI.
> (IoTDB also supports Prometheus Alert Manager)
>
> - get the historical data
>   * IoTDB supports that and grafana is good at visualization
>
> - last but not the least, if we provide all the features to industrial
> users,  all the GUIs should be integrated into one entrance.
>
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
> Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> >
> > Hi folks, long time lurker…
> >
> > A historian would be great. Does PLC4X queue data on the machine it’s
> > running on already?
> >
> > A common pattern in historians is to queue data on the runtime server
> until
> > data can be extracted to a sql server, preventing data loss, and reducing
> > the need for redundancy.
> >
> > There are a ton of edge cases to watch out for which I can elaborate on
> > further.
> >
> > I’m a former Industrial Controls Engineer turned software dev and can
> help
> > out where needed.
> >
> > -Ryan
> >
> > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> > wrote:
> >
> > >  Would you name it “Cake”?
> > >
> > > From: Christofer Dutz <ch...@c-ware.de>
> > > <ch...@c-ware.de>
> > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> dev@plc4x.apache.org>
> > > Date: November 26, 2022 at 10:36:23
> > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <dev@plc4x.apache.org
> >,
> > > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > > Subject:  Building a PLC4X and IoTDB Historian?
> > >
> > > Hi all,
> > >
> > > I am currently thinking, if it wouldn’t be a good idea to build
> something
> > > like an open-source Historian based on PLC4X and IoTDB.
> > >
> > > For those of you, who don’t know what a Historian is. In the
> manufacturing
> > > industry they have these extremely expensive servers (usually you buy
> them
> > > as a bundle of hard- and software).
> > > These servers are nothing else than a really crappy and brutally
> expensive
> > > Database for storing Time-Series data.
> > >
> > > The thing is most commercial products are currently really having
> trouble
> > > to keep up with the increasing amount of Data being sent.
> > >
> > > So instead of suggesting building an Historian at each customer’s
> site, I
> > > thought: Perhaps a ready-to-use solution based on open-source would be
> a
> > > good idea.
> > > Not 100% sure where I’d locate such an initiative, but I would tend to
> see
> > > it more on the Database side.
> > >
> > > I think IoTDB would be the perfect storage system, all we seem to need
> is
> > > some sort of rest-interface that matches the industry standards for
> > > querying the information and on the other side something like PLC4X to
> fill
> > > the database.
> > >
> > > Here some examples:
> > >
> > >
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > >
> https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > >
> > > What do you folks think?
> > >
> > >
> > > Chris
> > >
>

Re: Building a PLC4X and IoTDB Historian?

[Christofer Dutz <ch...@c-ware.de>]

Hi all,

Trying to respond to all (except the naming question ;-) ):

@Ryan: PLC4X Doesn��t have such a queueing, but IoTDB has �� so we could be collecting data on one node, writing that to TSFiles using the IoTDB-Client (As Xaingdong mentioned).
These could either be online and transfer data to the server right away, or operate in an offline-like mode and periodically fetch data in bursts from the client (Hope that��s correct, please correct me if I��m wrong)

@Xaiangdong: In general, PLC4X operates in an active mode, but we also have a proposed and prototypically proven ��passive-mode��. Here the software doesn��t actively participate in the data acquisition directly, but simply listens to the data stream and makes data available. We also have something we call a ��Data-Diode��, which technically makes interference impossible as it lets ethernet packets flow in one direction, but nothing in the other (Firewalls usually have the problem of letting packets pass in both directions for established connections). We invented this, because we knew we will never have validated and audited open-source software, to a level that it would be certified for some of these use cases.

So, either we can live with everything the SCADA system is already requesting, or we add an active PLC4X node in the secure network, that requests data, and simply ignores it, and a second �C passive mode �C node sits outside the secure network to capture the information.

I agree that this sort of thing needs to be a ��product��. The Automation Industry just doesn��t know how to work with frameworks. Ideally a one-box thing.
Right now, most tools I have seen don��t even need the calculation or the visualization. This is usually done on another level. Important would be that we could be somewhat API compatible with existing products. Them usually having SQL or REST APIs, should make it relatively easy to sort of produce frontends to our Apache Historian, that are API compatible with some existing industry products, so they are replaceable.

If we wanted to add a visualization layer or a data-curation layer later on, we should have a chat with the NiFi or StreamPipes folks as that��s what they already have.

For now, I would be proposing to build something that uses PLC4X for the data-acquisition, IoTDB for the storage and build a REST frontend for this, that��s somewhat API compatible with one of the major established products and to add more on a step-by-step basis.

I don��t even think it would be bad if an entity like Timecho would add enterprise offerings, because I know, that the industry won��t ��buy�� something, if there��s no commercial support or anyone, they can throw money at, even if it��s free (That might even make them more skeptical).


What do you all think? Does this make sense? If yes, where would it make sense to start working on something like this? In the IoTDB project? In the PLC4X project, in a separate (new) project?


Chris


From: Xiangdong Huang <sa...@gmail.com>
Date: Sunday, 27. November 2022 at 06:02
To: dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: Re: Building a PLC4X and IoTDB Historian?
Hi all,

It makes sense to use PLC4X+IoTDB to build a historian software.
Actually, there are several users using IoTDB as their historian
solution...

According to my knowledge, a historian software contains several features:

- get data from OPC/modubs server and write to IoTDB
  * use PLC4x + IoTDB-client. (we call this data collector)

- (optional) there is a single-way network gateway for security.
  * If the gateway is deployed between device and PLC4x program, then
we need to check whether plc4x supports that.
  * If the gateway is deployed between IoTDB-client and IoTDB, then we
need to check wheter IoTDB-client supports that.

- Calculation Engine, e.g., we want to write C to IoTDB when we
collect A and B from a device (for example, C=A+B/2 )
  *  IoTDB's trigger supports that but we need a Drag-and-drop
programming GUI. (maybe integrating some open source projects, like
Red-node is a good idea)

- get the latest data from IoTDB for visualization (usually users need
to draw a process flow diagram, and put the latest data on each device
icon on the diagram)
  * IoTDB supports getting latest data. but we need a new software for
drawing the diagram (I do not know if there is any open source project
for this. In our real applications, we ususally buy and integrate some
other commerical software)

- send alert message if the real time data meets some rules
  * IoTDB trigger supports the rule. but currently we have no a GUI.
(IoTDB also supports Prometheus Alert Manager)

- get the historical data
  * IoTDB supports that and grafana is good at visualization

- last but not the least, if we provide all the features to industrial
users,  all the GUIs should be integrated into one entrance.


Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 ����
�廪��ѧ ����ѧԺ

Ryan Truran <ry...@gmail.com> ��2022��11��27������ 08:18���
>
> Hi folks, long time lurker��
>
> A historian would be great. Does PLC4X queue data on the machine it��s
> running on already?
>
> A common pattern in historians is to queue data on the runtime server until
> data can be extracted to a sql server, preventing data loss, and reducing
> the need for redundancy.
>
> There are a ton of edge cases to watch out for which I can elaborate on
> further.
>
> I��m a former Industrial Controls Engineer turned software dev and can help
> out where needed.
>
> -Ryan
>
> On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> wrote:
>
> >  Would you name it ��Cake��?
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > <ch...@c-ware.de>
> > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> > Date: November 26, 2022 at 10:36:23
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>,
> > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > Subject:  Building a PLC4X and IoTDB Historian?
> >
> > Hi all,
> >
> > I am currently thinking, if it wouldn��t be a good idea to build something
> > like an open-source Historian based on PLC4X and IoTDB.
> >
> > For those of you, who don��t know what a Historian is. In the manufacturing
> > industry they have these extremely expensive servers (usually you buy them
> > as a bundle of hard- and software).
> > These servers are nothing else than a really crappy and brutally expensive
> > Database for storing Time-Series data.
> >
> > The thing is most commercial products are currently really having trouble
> > to keep up with the increasing amount of Data being sent.
> >
> > So instead of suggesting building an Historian at each customer��s site, I
> > thought: Perhaps a ready-to-use solution based on open-source would be a
> > good idea.
> > Not 100% sure where I��d locate such an initiative, but I would tend to see
> > it more on the Database side.
> >
> > I think IoTDB would be the perfect storage system, all we seem to need is
> > some sort of rest-interface that matches the industry standards for
> > querying the information and on the other side something like PLC4X to fill
> > the database.
> >
> > Here some examples:
> >
> > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> >
> > What do you folks think?
> >
> >
> > Chris
> >

Re: Building a PLC4X and IoTDB Historian?

[Xiangdong Huang <sa...@gmail.com>]

Hi all,

It makes sense to use PLC4X+IoTDB to build a historian software.
Actually, there are several users using IoTDB as their historian
solution...

According to my knowledge, a historian software contains several features:

- get data from OPC/modubs server and write to IoTDB
  * use PLC4x + IoTDB-client. (we call this data collector)

- (optional) there is a single-way network gateway for security.
  * If the gateway is deployed between device and PLC4x program, then
we need to check whether plc4x supports that.
  * If the gateway is deployed between IoTDB-client and IoTDB, then we
need to check wheter IoTDB-client supports that.

- Calculation Engine, e.g., we want to write C to IoTDB when we
collect A and B from a device (for example, C=A+B/2 )
  *  IoTDB's trigger supports that but we need a Drag-and-drop
programming GUI. (maybe integrating some open source projects, like
Red-node is a good idea)

- get the latest data from IoTDB for visualization (usually users need
to draw a process flow diagram, and put the latest data on each device
icon on the diagram)
  * IoTDB supports getting latest data. but we need a new software for
drawing the diagram (I do not know if there is any open source project
for this. In our real applications, we ususally buy and integrate some
other commerical software)

- send alert message if the real time data meets some rules
  * IoTDB trigger supports the rule. but currently we have no a GUI.
(IoTDB also supports Prometheus Alert Manager)

- get the historical data
  * IoTDB supports that and grafana is good at visualization

- last but not the least, if we provide all the features to industrial
users,  all the GUIs should be integrated into one entrance.


Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
>
> Hi folks, long time lurker…
>
> A historian would be great. Does PLC4X queue data on the machine it’s
> running on already?
>
> A common pattern in historians is to queue data on the runtime server until
> data can be extracted to a sql server, preventing data loss, and reducing
> the need for redundancy.
>
> There are a ton of edge cases to watch out for which I can elaborate on
> further.
>
> I’m a former Industrial Controls Engineer turned software dev and can help
> out where needed.
>
> -Ryan
>
> On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
> wrote:
>
> >  Would you name it “Cake”?
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > <ch...@c-ware.de>
> > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> > Date: November 26, 2022 at 10:36:23
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>,
> > dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> > Subject:  Building a PLC4X and IoTDB Historian?
> >
> > Hi all,
> >
> > I am currently thinking, if it wouldn’t be a good idea to build something
> > like an open-source Historian based on PLC4X and IoTDB.
> >
> > For those of you, who don’t know what a Historian is. In the manufacturing
> > industry they have these extremely expensive servers (usually you buy them
> > as a bundle of hard- and software).
> > These servers are nothing else than a really crappy and brutally expensive
> > Database for storing Time-Series data.
> >
> > The thing is most commercial products are currently really having trouble
> > to keep up with the increasing amount of Data being sent.
> >
> > So instead of suggesting building an Historian at each customer’s site, I
> > thought: Perhaps a ready-to-use solution based on open-source would be a
> > good idea.
> > Not 100% sure where I’d locate such an initiative, but I would tend to see
> > it more on the Database side.
> >
> > I think IoTDB would be the perfect storage system, all we seem to need is
> > some sort of rest-interface that matches the industry standards for
> > querying the information and on the other side something like PLC4X to fill
> > the database.
> >
> > Here some examples:
> >
> > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> >
> > What do you folks think?
> >
> >
> > Chris
> >

Re: Building a PLC4X and IoTDB Historian?

[Ryan Truran <ry...@gmail.com>]

Hi folks, long time lurker…

A historian would be great. Does PLC4X queue data on the machine it’s
running on already?

A common pattern in historians is to queue data on the runtime server until
data can be extracted to a sql server, preventing data loss, and reducing
the need for redundancy.

There are a ton of edge cases to watch out for which I can elaborate on
further.

I’m a former Industrial Controls Engineer turned software dev and can help
out where needed.

-Ryan

On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <ot...@gmail.com>
wrote:

>  Would you name it “Cake”?
>
> From: Christofer Dutz <ch...@c-ware.de>
> <ch...@c-ware.de>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Date: November 26, 2022 at 10:36:23
> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>,
> dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
> Subject:  Building a PLC4X and IoTDB Historian?
>
> Hi all,
>
> I am currently thinking, if it wouldn’t be a good idea to build something
> like an open-source Historian based on PLC4X and IoTDB.
>
> For those of you, who don’t know what a Historian is. In the manufacturing
> industry they have these extremely expensive servers (usually you buy them
> as a bundle of hard- and software).
> These servers are nothing else than a really crappy and brutally expensive
> Database for storing Time-Series data.
>
> The thing is most commercial products are currently really having trouble
> to keep up with the increasing amount of Data being sent.
>
> So instead of suggesting building an Historian at each customer’s site, I
> thought: Perhaps a ready-to-use solution based on open-source would be a
> good idea.
> Not 100% sure where I’d locate such an initiative, but I would tend to see
> it more on the Database side.
>
> I think IoTDB would be the perfect storage system, all we seem to need is
> some sort of rest-interface that matches the industry standards for
> querying the information and on the other side something like PLC4X to fill
> the database.
>
> Here some examples:
>
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
>
> What do you folks think?
>
>
> Chris
>

Re: Building a PLC4X and IoTDB Historian?

[Otto Fowler <ot...@gmail.com>]

 Would you name it “Cake”?

From: Christofer Dutz <ch...@c-ware.de>
<ch...@c-ware.de>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: November 26, 2022 at 10:36:23
To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>,
dev@iotdb.apache.org <de...@iotdb.apache.org> <de...@iotdb.apache.org>
Subject:  Building a PLC4X and IoTDB Historian?

Hi all,

I am currently thinking, if it wouldn’t be a good idea to build something
like an open-source Historian based on PLC4X and IoTDB.

For those of you, who don’t know what a Historian is. In the manufacturing
industry they have these extremely expensive servers (usually you buy them
as a bundle of hard- and software).
These servers are nothing else than a really crappy and brutally expensive
Database for storing Time-Series data.

The thing is most commercial products are currently really having trouble
to keep up with the increasing amount of Data being sent.

So instead of suggesting building an Historian at each customer’s site, I
thought: Perhaps a ready-to-use solution based on open-source would be a
good idea.
Not 100% sure where I’d locate such an initiative, but I would tend to see
it more on the Database side.

I think IoTDB would be the perfect storage system, all we seem to need is
some sort of rest-interface that matches the industry standards for
querying the information and on the other side something like PLC4X to fill
the database.

Here some examples:
https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf

What do you folks think?


Chris

Re: Building a PLC4X and IoTDB Historian?

[Lukas Ott <ot...@gmail.com>]

+1 for that . Apache driven Historian would be quite awesome :-)

Am Sa., 26. Nov. 2022 um 16:36 Uhr schrieb Christofer Dutz <
christofer.dutz@c-ware.de>:

> Hi all,
>
> I am currently thinking, if it wouldn’t be a good idea to build something
> like an open-source Historian based on PLC4X and IoTDB.
>
> For those of you, who don’t know what a Historian is. In the manufacturing
> industry they have these extremely expensive servers (usually you buy them
> as a bundle of hard- and software).
> These servers are nothing else than a really crappy and brutally expensive
> Database for storing Time-Series data.
>
> The thing is most commercial products are currently really having trouble
> to keep up with the increasing amount of Data being sent.
>
> So instead of suggesting building an Historian at each customer’s site, I
> thought: Perhaps a ready-to-use solution based on open-source would be a
> good idea.
> Not 100% sure where I’d locate such an initiative, but I would tend to see
> it more on the Database side.
>
> I think IoTDB would be the perfect storage system, all we seem to need is
> some sort of rest-interface that matches the industry standards for
> querying the information and on the other side something like PLC4X to fill
> the database.
>
> Here some examples:
>
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
>
> What do you folks think?
>
>
> Chris
>