Re: Building a PLC4X and IoTDB Historian?

[Otto Fowler <ot...@gmail.com>]

have you used OSI PI?  "I think with the technology one has today it is
very easy to "outclass" these established systems e.g. in performance,
efficiency and als user friendliness."   It would not be easy to replicate
or outclass what they have done in decades of work.

On Mon, Nov 28, 2022 at 3:17 AM Julian Feinauer <
j.feinauer@pragmaticindustries.de> wrote:

> Hi all,
>
> first of all.. sorry for coming in so late.
> I think this is a great idea and there really is a product market fit for
> such a solution.
> I think technology wise its not "that complicated" to build such a system,
> I think many of the details that are necessary for a good adoption of such
> a software is in the areas around like documentation, marketing, more
> marketing, even more and even bolder marketing and sales (how to sell an
> open source project??).
>
> But technology wise I really like the idea and I think with the technology
> one has today it is very easy to "outclass" these established systems e.g.
> in performance, efficiency and als user friendliness.
> The only thing which might be challenging for the future are things like
> plugins or an extension system to allow users to customize their
> installation.
>
> Regarding the "separate project" approach I am not 100% certain.
> Personally, I would consider starting EITHER as a subproject in an
> existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere
> else and not directly go to the Incubator or something.
> Because I think we should really find out if there is enough developer
> interest to build and sustain such a system.
>
> But I'm totally in for such a system either way!
>
> Julian
>
> PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with
> historians
>
> Von: Christofer Dutz <ch...@c-ware.de>
> Datum: Montag, 28. November 2022 um 08:56
> An: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Betreff: Re: Building a PLC4X and IoTDB Historian?
> Hi Xiangdong,
>
> that’s perfect that some of you folks would be on board. I should probably
> check, if there are enough PLC4X folks on board too … would be a shame if
> it was just me ;-)
>
> And regarding the name … yeah … you are absolutely right: Historian might
> really be the ideal choice … as it instantly explains what it is … A
> Historian from the Apache Software Foundation and probably also no bias
> towards any regional culture.
>
> So (Speaking to my fellow PLC4X folks … who would be on board with this?)
>
>
> Chris
>
>
>
> From: Xiangdong Huang <sa...@gmail.com>
> Date: Monday, 28. November 2022 at 04:24
> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: Re: Building a PLC4X and IoTDB Historian?
> > I guess I would sort of also vote for a separate Project. Would however
> only make sense if some people from both of our projects would join in.
>
> Agree, I think the iotdb developers in Timecho can join.
>
> > And name-wise … one of the names of Historic Historians (Ideally a Greek
> one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are
> “famous” Buttler names ;-)
>
> If we consider famous names (well, there will be culture bias), I'd
> like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as
> "Records of the Grand Historian" is tooo famous (at least in China).
>
> BTW. naming "Apache Historian" directly may be another option.
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
>
> Christofer Dutz <ch...@c-ware.de> 于2022年11月27日周日 22:56写道：
> >
> > How about Apache Ephorous? ;-)
> >
> > https://en.wikipedia.org/wiki/Ephorus
> > Ephorus of Cyme (/ˈɛfərəs/<
> https://en.wikipedia.org/wiki/Help:IPA/English>; Greek<
> https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, Ephoros
> ho Kymaios; c. 400 – 330 BC) was an ancient Greek<
> https://en.wikipedia.org/wiki/Ancient_Greece> historian<
> https://en.wikipedia.org/wiki/Historian> known for his universal history<
> https://en.wikipedia.org/wiki/Universal_history>.
> >
> > Chris
> >
> > From: Christofer Dutz <ch...@c-ware.de>
> > Date: Sunday, 27. November 2022 at 15:28
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > Hi all,
> >
> > I guess I would sort of also vote for a separate Project. Would however
> only make sense if some people from both of our projects would join in.
> >
> > And name-wise … one of the names of Historic Historians (Ideally a Greek
> one) would make it into my top 10 ;-) … sort of like Hudson or Jenkins are
> “famous” Buttler names ;-)
> > https://en.wikipedia.org/wiki/List_of_Greek_historiographers
> > https://en.wikipedia.org/wiki/List_of_historians
> >
> > I think anything but a turnkey-ready solution will not be accepted by
> anyone in the Automation industry.
> >
> > Chris
> >
> >
> >
> >
> > From: Xiangdong Huang <sa...@gmail.com>
> > Date: Sunday, 27. November 2022 at 14:05
> > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > Subject: Re: Building a PLC4X and IoTDB Historian?
> > If we just provide a solution, or demonstration, then either is ok.
> > If we want to provide an "one-box thing" (even without GUI), +1 for a
> > new project.
> >
> > > I don’t even think it would be bad if an entity like Timecho would add
> enterprise offerings, because I know, that the industry won’t “buy”
> something, if there’s no commercial support or anyone, they can throw money
> at, even if it’s free (That might even make them more skeptical).
> > Yes, indeed. :D
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> >
> > Ben Hutcheson <be...@gmail.com> 于2022年11月27日周日 18:38写道：
> > >
> > > Hi,
> > >
> > > I think it's a great idea.
> > >
> > > I'd vote for having a separate project for something like this, just so
> > > it's clear that it is a historian and people can build it separately. I
> > > wouldn't want it hidden away in an obscure folder in the PLC4X repo.
> > >
> > > Cake does go with Cafe.
> > >
> > > Ben
> > >
> > >
> > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <
> christofer.dutz@c-ware.de>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Trying to respond to all (except the naming question ;-) ):
> > > >
> > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we
> could be
> > > > collecting data on one node, writing that to TSFiles using the
> IoTDB-Client
> > > > (As Xaingdong mentioned).
> > > > These could either be online and transfer data to the server right
> away,
> > > > or operate in an offline-like mode and periodically fetch data in
> bursts
> > > > from the client (Hope that’s correct, please correct me if I’m wrong)
> > > >
> > > > @Xaiangdong: In general, PLC4X operates in an active mode, but we
> also
> > > > have a proposed and prototypically proven “passive-mode”. Here the
> software
> > > > doesn’t actively participate in the data acquisition directly, but
> simply
> > > > listens to the data stream and makes data available. We also have
> something
> > > > we call a “Data-Diode”, which technically makes interference
> impossible as
> > > > it lets ethernet packets flow in one direction, but nothing in the
> other
> > > > (Firewalls usually have the problem of letting packets pass in both
> > > > directions for established connections). We invented this, because
> we knew
> > > > we will never have validated and audited open-source software, to a
> level
> > > > that it would be certified for some of these use cases.
> > > >
> > > > So, either we can live with everything the SCADA system is already
> > > > requesting, or we add an active PLC4X node in the secure network,
> that
> > > > requests data, and simply ignores it, and a second – passive mode –
> node
> > > > sits outside the secure network to capture the information.
> > > >
> > > > I agree that this sort of thing needs to be a “product”. The
> Automation
> > > > Industry just doesn’t know how to work with frameworks. Ideally a
> one-box
> > > > thing.
> > > > Right now, most tools I have seen don’t even need the calculation or
> the
> > > > visualization. This is usually done on another level. Important
> would be
> > > > that we could be somewhat API compatible with existing products. Them
> > > > usually having SQL or REST APIs, should make it relatively easy to
> sort of
> > > > produce frontends to our Apache Historian, that are API compatible
> with
> > > > some existing industry products, so they are replaceable.
> > > >
> > > > If we wanted to add a visualization layer or a data-curation layer
> later
> > > > on, we should have a chat with the NiFi or StreamPipes folks as
> that’s what
> > > > they already have.
> > > >
> > > > For now, I would be proposing to build something that uses PLC4X for
> the
> > > > data-acquisition, IoTDB for the storage and build a REST frontend
> for this,
> > > > that’s somewhat API compatible with one of the major established
> products
> > > > and to add more on a step-by-step basis.
> > > >
> > > > I don’t even think it would be bad if an entity like Timecho would
> add
> > > > enterprise offerings, because I know, that the industry won’t “buy”
> > > > something, if there’s no commercial support or anyone, they can
> throw money
> > > > at, even if it’s free (That might even make them more skeptical).
> > > >
> > > >
> > > > What do you all think? Does this make sense? If yes, where would it
> make
> > > > sense to start working on something like this? In the IoTDB project?
> In the
> > > > PLC4X project, in a separate (new) project?
> > > >
> > > >
> > > > Chris
> > > >
> > > >
> > > > From: Xiangdong Huang <sa...@gmail.com>
> > > > Date: Sunday, 27. November 2022 at 06:02
> > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org>
> > > > Subject: Re: Building a PLC4X and IoTDB Historian?
> > > > Hi all,
> > > >
> > > > It makes sense to use PLC4X+IoTDB to build a historian software.
> > > > Actually, there are several users using IoTDB as their historian
> > > > solution...
> > > >
> > > > According to my knowledge, a historian software contains several
> features:
> > > >
> > > > - get data from OPC/modubs server and write to IoTDB
> > > >   * use PLC4x + IoTDB-client. (we call this data collector)
> > > >
> > > > - (optional) there is a single-way network gateway for security.
> > > >   * If the gateway is deployed between device and PLC4x program, then
> > > > we need to check whether plc4x supports that.
> > > >   * If the gateway is deployed between IoTDB-client and IoTDB, then
> we
> > > > need to check wheter IoTDB-client supports that.
> > > >
> > > > - Calculation Engine, e.g., we want to write C to IoTDB when we
> > > > collect A and B from a device (for example, C=A+B/2 )
> > > >   *  IoTDB's trigger supports that but we need a Drag-and-drop
> > > > programming GUI. (maybe integrating some open source projects, like
> > > > Red-node is a good idea)
> > > >
> > > > - get the latest data from IoTDB for visualization (usually users
> need
> > > > to draw a process flow diagram, and put the latest data on each
> device
> > > > icon on the diagram)
> > > >   * IoTDB supports getting latest data. but we need a new software
> for
> > > > drawing the diagram (I do not know if there is any open source
> project
> > > > for this. In our real applications, we ususally buy and integrate
> some
> > > > other commerical software)
> > > >
> > > > - send alert message if the real time data meets some rules
> > > >   * IoTDB trigger supports the rule. but currently we have no a GUI.
> > > > (IoTDB also supports Prometheus Alert Manager)
> > > >
> > > > - get the historical data
> > > >   * IoTDB supports that and grafana is good at visualization
> > > >
> > > > - last but not the least, if we provide all the features to
> industrial
> > > > users,  all the GUIs should be integrated into one entrance.
> > > >
> > > >
> > > > Best,
> > > > -----------------------------------
> > > > Xiangdong Huang
> > > > School of Software, Tsinghua University
> > > >
> > > >  黄向东
> > > > 清华大学 软件学院
> > > >
> > > > Ryan Truran <ry...@gmail.com> 于2022年11月27日周日 08:18写道：
> > > > >
> > > > > Hi folks, long time lurker…
> > > > >
> > > > > A historian would be great. Does PLC4X queue data on the machine
> it’s
> > > > > running on already?
> > > > >
> > > > > A common pattern in historians is to queue data on the runtime
> server
> > > > until
> > > > > data can be extracted to a sql server, preventing data loss, and
> reducing
> > > > > the need for redundancy.
> > > > >
> > > > > There are a ton of edge cases to watch out for which I can
> elaborate on
> > > > > further.
> > > > >
> > > > > I’m a former Industrial Controls Engineer turned software dev and
> can
> > > > help
> > > > > out where needed.
> > > > >
> > > > > -Ryan
> > > > >
> > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <
> ottobackwards@gmail.com>
> > > > > wrote:
> > > > >
> > > > > >  Would you name it “Cake”?
> > > > > >
> > > > > > From: Christofer Dutz <ch...@c-ware.de>
> > > > > > <ch...@c-ware.de>
> > > > > > Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> > > > dev@plc4x.apache.org>
> > > > > > Date: November 26, 2022 at 10:36:23
> > > > > > To: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> dev@plc4x.apache.org
> > > > >,
> > > > > > dev@iotdb.apache.org <de...@iotdb.apache.org> <
> dev@iotdb.apache.org>
> > > > > > Subject:  Building a PLC4X and IoTDB Historian?
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I am currently thinking, if it wouldn’t be a good idea to build
> > > > something
> > > > > > like an open-source Historian based on PLC4X and IoTDB.
> > > > > >
> > > > > > For those of you, who don’t know what a Historian is. In the
> > > > manufacturing
> > > > > > industry they have these extremely expensive servers (usually
> you buy
> > > > them
> > > > > > as a bundle of hard- and software).
> > > > > > These servers are nothing else than a really crappy and brutally
> > > > expensive
> > > > > > Database for storing Time-Series data.
> > > > > >
> > > > > > The thing is most commercial products are currently really having
> > > > trouble
> > > > > > to keep up with the increasing amount of Data being sent.
> > > > > >
> > > > > > So instead of suggesting building an Historian at each customer’s
> > > > site, I
> > > > > > thought: Perhaps a ready-to-use solution based on open-source
> would be
> > > > a
> > > > > > good idea.
> > > > > > Not 100% sure where I’d locate such an initiative, but I would
> tend to
> > > > see
> > > > > > it more on the Database side.
> > > > > >
> > > > > > I think IoTDB would be the perfect storage system, all we seem
> to need
> > > > is
> > > > > > some sort of rest-interface that matches the industry standards
> for
> > > > > > querying the information and on the other side something like
> PLC4X to
> > > > fill
> > > > > > the database.
> > > > > >
> > > > > > Here some examples:
> > > > > >
> > > > > >
> > > >
> https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html
> > > > > >
> > > >
> https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf
> > > > > >
> > > > > > What do you folks think?
> > > > > >
> > > > > >
> > > > > > Chris
> > > > > >
> > > >
>