You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hop.apache.org by Cesar Garcia <ce...@ceos.com.ve> on 2021/12/14 15:12:25 UTC
Apache PLC4X Integration
Hi all,
My name is Cesar Garcia, I am part of the Apache PLC4x team.
I have been evaluating integrating the PLC4x communication libraries to the
HOP environment, but I have the following doubts:
1. It is possible to share a connection between all pipelines.
2. In this aspect, it is correct to create the connection in an "Action"
and share it through "PluginRegistry.getInstance ()".
The pattern of use is that each pipelines creates its connection to the
databases, in our type of application the connection to the devices is
limited, therefore they must share the resource.
Grateful for your comments,
Best regards,
--
*CEOS Automatización, C.A.*
*GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
*PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
*FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
*Ing. César García*
*Cel: +58 414-760.98.95*
*Hotline Técnica SIEMENS: 0800 1005080*
*Email: support.aan.automation@siemens.com
<su...@siemens.com>*
Re: Apache PLC4X Integration
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Christofer,
Consider it done ;)
Cheers,
Hans
On Fri, 21 Jan 2022 at 16:01, Christofer Dutz <ch...@c-ware.de>
wrote:
> Hi Hans,
>
> you should probably consider subscribing to iot@apache.org ...it's
> currently a super low-volume list dedicated to fostering exchange within
> projects we count to the iot space.
>
> Chris :-)
>
> -----Original Message-----
> From: Hans Van Akelyen <ha...@gmail.com>
> Sent: Freitag, 21. Januar 2022 15:54
> To: dev@hop.apache.org
> Subject: Re: Apache PLC4X Integration
>
> Hi Cesar and Christofer,
>
> This is great news, I was also looking at the PLC4X projects a while back
> and thought it would be a great fit to create transforms for it.
> Unfortunately did not yet find the time to start the actual integration
> work. It is great to see how the Apache projects interact with each other.
> Very happy to have joined this community!
>
> If you need any help please let us know!
>
> Cheers,
> Hans
>
> On Fri, 21 Jan 2022 at 15:31, Cesar Garcia <ce...@ceos.com.ve>
> wrote:
>
> > Hello Chris,
> >
> > I Use another method to share the connection. The connection is
> > wrapped with a ReferenceCountUtil object from the Netty library, which
> > is shared by the PLC4XTransforms.
> >
> > I thought that the most laborious thing for the integration of the
> > libraries would be the handling of types, but everything was almost
> > transparent.
> >
> > I am currently attacking a memory leak that affects long-term
> > operation to consider integrating into Hop.
> >
> > I am very happy with the integration of both projects,
> >
> > Kind regards,
> >
> > El vie, 21 ene 2022 a las 7:54, Christofer Dutz (<cd...@apache.org>)
> > escribió:
> >
> > > Hi Hop community and @Cesar,
> > >
> > > coming a bit late to the party ... but how about using our Plc4x
> > > connection-pool/cache for this?
> > > We could share one instance of the connection pool and then each
> > operation
> > > leases and returns the connections and the pool should manage the
> > > concurrency.
> > >
> > > If it runs on multiple hosts it would obviously be multiple
> > > instances of the pool and hereby multiple connections to a PLC (one
> > > from each node)
> > ...
> > > Generally when working with PLCs I wouldn't reccomend to connect a
> > cluster
> > > to a plc as that's just not how PLCs work (Some only allow 1-3
> > > concurrent connections).
> > >
> > > Chris
> > >
> > >
> > > On 2021/12/15 13:57:51 Cesar Garcia wrote:
> > > > Hi Matt,
> > > >
> > > > Thank you for your comments,
> > > >
> > > > Indeed, as you point out, in this case, these libraries should run
> > > locally,
> > > > in order to transfer the information to the pipes.
> > > >
> > > > Excellent observation on how to store generic objects in pipes
> > > > with "getExtensionDataMap" and as you point out, the problem is
> > > > the life
> > cycle
> > > > management of objects between pipes (the entertaining part).
> > > >
> > > > I will be consulting any dua,
> > > >
> > > > Thanks a lot.
> > > >
> > > > El mar, 14 dic 2021 a las 14:59, Matt Casters
> > > > (<ma...@neo4j.com.invalid>) escribió:
> > > >
> > > > > Hi César,
> > > > >
> > > > > First of all: welcome to the Hop community.
> > > > >
> > > > > Usually we make a distinction between the connection metadata
> > > (hostname,
> > > > > port, username, password, and all sorts of other options).
> > > > > Implementing this is fairly easy. Take for example the source
> code
> > > > > of the Cassandra
> > > > > database
> > > > > <
> > > > >
> > >
> > https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassa
> > ndra/src/main/java/org/apache/hop/databases/cassandra/metadata/Cassand
> > raConnection.java
> > > > > >
> > > > > connection metadata.
> > > > > In the same package you can find the code for the user interface
> > > > > ( CassandraConnectionEditor.java <
> > > > >
> > >
> > https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassa
> > ndra/src/main/java/org/apache/hop/databases/cassandra/metadata/Cassand
> > raConnectionEditor.java
> > > > > >).
> > > > > Most of the user interface is generated automatically.
> > > > > Doing it like this will make a PLC4x connection a first class
> > metadata
> > > > > citizen and show up in the metadata perspective for example.
> > > > > You'd
> > also
> > > > > have direct access to the metadata widget in the plugin user
> > > interfaces.
> > > > > This makes the lives of the PLC4x users a lot easier.
> > > > >
> > > > > For the actual instantiation of a connection at runtime it's
> > > > > true
> > that
> > > > > usually every transform has its own connection but this doesn't
> > > > > have
> > > to be
> > > > > the case.
> > > > > Transforms can store and retrieve runtime objects in the current
> > > pipeline
> > > > > using *IPipelineEngine.getExtensionDataMap()*.
> > > > > You do need to be careful about race conditions since all
> > > > > transforms
> > > start
> > > > > in parallel. Obviously you can also use a singleton design
> > > > > pattern
> > in
> > > > > Java.
> > > > >
> > > > > Also note that this connection sharing would work for a local
> > > > > Hop
> > > execution
> > > > > engine but not for Beam where processing happens typically in a
> > > massively
> > > > > parallel fashion.
> > > > > While technically the pipelines would still work there would be
> > > > > no immediate way for the implementation to limit the number of
> > > connections.
> > > > >
> > > > > Please let us know if you need any help with the creation of
> > > > > PLC4x connectivity plugins themselves. We'd be happy to help out.
> > > > >
> > > > > Kind regard,
> > > > >
> > > > > Matt
> > > > >
> > > > > On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <
> > cesar.garcia@ceos.com.ve
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > My name is Cesar Garcia, I am part of the Apache PLC4x team.
> > > > > >
> > > > > > I have been evaluating integrating the PLC4x communication
> > libraries
> > > to
> > > > > the
> > > > > > HOP environment, but I have the following doubts:
> > > > > >
> > > > > > 1. It is possible to share a connection between all pipelines.
> > > > > > 2. In this aspect, it is correct to create the connection in
> > > > > > an
> > > "Action"
> > > > > > and share it through "PluginRegistry.getInstance ()".
> > > > > >
> > > > > > The pattern of use is that each pipelines creates its
> > > > > > connection to
> > > the
> > > > > > databases, in our type of application the connection to the
> > > > > > devices
> > > is
> > > > > > limited, therefore they must share the resource.
> > > > > >
> > > > > > Grateful for your comments,
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > --
> > > > > > *CEOS Automatización, C.A.*
> > > > > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,* *PISO 1,
> > > > > > OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > > > > >
> > > > > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO.
> > > > > > ANZOATEGUI* *Ing. César García*
> > > > > >
> > > > > > *Cel: +58 414-760.98.95*
> > > > > >
> > > > > > *Hotline Técnica SIEMENS: 0800 1005080*
> > > > > >
> > > > > > *Email: support.aan.automation@siemens.com
> > > > > > <su...@siemens.com>*
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Neo4j Chief Solutions Architect
> > > > > *✉ *matt.casters@neo4j.com
> > > > >
> > > >
> > > >
> > > > --
> > > > *CEOS Automatización, C.A.*
> > > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,* *PISO 1,
> > > > OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > > >
> > > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > > > *Ing. César García*
> > > >
> > > > *Cel: +58 414-760.98.95*
> > > >
> > > > *Hotline Técnica SIEMENS: 0800 1005080*
> > > >
> > > > *Email: support.aan.automation@siemens.com
> > > > <su...@siemens.com>*
> > > >
> > >
> >
> >
> > --
> > *CEOS Automatización, C.A.*
> > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,* *PISO 1, OFICINA 2,
> > AV. RAUL LEONI, SECTOR GUAMACHITO,*
> >
> > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI* *Ing.
> > César García*
> >
> > *Cel: +58 414-760.98.95*
> >
> > *Hotline Técnica SIEMENS: 0800 1005080*
> >
> > *Email: support.aan.automation@siemens.com
> > <su...@siemens.com>*
> >
>
RE: Apache PLC4X Integration
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Hans,
you should probably consider subscribing to iot@apache.org ...it's currently a super low-volume list dedicated to fostering exchange within projects we count to the iot space.
Chris :-)
-----Original Message-----
From: Hans Van Akelyen <ha...@gmail.com>
Sent: Freitag, 21. Januar 2022 15:54
To: dev@hop.apache.org
Subject: Re: Apache PLC4X Integration
Hi Cesar and Christofer,
This is great news, I was also looking at the PLC4X projects a while back and thought it would be a great fit to create transforms for it.
Unfortunately did not yet find the time to start the actual integration work. It is great to see how the Apache projects interact with each other.
Very happy to have joined this community!
If you need any help please let us know!
Cheers,
Hans
On Fri, 21 Jan 2022 at 15:31, Cesar Garcia <ce...@ceos.com.ve> wrote:
> Hello Chris,
>
> I Use another method to share the connection. The connection is
> wrapped with a ReferenceCountUtil object from the Netty library, which
> is shared by the PLC4XTransforms.
>
> I thought that the most laborious thing for the integration of the
> libraries would be the handling of types, but everything was almost
> transparent.
>
> I am currently attacking a memory leak that affects long-term
> operation to consider integrating into Hop.
>
> I am very happy with the integration of both projects,
>
> Kind regards,
>
> El vie, 21 ene 2022 a las 7:54, Christofer Dutz (<cd...@apache.org>)
> escribió:
>
> > Hi Hop community and @Cesar,
> >
> > coming a bit late to the party ... but how about using our Plc4x
> > connection-pool/cache for this?
> > We could share one instance of the connection pool and then each
> operation
> > leases and returns the connections and the pool should manage the
> > concurrency.
> >
> > If it runs on multiple hosts it would obviously be multiple
> > instances of the pool and hereby multiple connections to a PLC (one
> > from each node)
> ...
> > Generally when working with PLCs I wouldn't reccomend to connect a
> cluster
> > to a plc as that's just not how PLCs work (Some only allow 1-3
> > concurrent connections).
> >
> > Chris
> >
> >
> > On 2021/12/15 13:57:51 Cesar Garcia wrote:
> > > Hi Matt,
> > >
> > > Thank you for your comments,
> > >
> > > Indeed, as you point out, in this case, these libraries should run
> > locally,
> > > in order to transfer the information to the pipes.
> > >
> > > Excellent observation on how to store generic objects in pipes
> > > with "getExtensionDataMap" and as you point out, the problem is
> > > the life
> cycle
> > > management of objects between pipes (the entertaining part).
> > >
> > > I will be consulting any dua,
> > >
> > > Thanks a lot.
> > >
> > > El mar, 14 dic 2021 a las 14:59, Matt Casters
> > > (<ma...@neo4j.com.invalid>) escribió:
> > >
> > > > Hi César,
> > > >
> > > > First of all: welcome to the Hop community.
> > > >
> > > > Usually we make a distinction between the connection metadata
> > (hostname,
> > > > port, username, password, and all sorts of other options).
> > > > Implementing this is fairly easy. Take for example the source code
> > > > of the Cassandra
> > > > database
> > > > <
> > > >
> >
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassa
> ndra/src/main/java/org/apache/hop/databases/cassandra/metadata/Cassand
> raConnection.java
> > > > >
> > > > connection metadata.
> > > > In the same package you can find the code for the user interface
> > > > ( CassandraConnectionEditor.java <
> > > >
> >
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassa
> ndra/src/main/java/org/apache/hop/databases/cassandra/metadata/Cassand
> raConnectionEditor.java
> > > > >).
> > > > Most of the user interface is generated automatically.
> > > > Doing it like this will make a PLC4x connection a first class
> metadata
> > > > citizen and show up in the metadata perspective for example.
> > > > You'd
> also
> > > > have direct access to the metadata widget in the plugin user
> > interfaces.
> > > > This makes the lives of the PLC4x users a lot easier.
> > > >
> > > > For the actual instantiation of a connection at runtime it's
> > > > true
> that
> > > > usually every transform has its own connection but this doesn't
> > > > have
> > to be
> > > > the case.
> > > > Transforms can store and retrieve runtime objects in the current
> > pipeline
> > > > using *IPipelineEngine.getExtensionDataMap()*.
> > > > You do need to be careful about race conditions since all
> > > > transforms
> > start
> > > > in parallel. Obviously you can also use a singleton design
> > > > pattern
> in
> > > > Java.
> > > >
> > > > Also note that this connection sharing would work for a local
> > > > Hop
> > execution
> > > > engine but not for Beam where processing happens typically in a
> > massively
> > > > parallel fashion.
> > > > While technically the pipelines would still work there would be
> > > > no immediate way for the implementation to limit the number of
> > connections.
> > > >
> > > > Please let us know if you need any help with the creation of
> > > > PLC4x connectivity plugins themselves. We'd be happy to help out.
> > > >
> > > > Kind regard,
> > > >
> > > > Matt
> > > >
> > > > On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <
> cesar.garcia@ceos.com.ve
> > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > My name is Cesar Garcia, I am part of the Apache PLC4x team.
> > > > >
> > > > > I have been evaluating integrating the PLC4x communication
> libraries
> > to
> > > > the
> > > > > HOP environment, but I have the following doubts:
> > > > >
> > > > > 1. It is possible to share a connection between all pipelines.
> > > > > 2. In this aspect, it is correct to create the connection in
> > > > > an
> > "Action"
> > > > > and share it through "PluginRegistry.getInstance ()".
> > > > >
> > > > > The pattern of use is that each pipelines creates its
> > > > > connection to
> > the
> > > > > databases, in our type of application the connection to the
> > > > > devices
> > is
> > > > > limited, therefore they must share the resource.
> > > > >
> > > > > Grateful for your comments,
> > > > >
> > > > > Best regards,
> > > > >
> > > > > --
> > > > > *CEOS Automatización, C.A.*
> > > > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,* *PISO 1,
> > > > > OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > > > >
> > > > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO.
> > > > > ANZOATEGUI* *Ing. César García*
> > > > >
> > > > > *Cel: +58 414-760.98.95*
> > > > >
> > > > > *Hotline Técnica SIEMENS: 0800 1005080*
> > > > >
> > > > > *Email: support.aan.automation@siemens.com
> > > > > <su...@siemens.com>*
> > > > >
> > > >
> > > >
> > > > --
> > > > Neo4j Chief Solutions Architect
> > > > *✉ *matt.casters@neo4j.com
> > > >
> > >
> > >
> > > --
> > > *CEOS Automatización, C.A.*
> > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,* *PISO 1,
> > > OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > >
> > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > > *Ing. César García*
> > >
> > > *Cel: +58 414-760.98.95*
> > >
> > > *Hotline Técnica SIEMENS: 0800 1005080*
> > >
> > > *Email: support.aan.automation@siemens.com
> > > <su...@siemens.com>*
> > >
> >
>
>
> --
> *CEOS Automatización, C.A.*
> *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,* *PISO 1, OFICINA 2,
> AV. RAUL LEONI, SECTOR GUAMACHITO,*
>
> *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI* *Ing.
> César García*
>
> *Cel: +58 414-760.98.95*
>
> *Hotline Técnica SIEMENS: 0800 1005080*
>
> *Email: support.aan.automation@siemens.com
> <su...@siemens.com>*
>
Re: Apache PLC4X Integration
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Cesar and Christofer,
This is great news, I was also looking at the PLC4X projects a while back
and thought it would be a great fit to create transforms for it.
Unfortunately did not yet find the time to start the actual integration
work. It is great to see how the Apache projects interact with each other.
Very happy to have joined this community!
If you need any help please let us know!
Cheers,
Hans
On Fri, 21 Jan 2022 at 15:31, Cesar Garcia <ce...@ceos.com.ve> wrote:
> Hello Chris,
>
> I Use another method to share the connection. The connection is wrapped
> with a ReferenceCountUtil object from the Netty library, which is shared by
> the PLC4XTransforms.
>
> I thought that the most laborious thing for the integration of the
> libraries would be the handling of types, but everything was almost
> transparent.
>
> I am currently attacking a memory leak that affects long-term operation to
> consider integrating into Hop.
>
> I am very happy with the integration of both projects,
>
> Kind regards,
>
> El vie, 21 ene 2022 a las 7:54, Christofer Dutz (<cd...@apache.org>)
> escribió:
>
> > Hi Hop community and @Cesar,
> >
> > coming a bit late to the party ... but how about using our Plc4x
> > connection-pool/cache for this?
> > We could share one instance of the connection pool and then each
> operation
> > leases and returns the connections and the pool should manage the
> > concurrency.
> >
> > If it runs on multiple hosts it would obviously be multiple instances of
> > the pool and hereby multiple connections to a PLC (one from each node)
> ...
> > Generally when working with PLCs I wouldn't reccomend to connect a
> cluster
> > to a plc as that's just not how PLCs work (Some only allow 1-3 concurrent
> > connections).
> >
> > Chris
> >
> >
> > On 2021/12/15 13:57:51 Cesar Garcia wrote:
> > > Hi Matt,
> > >
> > > Thank you for your comments,
> > >
> > > Indeed, as you point out, in this case, these libraries should run
> > locally,
> > > in order to transfer the information to the pipes.
> > >
> > > Excellent observation on how to store generic objects in pipes with
> > > "getExtensionDataMap" and as you point out, the problem is the life
> cycle
> > > management of objects between pipes (the entertaining part).
> > >
> > > I will be consulting any dua,
> > >
> > > Thanks a lot.
> > >
> > > El mar, 14 dic 2021 a las 14:59, Matt Casters
> > > (<ma...@neo4j.com.invalid>) escribió:
> > >
> > > > Hi César,
> > > >
> > > > First of all: welcome to the Hop community.
> > > >
> > > > Usually we make a distinction between the connection metadata
> > (hostname,
> > > > port, username, password, and all sorts of other options).
> > > > Implementing this is fairly easy. Take for example the source code
> > > > of the Cassandra
> > > > database
> > > > <
> > > >
> >
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnection.java
> > > > >
> > > > connection metadata.
> > > > In the same package you can find the code for the user interface (
> > > > CassandraConnectionEditor.java
> > > > <
> > > >
> >
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnectionEditor.java
> > > > >).
> > > > Most of the user interface is generated automatically.
> > > > Doing it like this will make a PLC4x connection a first class
> metadata
> > > > citizen and show up in the metadata perspective for example. You'd
> also
> > > > have direct access to the metadata widget in the plugin user
> > interfaces.
> > > > This makes the lives of the PLC4x users a lot easier.
> > > >
> > > > For the actual instantiation of a connection at runtime it's true
> that
> > > > usually every transform has its own connection but this doesn't have
> > to be
> > > > the case.
> > > > Transforms can store and retrieve runtime objects in the current
> > pipeline
> > > > using *IPipelineEngine.getExtensionDataMap()*.
> > > > You do need to be careful about race conditions since all transforms
> > start
> > > > in parallel. Obviously you can also use a singleton design pattern
> in
> > > > Java.
> > > >
> > > > Also note that this connection sharing would work for a local Hop
> > execution
> > > > engine but not for Beam where processing happens typically in a
> > massively
> > > > parallel fashion.
> > > > While technically the pipelines would still work there would be no
> > > > immediate way for the implementation to limit the number of
> > connections.
> > > >
> > > > Please let us know if you need any help with the creation of PLC4x
> > > > connectivity plugins themselves. We'd be happy to help out.
> > > >
> > > > Kind regard,
> > > >
> > > > Matt
> > > >
> > > > On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <
> cesar.garcia@ceos.com.ve
> > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > My name is Cesar Garcia, I am part of the Apache PLC4x team.
> > > > >
> > > > > I have been evaluating integrating the PLC4x communication
> libraries
> > to
> > > > the
> > > > > HOP environment, but I have the following doubts:
> > > > >
> > > > > 1. It is possible to share a connection between all pipelines.
> > > > > 2. In this aspect, it is correct to create the connection in an
> > "Action"
> > > > > and share it through "PluginRegistry.getInstance ()".
> > > > >
> > > > > The pattern of use is that each pipelines creates its connection to
> > the
> > > > > databases, in our type of application the connection to the devices
> > is
> > > > > limited, therefore they must share the resource.
> > > > >
> > > > > Grateful for your comments,
> > > > >
> > > > > Best regards,
> > > > >
> > > > > --
> > > > > *CEOS Automatización, C.A.*
> > > > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> > > > > *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > > > >
> > > > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > > > > *Ing. César García*
> > > > >
> > > > > *Cel: +58 414-760.98.95*
> > > > >
> > > > > *Hotline Técnica SIEMENS: 0800 1005080*
> > > > >
> > > > > *Email: support.aan.automation@siemens.com
> > > > > <su...@siemens.com>*
> > > > >
> > > >
> > > >
> > > > --
> > > > Neo4j Chief Solutions Architect
> > > > *✉ *matt.casters@neo4j.com
> > > >
> > >
> > >
> > > --
> > > *CEOS Automatización, C.A.*
> > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> > > *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > >
> > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > > *Ing. César García*
> > >
> > > *Cel: +58 414-760.98.95*
> > >
> > > *Hotline Técnica SIEMENS: 0800 1005080*
> > >
> > > *Email: support.aan.automation@siemens.com
> > > <su...@siemens.com>*
> > >
> >
>
>
> --
> *CEOS Automatización, C.A.*
> *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
>
> *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> *Ing. César García*
>
> *Cel: +58 414-760.98.95*
>
> *Hotline Técnica SIEMENS: 0800 1005080*
>
> *Email: support.aan.automation@siemens.com
> <su...@siemens.com>*
>
Re: Apache PLC4X Integration
Posted by Cesar Garcia <ce...@ceos.com.ve>.
Hello Chris,
I Use another method to share the connection. The connection is wrapped
with a ReferenceCountUtil object from the Netty library, which is shared by
the PLC4XTransforms.
I thought that the most laborious thing for the integration of the
libraries would be the handling of types, but everything was almost
transparent.
I am currently attacking a memory leak that affects long-term operation to
consider integrating into Hop.
I am very happy with the integration of both projects,
Kind regards,
El vie, 21 ene 2022 a las 7:54, Christofer Dutz (<cd...@apache.org>)
escribió:
> Hi Hop community and @Cesar,
>
> coming a bit late to the party ... but how about using our Plc4x
> connection-pool/cache for this?
> We could share one instance of the connection pool and then each operation
> leases and returns the connections and the pool should manage the
> concurrency.
>
> If it runs on multiple hosts it would obviously be multiple instances of
> the pool and hereby multiple connections to a PLC (one from each node) ...
> Generally when working with PLCs I wouldn't reccomend to connect a cluster
> to a plc as that's just not how PLCs work (Some only allow 1-3 concurrent
> connections).
>
> Chris
>
>
> On 2021/12/15 13:57:51 Cesar Garcia wrote:
> > Hi Matt,
> >
> > Thank you for your comments,
> >
> > Indeed, as you point out, in this case, these libraries should run
> locally,
> > in order to transfer the information to the pipes.
> >
> > Excellent observation on how to store generic objects in pipes with
> > "getExtensionDataMap" and as you point out, the problem is the life cycle
> > management of objects between pipes (the entertaining part).
> >
> > I will be consulting any dua,
> >
> > Thanks a lot.
> >
> > El mar, 14 dic 2021 a las 14:59, Matt Casters
> > (<ma...@neo4j.com.invalid>) escribió:
> >
> > > Hi César,
> > >
> > > First of all: welcome to the Hop community.
> > >
> > > Usually we make a distinction between the connection metadata
> (hostname,
> > > port, username, password, and all sorts of other options).
> > > Implementing this is fairly easy. Take for example the source code
> > > of the Cassandra
> > > database
> > > <
> > >
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnection.java
> > > >
> > > connection metadata.
> > > In the same package you can find the code for the user interface (
> > > CassandraConnectionEditor.java
> > > <
> > >
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnectionEditor.java
> > > >).
> > > Most of the user interface is generated automatically.
> > > Doing it like this will make a PLC4x connection a first class metadata
> > > citizen and show up in the metadata perspective for example. You'd also
> > > have direct access to the metadata widget in the plugin user
> interfaces.
> > > This makes the lives of the PLC4x users a lot easier.
> > >
> > > For the actual instantiation of a connection at runtime it's true that
> > > usually every transform has its own connection but this doesn't have
> to be
> > > the case.
> > > Transforms can store and retrieve runtime objects in the current
> pipeline
> > > using *IPipelineEngine.getExtensionDataMap()*.
> > > You do need to be careful about race conditions since all transforms
> start
> > > in parallel. Obviously you can also use a singleton design pattern in
> > > Java.
> > >
> > > Also note that this connection sharing would work for a local Hop
> execution
> > > engine but not for Beam where processing happens typically in a
> massively
> > > parallel fashion.
> > > While technically the pipelines would still work there would be no
> > > immediate way for the implementation to limit the number of
> connections.
> > >
> > > Please let us know if you need any help with the creation of PLC4x
> > > connectivity plugins themselves. We'd be happy to help out.
> > >
> > > Kind regard,
> > >
> > > Matt
> > >
> > > On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <cesar.garcia@ceos.com.ve
> >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > My name is Cesar Garcia, I am part of the Apache PLC4x team.
> > > >
> > > > I have been evaluating integrating the PLC4x communication libraries
> to
> > > the
> > > > HOP environment, but I have the following doubts:
> > > >
> > > > 1. It is possible to share a connection between all pipelines.
> > > > 2. In this aspect, it is correct to create the connection in an
> "Action"
> > > > and share it through "PluginRegistry.getInstance ()".
> > > >
> > > > The pattern of use is that each pipelines creates its connection to
> the
> > > > databases, in our type of application the connection to the devices
> is
> > > > limited, therefore they must share the resource.
> > > >
> > > > Grateful for your comments,
> > > >
> > > > Best regards,
> > > >
> > > > --
> > > > *CEOS Automatización, C.A.*
> > > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> > > > *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > > >
> > > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > > > *Ing. César García*
> > > >
> > > > *Cel: +58 414-760.98.95*
> > > >
> > > > *Hotline Técnica SIEMENS: 0800 1005080*
> > > >
> > > > *Email: support.aan.automation@siemens.com
> > > > <su...@siemens.com>*
> > > >
> > >
> > >
> > > --
> > > Neo4j Chief Solutions Architect
> > > *✉ *matt.casters@neo4j.com
> > >
> >
> >
> > --
> > *CEOS Automatización, C.A.*
> > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> > *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> >
> > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > *Ing. César García*
> >
> > *Cel: +58 414-760.98.95*
> >
> > *Hotline Técnica SIEMENS: 0800 1005080*
> >
> > *Email: support.aan.automation@siemens.com
> > <su...@siemens.com>*
> >
>
--
*CEOS Automatización, C.A.*
*GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
*PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
*FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
*Ing. César García*
*Cel: +58 414-760.98.95*
*Hotline Técnica SIEMENS: 0800 1005080*
*Email: support.aan.automation@siemens.com
<su...@siemens.com>*
Re: Apache PLC4X Integration
Posted by Christofer Dutz <cd...@apache.org>.
Hi Hop community and @Cesar,
coming a bit late to the party ... but how about using our Plc4x connection-pool/cache for this?
We could share one instance of the connection pool and then each operation leases and returns the connections and the pool should manage the concurrency.
If it runs on multiple hosts it would obviously be multiple instances of the pool and hereby multiple connections to a PLC (one from each node) ... Generally when working with PLCs I wouldn't reccomend to connect a cluster to a plc as that's just not how PLCs work (Some only allow 1-3 concurrent connections).
Chris
On 2021/12/15 13:57:51 Cesar Garcia wrote:
> Hi Matt,
>
> Thank you for your comments,
>
> Indeed, as you point out, in this case, these libraries should run locally,
> in order to transfer the information to the pipes.
>
> Excellent observation on how to store generic objects in pipes with
> "getExtensionDataMap" and as you point out, the problem is the life cycle
> management of objects between pipes (the entertaining part).
>
> I will be consulting any dua,
>
> Thanks a lot.
>
> El mar, 14 dic 2021 a las 14:59, Matt Casters
> (<ma...@neo4j.com.invalid>) escribió:
>
> > Hi César,
> >
> > First of all: welcome to the Hop community.
> >
> > Usually we make a distinction between the connection metadata (hostname,
> > port, username, password, and all sorts of other options).
> > Implementing this is fairly easy. Take for example the source code
> > of the Cassandra
> > database
> > <
> > https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnection.java
> > >
> > connection metadata.
> > In the same package you can find the code for the user interface (
> > CassandraConnectionEditor.java
> > <
> > https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnectionEditor.java
> > >).
> > Most of the user interface is generated automatically.
> > Doing it like this will make a PLC4x connection a first class metadata
> > citizen and show up in the metadata perspective for example. You'd also
> > have direct access to the metadata widget in the plugin user interfaces.
> > This makes the lives of the PLC4x users a lot easier.
> >
> > For the actual instantiation of a connection at runtime it's true that
> > usually every transform has its own connection but this doesn't have to be
> > the case.
> > Transforms can store and retrieve runtime objects in the current pipeline
> > using *IPipelineEngine.getExtensionDataMap()*.
> > You do need to be careful about race conditions since all transforms start
> > in parallel. Obviously you can also use a singleton design pattern in
> > Java.
> >
> > Also note that this connection sharing would work for a local Hop execution
> > engine but not for Beam where processing happens typically in a massively
> > parallel fashion.
> > While technically the pipelines would still work there would be no
> > immediate way for the implementation to limit the number of connections.
> >
> > Please let us know if you need any help with the creation of PLC4x
> > connectivity plugins themselves. We'd be happy to help out.
> >
> > Kind regard,
> >
> > Matt
> >
> > On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <ce...@ceos.com.ve>
> > wrote:
> >
> > > Hi all,
> > >
> > > My name is Cesar Garcia, I am part of the Apache PLC4x team.
> > >
> > > I have been evaluating integrating the PLC4x communication libraries to
> > the
> > > HOP environment, but I have the following doubts:
> > >
> > > 1. It is possible to share a connection between all pipelines.
> > > 2. In this aspect, it is correct to create the connection in an "Action"
> > > and share it through "PluginRegistry.getInstance ()".
> > >
> > > The pattern of use is that each pipelines creates its connection to the
> > > databases, in our type of application the connection to the devices is
> > > limited, therefore they must share the resource.
> > >
> > > Grateful for your comments,
> > >
> > > Best regards,
> > >
> > > --
> > > *CEOS Automatización, C.A.*
> > > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> > > *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> > >
> > > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > > *Ing. César García*
> > >
> > > *Cel: +58 414-760.98.95*
> > >
> > > *Hotline Técnica SIEMENS: 0800 1005080*
> > >
> > > *Email: support.aan.automation@siemens.com
> > > <su...@siemens.com>*
> > >
> >
> >
> > --
> > Neo4j Chief Solutions Architect
> > *✉ *matt.casters@neo4j.com
> >
>
>
> --
> *CEOS Automatización, C.A.*
> *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
>
> *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> *Ing. César García*
>
> *Cel: +58 414-760.98.95*
>
> *Hotline Técnica SIEMENS: 0800 1005080*
>
> *Email: support.aan.automation@siemens.com
> <su...@siemens.com>*
>
Re: Apache PLC4X Integration
Posted by Cesar Garcia <ce...@ceos.com.ve>.
Hi Matt,
Thank you for your comments,
Indeed, as you point out, in this case, these libraries should run locally,
in order to transfer the information to the pipes.
Excellent observation on how to store generic objects in pipes with
"getExtensionDataMap" and as you point out, the problem is the life cycle
management of objects between pipes (the entertaining part).
I will be consulting any dua,
Thanks a lot.
El mar, 14 dic 2021 a las 14:59, Matt Casters
(<ma...@neo4j.com.invalid>) escribió:
> Hi César,
>
> First of all: welcome to the Hop community.
>
> Usually we make a distinction between the connection metadata (hostname,
> port, username, password, and all sorts of other options).
> Implementing this is fairly easy. Take for example the source code
> of the Cassandra
> database
> <
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnection.java
> >
> connection metadata.
> In the same package you can find the code for the user interface (
> CassandraConnectionEditor.java
> <
> https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnectionEditor.java
> >).
> Most of the user interface is generated automatically.
> Doing it like this will make a PLC4x connection a first class metadata
> citizen and show up in the metadata perspective for example. You'd also
> have direct access to the metadata widget in the plugin user interfaces.
> This makes the lives of the PLC4x users a lot easier.
>
> For the actual instantiation of a connection at runtime it's true that
> usually every transform has its own connection but this doesn't have to be
> the case.
> Transforms can store and retrieve runtime objects in the current pipeline
> using *IPipelineEngine.getExtensionDataMap()*.
> You do need to be careful about race conditions since all transforms start
> in parallel. Obviously you can also use a singleton design pattern in
> Java.
>
> Also note that this connection sharing would work for a local Hop execution
> engine but not for Beam where processing happens typically in a massively
> parallel fashion.
> While technically the pipelines would still work there would be no
> immediate way for the implementation to limit the number of connections.
>
> Please let us know if you need any help with the creation of PLC4x
> connectivity plugins themselves. We'd be happy to help out.
>
> Kind regard,
>
> Matt
>
> On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <ce...@ceos.com.ve>
> wrote:
>
> > Hi all,
> >
> > My name is Cesar Garcia, I am part of the Apache PLC4x team.
> >
> > I have been evaluating integrating the PLC4x communication libraries to
> the
> > HOP environment, but I have the following doubts:
> >
> > 1. It is possible to share a connection between all pipelines.
> > 2. In this aspect, it is correct to create the connection in an "Action"
> > and share it through "PluginRegistry.getInstance ()".
> >
> > The pattern of use is that each pipelines creates its connection to the
> > databases, in our type of application the connection to the devices is
> > limited, therefore they must share the resource.
> >
> > Grateful for your comments,
> >
> > Best regards,
> >
> > --
> > *CEOS Automatización, C.A.*
> > *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> > *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
> >
> > *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> > *Ing. César García*
> >
> > *Cel: +58 414-760.98.95*
> >
> > *Hotline Técnica SIEMENS: 0800 1005080*
> >
> > *Email: support.aan.automation@siemens.com
> > <su...@siemens.com>*
> >
>
>
> --
> Neo4j Chief Solutions Architect
> *✉ *matt.casters@neo4j.com
>
--
*CEOS Automatización, C.A.*
*GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
*PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
*FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
*Ing. César García*
*Cel: +58 414-760.98.95*
*Hotline Técnica SIEMENS: 0800 1005080*
*Email: support.aan.automation@siemens.com
<su...@siemens.com>*
Re: Apache PLC4X Integration
Posted by Matt Casters <ma...@neo4j.com.INVALID>.
Hi César,
First of all: welcome to the Hop community.
Usually we make a distinction between the connection metadata (hostname,
port, username, password, and all sorts of other options).
Implementing this is fairly easy. Take for example the source code
of the Cassandra
database
<https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnection.java>
connection metadata.
In the same package you can find the code for the user interface (
CassandraConnectionEditor.java
<https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnectionEditor.java>).
Most of the user interface is generated automatically.
Doing it like this will make a PLC4x connection a first class metadata
citizen and show up in the metadata perspective for example. You'd also
have direct access to the metadata widget in the plugin user interfaces.
This makes the lives of the PLC4x users a lot easier.
For the actual instantiation of a connection at runtime it's true that
usually every transform has its own connection but this doesn't have to be
the case.
Transforms can store and retrieve runtime objects in the current pipeline
using *IPipelineEngine.getExtensionDataMap()*.
You do need to be careful about race conditions since all transforms start
in parallel. Obviously you can also use a singleton design pattern in Java.
Also note that this connection sharing would work for a local Hop execution
engine but not for Beam where processing happens typically in a massively
parallel fashion.
While technically the pipelines would still work there would be no
immediate way for the implementation to limit the number of connections.
Please let us know if you need any help with the creation of PLC4x
connectivity plugins themselves. We'd be happy to help out.
Kind regard,
Matt
On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <ce...@ceos.com.ve>
wrote:
> Hi all,
>
> My name is Cesar Garcia, I am part of the Apache PLC4x team.
>
> I have been evaluating integrating the PLC4x communication libraries to the
> HOP environment, but I have the following doubts:
>
> 1. It is possible to share a connection between all pipelines.
> 2. In this aspect, it is correct to create the connection in an "Action"
> and share it through "PluginRegistry.getInstance ()".
>
> The pattern of use is that each pipelines creates its connection to the
> databases, in our type of application the connection to the devices is
> limited, therefore they must share the resource.
>
> Grateful for your comments,
>
> Best regards,
>
> --
> *CEOS Automatización, C.A.*
> *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
>
> *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> *Ing. César García*
>
> *Cel: +58 414-760.98.95*
>
> *Hotline Técnica SIEMENS: 0800 1005080*
>
> *Email: support.aan.automation@siemens.com
> <su...@siemens.com>*
>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: Apache PLC4X Integration
Posted by Matt Casters <ma...@neo4j.com.INVALID>.
Hi César,
First of all: welcome to the Hop community.
Usually we make a distinction between the connection metadata (hostname,
port, username, password, and all sorts of other options).
Implementing this is fairly easy. Take for example the source code
of the Cassandra
database
<https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnection.java>
connection metadata.
In the same package you can find the code for the user interface (
CassandraConnectionEditor.java
<https://github.com/apache/incubator-hop/blob/master/plugins/tech/cassandra/src/main/java/org/apache/hop/databases/cassandra/metadata/CassandraConnectionEditor.java>).
Most of the user interface is generated automatically.
Doing it like this will make a PLC4x connection a first class metadata
citizen and show up in the metadata perspective for example. You'd also
have direct access to the metadata widget in the plugin user interfaces.
This makes the lives of the PLC4x users a lot easier.
For the actual instantiation of a connection at runtime it's true that
usually every transform has its own connection but this doesn't have to be
the case.
Transforms can store and retrieve runtime objects in the current pipeline
using *IPipelineEngine.getExtensionDataMap()*.
You do need to be careful about race conditions since all transforms start
in parallel. Obviously you can also use a singleton design pattern in Java.
Also note that this connection sharing would work for a local Hop execution
engine but not for Beam where processing happens typically in a massively
parallel fashion.
While technically the pipelines would still work there would be no
immediate way for the implementation to limit the number of connections.
Please let us know if you need any help with the creation of PLC4x
connectivity plugins themselves. We'd be happy to help out.
Kind regard,
Matt
On Tue, Dec 14, 2021 at 4:12 PM Cesar Garcia <ce...@ceos.com.ve>
wrote:
> Hi all,
>
> My name is Cesar Garcia, I am part of the Apache PLC4x team.
>
> I have been evaluating integrating the PLC4x communication libraries to the
> HOP environment, but I have the following doubts:
>
> 1. It is possible to share a connection between all pipelines.
> 2. In this aspect, it is correct to create the connection in an "Action"
> and share it through "PluginRegistry.getInstance ()".
>
> The pattern of use is that each pipelines creates its connection to the
> databases, in our type of application the connection to the devices is
> limited, therefore they must share the resource.
>
> Grateful for your comments,
>
> Best regards,
>
> --
> *CEOS Automatización, C.A.*
> *GALPON SERVICIO INDUSTRIALES Y NAVALES FA, C.A.,*
> *PISO 1, OFICINA 2, AV. RAUL LEONI, SECTOR GUAMACHITO,*
>
> *FRENTE A LA ASOCIACION DE GANADEROS,BARCELONA,EDO. ANZOATEGUI*
> *Ing. César García*
>
> *Cel: +58 414-760.98.95*
>
> *Hotline Técnica SIEMENS: 0800 1005080*
>
> *Email: support.aan.automation@siemens.com
> <su...@siemens.com>*
>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com