You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by "bossenti (via GitHub)" <gi...@apache.org> on 2023/06/02 18:57:32 UTC
[D] Action Plan OPC-UA Adapter π (streampipes)
GitHub user bossenti created a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency. This should include:
- [ ] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [x] remove `nodeId` & `namespaceIndex` from `SpOocUaConfig`
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
- [ ] https://github.com/apache/streampipes/issues/1693
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [x] https://github.com/apache/streampipes/issues/1662
- [ ] https://github.com/apache/streampipes/issues/1713
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi thanks a lot for reporting this.
In the current version it should be possible to us a pipeline element for arrays to transform the array into a basic value.
However, I like the idea of adding such a feature. But we need to make sure that it works as expected for all cases.
How many elements can be part of the array in your case? Is it always one?
Cheers,
Philipp
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6176772
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "curichan (via GitHub)" <gi...@apache.org>.
GitHub user curichan added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi @tenthe,
https://reference.opcfoundation.org/Core/Part3/v105/docs/5.2.5 would imply that the provision of a display name is optional.
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6149002
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Action Plan OPC-UA Adapter π
Ok thanks, in this case we need to discuss how we can achieve this.
Since the event schema of a stream created by the adapter is static, this means that we cannot dynamically change the amount of properties. In the case when there is only one entry in the array we could map it to a double property in the event.
What exactly is the data type of the field? On the screenshot it looks like it is `Double` or is it `Double Array`?
In case it is `Double Array` we could represent it as an array of doubles in the event. However, this will not solve the problem for the data lake and the data explorer π€
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6187438
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [ ] Unify the labels for adapter & sink (#1664)
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [ ] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi,
I've started to work on asynchronous browsing. A screenshot of the current prototype is below. Users can browse through all nodes, see details on every node and nodes can be selected before navigating to the next step of the StreamPipes Connect wizard.
Do you have any feedback/wishes? Especially I'd be interested in the metadata that should be shown when selecting a node - what metadata would be interesting/important to have in this section?
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6129739
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [x] remove `nodeId` & `namespaceIndex` from `SpOocUaConfig`
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
- [ ] https://github.com/apache/streampipes/issues/1693
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [x] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "curichan (via GitHub)" <gi...@apache.org>.
GitHub user curichan added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi @dominikriemer,
many thanks for this! I think this will be a huge improvement. It may be worth showing the current value of the variable as well.
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6148457
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [ ] remove `nodeId` & `namespaceIndex` from `SpOocUaConfig`
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [x] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [ ] remove `nodeId` & `namespaceIndex` from `SpOocUaConfig`
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [ ] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [x] remove `nodeId` & `namespaceIndex` from `SpOocUaConfig`
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
- [ ] https://github.com/apache/streampipes/issues/1693
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [x] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "curichan (via GitHub)" <gi...@apache.org>.
GitHub user curichan added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi @tenthe,
the workaround using a pipeline element hadn't occured to me. Thanks for that! Turns out in my case the number of elements in the array can change during runtime as it depends on the production process currently active in the machine feeding the OPC UA server. So a more general way of dealing with such a situation might be called for...
Cheers,
Christian.
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6182534
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "curichan (via GitHub)" <gi...@apache.org>.
GitHub user curichan added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi @tenthe,
here a more complete screenshot:
My understanding is that in this particular case the data type is set to "`Double`", but there is a node property "`is_array`" which is "`true`". So I assume that OPC UA allows any data type to be associated with arrays, not just `Double`. So may be the way forward would be to represent the value of a node as an array in the event and really delegate dealing with arrays to pipeline elements?
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6193071
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "curichan (via GitHub)" <gi...@apache.org>.
GitHub user curichan edited a comment on the discussion: Action Plan OPC-UA Adapter π
Hi @dominikriemer,
many thanks for this! I think this will be a huge improvement. It may be worth showing the current value of the variable as well.
Also the timestamp of the last update of the node would be useful. Sometimes there are "plausible" values reported, but they are "old". Seeing the timestamp of the current value would help to see whether a node is "stale" or active.
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6148457
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "vogler75 (via GitHub)" <gi...@apache.org>.
GitHub user vogler75 added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi Dominik,
Looks good! For the DataType it would be good to see the data-type instead of this NodeId.
DisplayName would also be good - if you havenβt used this in the tree ;-)
regards,
Andreas
> Am 09.06.2023 um 09:04 schrieb Dominik Riemer ***@***.***>:
>
>
> Hi,
> I've started to work on asynchronous browsing. A screenshot of the current prototype is below. Users can browse through all nodes, see details on every node and nodes can be selected before navigating to the next step of the StreamPipes Connect wizard.
>
> Do you have any feedback/wishes? Especially I'd be interested in the metadata that should be shown when selecting a node - what metadata would be interesting/important to have in this section?
>
> <https://user-images.githubusercontent.com/33908476/244617252-5bfb8bfb-b247-4612-83d9-cce4d0baec2f.png>
> β
> Reply to this email directly, view it on GitHub <https://github.com/apache/streampipes/discussions/1650#discussioncomment-6129739>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGEZIGDYE45QNTRTM4Q6BKTXKLDJTANCNFSM6AAAAAAYYVVBUY>.
> You are receiving this because you were mentioned.
>
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6129808
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [ ] Unify the labels for adapter & sink (#1664)
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [ ] Unify the labels for adapter & sink
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [ ] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Action Plan OPC-UA Adapter π
Yes, I think that's a good idea. I'll add an issue for this to the task above.
If arrays are a common case, we can also think about including a visualization for array values in the Data Explorer. But that is another discussion.
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6193310
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "curichan (via GitHub)" <gi...@apache.org>.
GitHub user curichan added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi all,
not sure this really belongs here, but I also noticed that Streampipes has a problem dealing with node values of type (Double) array:
The adapter creation page reads the node content correctly:
The result of reading this variable is shown correctly as an array with a single element. However, there is currently no way to "extract" this one element as an individual value. So, trying to use the array values in further processing, like storing in the local data lake or displaying the value in a visual in a dashboard or the data explorer is not possible. No error messages are shown. The values are simply not stored or displayed. I think it'd be useful to add a possibility to edit arrays in the "configure fields" step of adapter creation. It should be possible to store (may be not display) arrays in the local data lake as they are. But it should also be possible to configure the array field such that individual elements of the array are extracted, stored and displayed as individual values.
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6170814
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [ ] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [x] remove `nodeId` & `namespaceIndex` from `SpOocUaConfig`
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [x] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Action Plan OPC-UA Adapter π
Hi @vogler75,
thank you very much for your reply.
As discussed with @curichan, we also want to support all identifier types (see #1579) in the future.
Do you know if the display name is always available regardless of the identifier type?
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6130930
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Action Plan OPC-UA Adapter π (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Action Plan OPC-UA Adapter π
Just a quick addition. My comment referred to the data type.
So how can you display the data type independently of the identifier type.
Sorry for the confusion.
GitHub link: https://github.com/apache/streampipes/discussions/1650#discussioncomment-6131161
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti edited a discussion: Action Plan OPC-UA Adapter π
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti edited a discussion: Action Plan OPC-UA Adapter π
During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (`0.93.0`).
I browsed through the existing issues and discussions and came up with the following overview.
**1) Migrate existing OPC UA adapter and sink to a common module**
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for [influx](https://github.com/apache/streampipes/tree/dev/streampipes-extensions/streampipes-connectors-influx)). This should include:
- [x] the actual refactoring to a shared module (https://github.com/apache/streampipes/issues/1648)
- [x] Unify the labels for adapter & sink (#1664)
- [x] remove `nodeId` & `namespaceIndex` from `SpOocUaConfig`
- [ ] https://github.com/apache/streampipes/issues/1693
- [ ] add description to README how adapters can be integrated that build upon the OPC UA adapter
**2) Pull & Subscription handling**
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.
In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two `x` nodes, then updates for all nodes can result in up to `x` events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.
**3) Support OPC UA timestamps**
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: https://github.com/apache/streampipes/discussions/886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)
- [ ] https://github.com/apache/streampipes/issues/889
**4) Fully support OPC UA types**
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.
- [ ] https://github.com/apache/streampipes/issues/1579
- [ ] https://github.com/apache/streampipes/issues/1693
**5) Remove Requirement of NodeId (& namespace)**
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: https://github.com/apache/streampipes/discussions/888 & ttps://github.com/apache/streampipes/discussions/890
This requires a asynchronous browsing of available nodes (see **8)**.
**6) Connection Caching**
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See https://github.com/apache/streampipes/discussions/888
**7) Support encrypted connection**
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See https://github.com/apache/streampipes/discussions/1449
**8) Load available nodes on demand**
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See https://github.com/apache/streampipes/discussions/895
- [x] https://github.com/apache/streampipes/issues/1662
**9) Add integration test**
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?
**10) Handling null values**
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: https://github.com/apache/streampipes/discussions/1596 & https://github.com/apache/streampipes/issues/1485
**11) Celebrate new OPC UA adapter**
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.
- [ ] https://github.com/apache/streampipes/issues/1551
In the midterm we could also think about having a look at the companion specifications and support some aspects there.
This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.
cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe
GitHub link: https://github.com/apache/streampipes/discussions/1650
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org