You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@plc4x.apache.org by Julian Feinauer <j....@pragmaticminds.de> on 2018/07/26 11:59:18 UTC
Warm welcome and Batching
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hey Chris,
this is great!
We here have only access to one S7 1500 and a Modbus device.
Can you create an account for me?
Or do I have to go through a more formal request process? : )
Best
Julian
Am 31.07.18, 14:51 schrieb "Christofer Dutz" <ch...@c-ware.de>:
And I think I forgot one important thing ...
If you would like to communicate with real PLCs, we have setup a VPN in our Frankfurt office where we have all sorts of different PLC types connected.
Any used wanting access to this, can request an account off list and I'll take care of having that setup.
At least for this part I have the documentation online ;-)
http://plc4x.apache.org/developers/vpn.html
Chris
Am 27.07.18, 16:52 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey Chris,
please, take my words more as observations than as criticism.
I am really impressed by the current state of the project with regard to coding styles and especially architecture. And I think it is great that your company supports this project that massively.
I had now a deep look inside the code (unfortunately I have no S7 here for some "live" debugging) and I think I understand the control flow now. I never used netty before and was a bit confused by the message conversion and it took me a while to get the complete flow from request to the S7 ; )
And of course I totally understand that one concentrates on features more than on documentation (who doesn't?).
And if it's okay for you I would open a branch and add some comments or small refactorings that I see while going through the code that you can then review and hopefully integrate.
How do you currently manage the contributions, feature branches and PRs or repository forking?
I also observed that the jira seems a bit orphaned (again no criticism, just observation!) and if it's okay for you I will open one or two tickets for the our needs (the batch processing) that we can have a discussion or specification there.
Best
Julian
Am 27.07.18, 16:01 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
you are absolutely correct, that our documentation leaves quite some room for improvement ;-)
Currently being funded by my company for working on this, I was concentrating on getting PLC4X to an initially usable state and have to admit that I did sacrifice documentation for features in the past.
But if we want to grow the community, documentation is of essential value. I promise to start writing more documentation. Probably some simple user documentation would be the best starting point.
I will increase my efforts ... promised :-)
For now, you could have a look at the Kafka Bridge example as this builds a PlcReadRequest with multiple items (separate addresses).
Chris
Am 26.07.18, 16:54 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hi Chris and hi Sebastian,
thank both of you for the happy welcoming.
@Chris
Thank you for your hints, I went a bit into the code but the thing I'm currently missing (is there a central documentation) is how one can request a larger portion of memory (possible even multiple PDUs).
As my understanding of the API is that I can only request single Values?
Where is the best place to get a good insight into the API (docu or code)?
Thanks!
Julian
PS.: I'm also mathematician but went more for statistics and number theory (beautiful but mostly unusable) but we have done some stochastic optimization (which is especially strong for high dimensional problems) so I can definitively try to assist in solving the problem
Am 26.07.18, 14:51 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
welcome to our list ( ... reading your post definitely made my day (Even if I thought it couldn't get better)
Regarding your first Idea ... this is definitely on my mind and was taken into consideration when implementing the S7 driver.
Right now the S7MessageProcessor interface provides an extension point for things like this. Currently there is only one implementation. The DefaultS7MessageProcessor is used to automatically split up large reads and writes into multiple requests depending on the negotiated max PDU size. This would be the place to re-write the S7Message requests to do what you are planning on doing. I was planning on getting a friend of mine on board to address this sort of thing (She's a mathematician, specialized on packing and optimization problems), but if you are interested and want to work on this, I would be glad to help you with this. I think the only thing currently not supported, is to read information which would exceed the PDU size if sent in a single PDU.
Regarding batch reads and writes ... I think such a thing would be a great extension. However we would need to be able to configure this sort of thing. As this could be something that might confuse people, manually turning batch reads/writes on and configuring this in the connection string would be a good way to go. The way PLC4X is built up, it shouldn't be a problem to implement something like this. The Plc4XS7Protocol class could simply queue all read/write items and have a separate worker drain that queue. This could be similar to the queuing implemented in the S7Protocol class, just the trigger would be a different one (Eventually the driver could send some sort of Batch operation heartbeat event). Here too I would be more than happy to help implementing this.
Chris
Am 26.07.18, 13:59 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Christofer Dutz <ch...@c-ware.de>.
And I think I forgot one important thing ...
If you would like to communicate with real PLCs, we have setup a VPN in our Frankfurt office where we have all sorts of different PLC types connected.
Any used wanting access to this, can request an account off list and I'll take care of having that setup.
At least for this part I have the documentation online ;-)
http://plc4x.apache.org/developers/vpn.html
Chris
Am 27.07.18, 16:52 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey Chris,
please, take my words more as observations than as criticism.
I am really impressed by the current state of the project with regard to coding styles and especially architecture. And I think it is great that your company supports this project that massively.
I had now a deep look inside the code (unfortunately I have no S7 here for some "live" debugging) and I think I understand the control flow now. I never used netty before and was a bit confused by the message conversion and it took me a while to get the complete flow from request to the S7 ; )
And of course I totally understand that one concentrates on features more than on documentation (who doesn't?).
And if it's okay for you I would open a branch and add some comments or small refactorings that I see while going through the code that you can then review and hopefully integrate.
How do you currently manage the contributions, feature branches and PRs or repository forking?
I also observed that the jira seems a bit orphaned (again no criticism, just observation!) and if it's okay for you I will open one or two tickets for the our needs (the batch processing) that we can have a discussion or specification there.
Best
Julian
Am 27.07.18, 16:01 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
you are absolutely correct, that our documentation leaves quite some room for improvement ;-)
Currently being funded by my company for working on this, I was concentrating on getting PLC4X to an initially usable state and have to admit that I did sacrifice documentation for features in the past.
But if we want to grow the community, documentation is of essential value. I promise to start writing more documentation. Probably some simple user documentation would be the best starting point.
I will increase my efforts ... promised :-)
For now, you could have a look at the Kafka Bridge example as this builds a PlcReadRequest with multiple items (separate addresses).
Chris
Am 26.07.18, 16:54 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hi Chris and hi Sebastian,
thank both of you for the happy welcoming.
@Chris
Thank you for your hints, I went a bit into the code but the thing I'm currently missing (is there a central documentation) is how one can request a larger portion of memory (possible even multiple PDUs).
As my understanding of the API is that I can only request single Values?
Where is the best place to get a good insight into the API (docu or code)?
Thanks!
Julian
PS.: I'm also mathematician but went more for statistics and number theory (beautiful but mostly unusable) but we have done some stochastic optimization (which is especially strong for high dimensional problems) so I can definitively try to assist in solving the problem
Am 26.07.18, 14:51 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
welcome to our list ( ... reading your post definitely made my day (Even if I thought it couldn't get better)
Regarding your first Idea ... this is definitely on my mind and was taken into consideration when implementing the S7 driver.
Right now the S7MessageProcessor interface provides an extension point for things like this. Currently there is only one implementation. The DefaultS7MessageProcessor is used to automatically split up large reads and writes into multiple requests depending on the negotiated max PDU size. This would be the place to re-write the S7Message requests to do what you are planning on doing. I was planning on getting a friend of mine on board to address this sort of thing (She's a mathematician, specialized on packing and optimization problems), but if you are interested and want to work on this, I would be glad to help you with this. I think the only thing currently not supported, is to read information which would exceed the PDU size if sent in a single PDU.
Regarding batch reads and writes ... I think such a thing would be a great extension. However we would need to be able to configure this sort of thing. As this could be something that might confuse people, manually turning batch reads/writes on and configuring this in the connection string would be a good way to go. The way PLC4X is built up, it shouldn't be a problem to implement something like this. The Plc4XS7Protocol class could simply queue all read/write items and have a separate worker drain that queue. This could be similar to the queuing implemented in the S7Protocol class, just the trigger would be a different one (Eventually the driver could send some sort of Batch operation heartbeat event). Here too I would be more than happy to help implementing this.
Chris
Am 26.07.18, 13:59 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Julian,
I wasn't taking them as criticism ... no worries ;-) ... your observations are more than welcome as you do point out things I might have been missing. So please continue ...
And regarding Jira ... I did start writing down some low-hanging-fruit issues recently. But due to the fact that currently it was mostly Sebastian and me working on things,
I think we sort of didn't write down things we found, just to immediately fix them and rather just fixed them as they came by. If you do find things, Issues are more than welcome.
All you need to do, is to sign-up for an account
(I would recommend checking if an apache-id is already taken here http://people.apache.org/committer-index.html
when choosing a Jira name, as a matching Jira and apache username does simplify things in the future)
Thank you for your kind words about what we have achieved till now :-)
If you want to contribute, the probably simplest workflow would be to fork the GitHub repo at:
https://github.com/apache/incubator-plc4x using the GitHub web interface, checkout your fork locally and do all operations (commits, pushes etc.) on that.
Here you could create your branch and as soon as you are satisfied with your changes, you would create a pull request and we could merge those changes into the official repository.
If this is just a minor change or two this would be all which is required. If the changes or additions are more than some minor changes, we would require you to sign a so-called ICLA (Individual Contributor License Agreement) (This donates your contributions to Apache projects to the Apache Software Foundation) or if operating on behalf of a company, additionally a CCLA (Corporate Contributor License Agreement) (The latter is usually required as most employment contracts include paragraphs, that all code created at work belongs to the company someone works for. With the CCLA the company hands over the contributions to Apache projects to the Apache Software Foundation)
You can find the contract forms here:
https://www.apache.org/licenses/icla.pdf
https://www.apache.org/licenses/cla-corporate.txt
Really looking forward to your input and contributions.
With kind regards and a great weekend,
Chris
PS: I should also add this information to another orphaned page on our website ;-) (http://plc4x.apache.org/developers/contributing.html)
Am 27.07.18, 16:52 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey Chris,
please, take my words more as observations than as criticism.
I am really impressed by the current state of the project with regard to coding styles and especially architecture. And I think it is great that your company supports this project that massively.
I had now a deep look inside the code (unfortunately I have no S7 here for some "live" debugging) and I think I understand the control flow now. I never used netty before and was a bit confused by the message conversion and it took me a while to get the complete flow from request to the S7 ; )
And of course I totally understand that one concentrates on features more than on documentation (who doesn't?).
And if it's okay for you I would open a branch and add some comments or small refactorings that I see while going through the code that you can then review and hopefully integrate.
How do you currently manage the contributions, feature branches and PRs or repository forking?
I also observed that the jira seems a bit orphaned (again no criticism, just observation!) and if it's okay for you I will open one or two tickets for the our needs (the batch processing) that we can have a discussion or specification there.
Best
Julian
Am 27.07.18, 16:01 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
you are absolutely correct, that our documentation leaves quite some room for improvement ;-)
Currently being funded by my company for working on this, I was concentrating on getting PLC4X to an initially usable state and have to admit that I did sacrifice documentation for features in the past.
But if we want to grow the community, documentation is of essential value. I promise to start writing more documentation. Probably some simple user documentation would be the best starting point.
I will increase my efforts ... promised :-)
For now, you could have a look at the Kafka Bridge example as this builds a PlcReadRequest with multiple items (separate addresses).
Chris
Am 26.07.18, 16:54 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hi Chris and hi Sebastian,
thank both of you for the happy welcoming.
@Chris
Thank you for your hints, I went a bit into the code but the thing I'm currently missing (is there a central documentation) is how one can request a larger portion of memory (possible even multiple PDUs).
As my understanding of the API is that I can only request single Values?
Where is the best place to get a good insight into the API (docu or code)?
Thanks!
Julian
PS.: I'm also mathematician but went more for statistics and number theory (beautiful but mostly unusable) but we have done some stochastic optimization (which is especially strong for high dimensional problems) so I can definitively try to assist in solving the problem
Am 26.07.18, 14:51 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
welcome to our list ( ... reading your post definitely made my day (Even if I thought it couldn't get better)
Regarding your first Idea ... this is definitely on my mind and was taken into consideration when implementing the S7 driver.
Right now the S7MessageProcessor interface provides an extension point for things like this. Currently there is only one implementation. The DefaultS7MessageProcessor is used to automatically split up large reads and writes into multiple requests depending on the negotiated max PDU size. This would be the place to re-write the S7Message requests to do what you are planning on doing. I was planning on getting a friend of mine on board to address this sort of thing (She's a mathematician, specialized on packing and optimization problems), but if you are interested and want to work on this, I would be glad to help you with this. I think the only thing currently not supported, is to read information which would exceed the PDU size if sent in a single PDU.
Regarding batch reads and writes ... I think such a thing would be a great extension. However we would need to be able to configure this sort of thing. As this could be something that might confuse people, manually turning batch reads/writes on and configuring this in the connection string would be a good way to go. The way PLC4X is built up, it shouldn't be a problem to implement something like this. The Plc4XS7Protocol class could simply queue all read/write items and have a separate worker drain that queue. This could be similar to the queuing implemented in the S7Protocol class, just the trigger would be a different one (Eventually the driver could send some sort of Batch operation heartbeat event). Here too I would be more than happy to help implementing this.
Chris
Am 26.07.18, 13:59 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hey Chris,
please, take my words more as observations than as criticism.
I am really impressed by the current state of the project with regard to coding styles and especially architecture. And I think it is great that your company supports this project that massively.
I had now a deep look inside the code (unfortunately I have no S7 here for some "live" debugging) and I think I understand the control flow now. I never used netty before and was a bit confused by the message conversion and it took me a while to get the complete flow from request to the S7 ; )
And of course I totally understand that one concentrates on features more than on documentation (who doesn't?).
And if it's okay for you I would open a branch and add some comments or small refactorings that I see while going through the code that you can then review and hopefully integrate.
How do you currently manage the contributions, feature branches and PRs or repository forking?
I also observed that the jira seems a bit orphaned (again no criticism, just observation!) and if it's okay for you I will open one or two tickets for the our needs (the batch processing) that we can have a discussion or specification there.
Best
Julian
Am 27.07.18, 16:01 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
you are absolutely correct, that our documentation leaves quite some room for improvement ;-)
Currently being funded by my company for working on this, I was concentrating on getting PLC4X to an initially usable state and have to admit that I did sacrifice documentation for features in the past.
But if we want to grow the community, documentation is of essential value. I promise to start writing more documentation. Probably some simple user documentation would be the best starting point.
I will increase my efforts ... promised :-)
For now, you could have a look at the Kafka Bridge example as this builds a PlcReadRequest with multiple items (separate addresses).
Chris
Am 26.07.18, 16:54 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hi Chris and hi Sebastian,
thank both of you for the happy welcoming.
@Chris
Thank you for your hints, I went a bit into the code but the thing I'm currently missing (is there a central documentation) is how one can request a larger portion of memory (possible even multiple PDUs).
As my understanding of the API is that I can only request single Values?
Where is the best place to get a good insight into the API (docu or code)?
Thanks!
Julian
PS.: I'm also mathematician but went more for statistics and number theory (beautiful but mostly unusable) but we have done some stochastic optimization (which is especially strong for high dimensional problems) so I can definitively try to assist in solving the problem
Am 26.07.18, 14:51 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
welcome to our list ( ... reading your post definitely made my day (Even if I thought it couldn't get better)
Regarding your first Idea ... this is definitely on my mind and was taken into consideration when implementing the S7 driver.
Right now the S7MessageProcessor interface provides an extension point for things like this. Currently there is only one implementation. The DefaultS7MessageProcessor is used to automatically split up large reads and writes into multiple requests depending on the negotiated max PDU size. This would be the place to re-write the S7Message requests to do what you are planning on doing. I was planning on getting a friend of mine on board to address this sort of thing (She's a mathematician, specialized on packing and optimization problems), but if you are interested and want to work on this, I would be glad to help you with this. I think the only thing currently not supported, is to read information which would exceed the PDU size if sent in a single PDU.
Regarding batch reads and writes ... I think such a thing would be a great extension. However we would need to be able to configure this sort of thing. As this could be something that might confuse people, manually turning batch reads/writes on and configuring this in the connection string would be a good way to go. The way PLC4X is built up, it shouldn't be a problem to implement something like this. The Plc4XS7Protocol class could simply queue all read/write items and have a separate worker drain that queue. This could be similar to the queuing implemented in the S7Protocol class, just the trigger would be a different one (Eventually the driver could send some sort of Batch operation heartbeat event). Here too I would be more than happy to help implementing this.
Chris
Am 26.07.18, 13:59 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Julian,
you are absolutely correct, that our documentation leaves quite some room for improvement ;-)
Currently being funded by my company for working on this, I was concentrating on getting PLC4X to an initially usable state and have to admit that I did sacrifice documentation for features in the past.
But if we want to grow the community, documentation is of essential value. I promise to start writing more documentation. Probably some simple user documentation would be the best starting point.
I will increase my efforts ... promised :-)
For now, you could have a look at the Kafka Bridge example as this builds a PlcReadRequest with multiple items (separate addresses).
Chris
Am 26.07.18, 16:54 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hi Chris and hi Sebastian,
thank both of you for the happy welcoming.
@Chris
Thank you for your hints, I went a bit into the code but the thing I'm currently missing (is there a central documentation) is how one can request a larger portion of memory (possible even multiple PDUs).
As my understanding of the API is that I can only request single Values?
Where is the best place to get a good insight into the API (docu or code)?
Thanks!
Julian
PS.: I'm also mathematician but went more for statistics and number theory (beautiful but mostly unusable) but we have done some stochastic optimization (which is especially strong for high dimensional problems) so I can definitively try to assist in solving the problem
Am 26.07.18, 14:51 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
welcome to our list ( ... reading your post definitely made my day (Even if I thought it couldn't get better)
Regarding your first Idea ... this is definitely on my mind and was taken into consideration when implementing the S7 driver.
Right now the S7MessageProcessor interface provides an extension point for things like this. Currently there is only one implementation. The DefaultS7MessageProcessor is used to automatically split up large reads and writes into multiple requests depending on the negotiated max PDU size. This would be the place to re-write the S7Message requests to do what you are planning on doing. I was planning on getting a friend of mine on board to address this sort of thing (She's a mathematician, specialized on packing and optimization problems), but if you are interested and want to work on this, I would be glad to help you with this. I think the only thing currently not supported, is to read information which would exceed the PDU size if sent in a single PDU.
Regarding batch reads and writes ... I think such a thing would be a great extension. However we would need to be able to configure this sort of thing. As this could be something that might confuse people, manually turning batch reads/writes on and configuring this in the connection string would be a good way to go. The way PLC4X is built up, it shouldn't be a problem to implement something like this. The Plc4XS7Protocol class could simply queue all read/write items and have a separate worker drain that queue. This could be similar to the queuing implemented in the S7Protocol class, just the trigger would be a different one (Eventually the driver could send some sort of Batch operation heartbeat event). Here too I would be more than happy to help implementing this.
Chris
Am 26.07.18, 13:59 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hi Chris and hi Sebastian,
thank both of you for the happy welcoming.
@Chris
Thank you for your hints, I went a bit into the code but the thing I'm currently missing (is there a central documentation) is how one can request a larger portion of memory (possible even multiple PDUs).
As my understanding of the API is that I can only request single Values?
Where is the best place to get a good insight into the API (docu or code)?
Thanks!
Julian
PS.: I'm also mathematician but went more for statistics and number theory (beautiful but mostly unusable) but we have done some stochastic optimization (which is especially strong for high dimensional problems) so I can definitively try to assist in solving the problem
Am 26.07.18, 14:51 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Julian,
welcome to our list ( ... reading your post definitely made my day (Even if I thought it couldn't get better)
Regarding your first Idea ... this is definitely on my mind and was taken into consideration when implementing the S7 driver.
Right now the S7MessageProcessor interface provides an extension point for things like this. Currently there is only one implementation. The DefaultS7MessageProcessor is used to automatically split up large reads and writes into multiple requests depending on the negotiated max PDU size. This would be the place to re-write the S7Message requests to do what you are planning on doing. I was planning on getting a friend of mine on board to address this sort of thing (She's a mathematician, specialized on packing and optimization problems), but if you are interested and want to work on this, I would be glad to help you with this. I think the only thing currently not supported, is to read information which would exceed the PDU size if sent in a single PDU.
Regarding batch reads and writes ... I think such a thing would be a great extension. However we would need to be able to configure this sort of thing. As this could be something that might confuse people, manually turning batch reads/writes on and configuring this in the connection string would be a good way to go. The way PLC4X is built up, it shouldn't be a problem to implement something like this. The Plc4XS7Protocol class could simply queue all read/write items and have a separate worker drain that queue. This could be similar to the queuing implemented in the S7Protocol class, just the trigger would be a different one (Eventually the driver could send some sort of Batch operation heartbeat event). Here too I would be more than happy to help implementing this.
Chris
Am 26.07.18, 13:59 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Julian,
welcome to our list ( ... reading your post definitely made my day (Even if I thought it couldn't get better)
Regarding your first Idea ... this is definitely on my mind and was taken into consideration when implementing the S7 driver.
Right now the S7MessageProcessor interface provides an extension point for things like this. Currently there is only one implementation. The DefaultS7MessageProcessor is used to automatically split up large reads and writes into multiple requests depending on the negotiated max PDU size. This would be the place to re-write the S7Message requests to do what you are planning on doing. I was planning on getting a friend of mine on board to address this sort of thing (She's a mathematician, specialized on packing and optimization problems), but if you are interested and want to work on this, I would be glad to help you with this. I think the only thing currently not supported, is to read information which would exceed the PDU size if sent in a single PDU.
Regarding batch reads and writes ... I think such a thing would be a great extension. However we would need to be able to configure this sort of thing. As this could be something that might confuse people, manually turning batch reads/writes on and configuring this in the connection string would be a good way to go. The way PLC4X is built up, it shouldn't be a problem to implement something like this. The Plc4XS7Protocol class could simply queue all read/write items and have a separate worker drain that queue. This could be similar to the queuing implemented in the S7Protocol class, just the trigger would be a different one (Eventually the driver could send some sort of Batch operation heartbeat event). Here too I would be more than happy to help implementing this.
Chris
Am 26.07.18, 13:59 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hey everybody,
first, a great compliment for starting this excellent project and bringing it this far forward.
I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
And let me start with a first idea of a feature, I think we would need.
For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
The main reason for this is to reduce the communication load on the PLC side.
I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
What do you think of this approach?
Best!
Julian
pragmatic industries
Re: Warm welcome and Batching
Posted by Sebastian Rühl <se...@googlemail.com.INVALID>.
Hi Julian,
Welcome to the list and the community. :)
By now Christofer should hopefully answered all you question in his response.
Im working mostly on the Beckhoff ADS and Modbus implementation at the time being. So in case you reach one of these protocols I am happy to help and feedback is always welcome.
Best Regards, Sebastian
> Am 26.07.2018 um 13:59 schrieb Julian Feinauer <j....@pragmaticminds.de>:
>
> Hey everybody,
>
> first, a great compliment for starting this excellent project and bringing it this far forward.
> I am on the dev mailing list since some months and was following the development very interested but was not able to give any input due to other duties.
> We are a small german start-up doing (at least some of) the IoT and Industry 4.0 stuff everybody is talking about. Currently, we have our own home-build implementation of the S7 protocol as we do a lot of communication with Siemens PLCs.
> We also had some discussions and ideas of a more general interface as we start to incorporate other devices and protokolls in our stack. And from my perspective PLC4J is the “ideal” approach and fits nearly perfect with our ideas and thus, we are currently thinking about replacing our own code with PLC4J and in turn, try to bring this project forward in the directions we aim to go.
>
> Thus, from my side a warm welcome to the community and all contributors and I’m really looking forward for the discussions with you!
>
> And let me start with a first idea of a feature, I think we would need.
> For a short background, we are doing a lot of interaction with PLCs and usually read a lot of data (many variables) from the PLC with high frequencies.
> Thus, our approach is not to read single addresses and cast them in one step but to read one connected memory region (datablock) and extract all variables of interest from the block and cast them to the respective types.
> The main reason for this is to reduce the communication load on the PLC side.
> I hope that I’m not duplicating existing ideas or concepts but I didn’t find anything specific to this problem.
>
> To be able to switch to PLC4J my idea was to have a concept of “Batch” Reads or Writes, kind of like the JDBCs batch mode.
> This could have two variants. One would be the possibility to give the driver a list of variables to read and the driver batches them as it makes sense and then performs the read opaque and returns all the result.
> The other option, especially in the async case could be that the reader internally batches subsequent calls until certain thresholds (number of commands, time limit) are met and then performs batched reads of consecutive memory regions.
>
> What do you think of this approach?
>
> Best!
> Julian
> pragmatic industries