You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@plc4x.apache.org by Julian Feinauer <j....@pragmaticminds.de> on 2019/04/29 06:20:26 UTC
[generation] Another approach to generated drivers
Hi all,
just wanted to sum up some talks and discussions we had off-list about the whole topic of driver generation / providing drivers in other languages.
Currently, there are the following two approaches going on:
Driver Generation based on DFDL by Chris:
Chris already shared his branch and is working on the generation of drivers based on the specification of the messages and a state machine.
This should then generate Code based on Freemarker Templates.
Proxy Drivers by me, Chris and also others:
We have a thrift based server / client spec.
A simple Java Server is implemented as Interop Server and we work on providing client in other languages.
This is a separate feature (as Proxy) but in a mode, where the Client itself starts the server in the background, this is an intermediate solution to already provide other language bindings (although at a cost).
All work is done in the PLC4X-111 branch and I hope that we will be able to Merge that soon (Chris spend a lot of effort to include all the new stuff in our build).
And there is a new thing coming currently which is mostly Driven by Matthias and myself regarding the driver code generation.
We went over Chris example code (and the xmls) and our heads nearly exploded as it is so abstract.
And as Matthias does a lot with model based code generation we had a long discussion about using a model based approach (probably with a DSL).
So we currently try to investigate that a bit but also with a focus on research. In fact, we have the potential that we can engage some students from his institute to participate at the work.
In fact, we even started a private Repo where we prepare a Paper to discuss the matter.
As this would be our first “PLC4X Paper” everybody is invited to join us and should simply ping me (with github credentials), to get repo access.
If we make it through we will will of course list everybody who contributed as author.
To make it clear, this work, the DSL based driver generation tries to achieve the same as Chris approach based on DFDL just through another way. And right now I’m unable to say which one is better, could be better and where are the drawbacks and advantages. So we want to investigate that to have a basis for a discussion and decision. In fact, both approaches should be equally powerful, so one could be able to translate one to the other and vice versa, in theory.
I consider it highly important to have a good and easy way to develop and maintain drivers as this is the crucial thing for the future of PLC4X.
So please feel free to comment or discuss, if you feel like : )
Julian
Re: [generation] Another approach to generated drivers
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Julian,
I'm totally fine with that ;-)
Chris
Am 29.04.19, 10:43 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:
Hi Chris,
Hi Niclas,
@Niclas:
Basically Chris answered all those questions as I would. I also looked at Meta Languages like Haxe but there usually where some drawbacks.
But what we discussed was the idea to generate the DSL or the AST from python code (very limited python code), so that our DSL would be equivalent to a subset of Python (which would allow easy and efficient debugging and such).
@Chris:
I wrote this email rather longish to ensure, that you don’t get me wrong. I see both approaches simply as two contenders for the best one, and indeed we should evolve both and then discuss (and finally VOTE) which route to follow, this is the way to do it.
And regarding the "in-between" I agree with you. I can imagine that DFDL is the way to go for the messages and types and that we replace the SCXML with something which comes out of our approach.
And with "head exploding" we did not mention some of your works but simply the thought to develop a SCXML description of a complex protocol, so please don’t take this as any offence.
To the two other statements... I do not want to couple project stuff with students work its just a possibility to explore things more in depth and e.g. develop some tooling which is not strictly necessary but nice to have.
And for the private Repo... It also felt wrong for me to write this. And strictly speaking its not a PLC4X Paper but a Paper which "talks about" PLC4X. The main reason for the private repo are license concerns as the authors usually have to hand over several rights to the Publisher which would not be possible in case of the Apache License. You don’t have to like that but that’s how it works.
Currently, I see it as an research engagement of Matthias and myself but want to invite everybody to participate but (also for the reason above) would not make it a central concern of the project.
I hope you can agree on that?
Julian
Am 29.04.19, 10:05 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi All,
to answer Niclas' question first ... yes a lot of tools allow writing code in for example Java and have that compiled to c-like programs.
I had a look at a lot of these options before starting to work on PLC4X and continued since then.
I was never quite happy with the results. While the performance of the original is ... well ... 100% you usually have great losses on the cross compiled outputs. Also there is the problem that the cross-compiled version usually needs quite a lot more resources as the original and especially as the native implementation.
For example when writing Raw-Socket code in java, we use some extra hoops to do so, in C, C++, C# (maybe even python) we could directly use the network hardware. So depending on the language you use and the platform you build for the drivers could be quite different.
And regarding the API: We would be translating an API of one language to others ... what would we use for C support? Mapping Java to C isn't trivial and the result would be quite evil looking, I bet. The way we were currently planning on doing things, would be to manually implement an API module that perfectly fits for the language we are targeting and to generate the drivers for that language.
Regarding Julian's proposal:
Well if you wouldn't be spawning new initiatives that bind me with preparing the build all the time, I would have a chance to actually work on my proposal ;-) (Haven't worked on that for weeks)
Also didn't I ever have the chance to explain it to anyone yet, so calling it a "mind detonator" is a little unfair, cause I think it's a lot simpler than our current hand-written drivers. I do agree that a specially hand-crafted and well implemented DSL for exactly this use-case will probably be easier, but you would have to implement the DSL first and then the tooling (Keep in mind with Daffodil/DFDL we get a full blown toolkit to write unit-tests and document the protocol). I was trying to use, what's already out there, cause I knew that inventing something that achieves 90% of what you are doing is super-easy, however to get to 100% you need to put in great efforts.
I would be the last to object in general, but I wouldn't want to rely on a student project where we have to wait till someone else finishes something and hope that he does. Especially if the key-person in charge of this initiative seems to have great problems communicating on this list (Me "waiving with a fence pole" here :-P). I would only call it a PLC4X Paper, if it's actually done by involving the community ... and doing that in a private repo isn't quite the way we do things at Apache. So for this to be a real Apache initiative ... do it in public and do discussions on the list.
So if I finally manage to clean up and prepare the build for the multi-language part ... please at least give me the chance to work on my proposal a little more and at least explain what I had planned.
I would suggest you work on the other option and then we let the community decide which path to take. I am really fond of the DFDL part, but not so overwhelmed by the SCXML part of my proposal... perhaps the ideal solution would be something in-between.
Chris
Am 29.04.19, 08:53 schrieb "Niclas Hedhman" <ni...@hedhman.org>:
Outsider looking in; Many language runtimes allows embedding of other
languages' runtimes within. So why not pick a languages that is reasonably
easy to integrate into other languages, and then write the drivers in a
fully fledged programming language, rather than the DFDL abstraction or
creating a new DSL with a whole slew of consequences later on.
Languages that might be suitable; JavaScript, Lua, Forth, microPython or
even C...
Niclas
On Mon, Apr 29, 2019 at 2:20 PM Julian Feinauer <
j.feinauer@pragmaticminds.de> wrote:
> Hi all,
>
> just wanted to sum up some talks and discussions we had off-list about the
> whole topic of driver generation / providing drivers in other languages.
> Currently, there are the following two approaches going on:
>
> Driver Generation based on DFDL by Chris:
> Chris already shared his branch and is working on the generation of
> drivers based on the specification of the messages and a state machine.
> This should then generate Code based on Freemarker Templates.
>
> Proxy Drivers by me, Chris and also others:
> We have a thrift based server / client spec.
> A simple Java Server is implemented as Interop Server and we work on
> providing client in other languages.
> This is a separate feature (as Proxy) but in a mode, where the Client
> itself starts the server in the background, this is an intermediate
> solution to already provide other language bindings (although at a cost).
> All work is done in the PLC4X-111 branch and I hope that we will be able
> to Merge that soon (Chris spend a lot of effort to include all the new
> stuff in our build).
>
> And there is a new thing coming currently which is mostly Driven by
> Matthias and myself regarding the driver code generation.
> We went over Chris example code (and the xmls) and our heads nearly
> exploded as it is so abstract.
> And as Matthias does a lot with model based code generation we had a long
> discussion about using a model based approach (probably with a DSL).
> So we currently try to investigate that a bit but also with a focus on
> research. In fact, we have the potential that we can engage some students
> from his institute to participate at the work.
> In fact, we even started a private Repo where we prepare a Paper to
> discuss the matter.
> As this would be our first “PLC4X Paper” everybody is invited to join us
> and should simply ping me (with github credentials), to get repo access.
> If we make it through we will will of course list everybody who
> contributed as author.
>
> To make it clear, this work, the DSL based driver generation tries to
> achieve the same as Chris approach based on DFDL just through another way.
> And right now I’m unable to say which one is better, could be better and
> where are the drawbacks and advantages. So we want to investigate that to
> have a basis for a discussion and decision. In fact, both approaches should
> be equally powerful, so one could be able to translate one to the other and
> vice versa, in theory.
> I consider it highly important to have a good and easy way to develop and
> maintain drivers as this is the crucial thing for the future of PLC4X.
>
> So please feel free to comment or discuss, if you feel like : )
>
> Julian
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: [generation] Another approach to generated drivers
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hi,
thanks!
Of course the Paper will only speak about the project in regular 3rd person view so I see no issue for policy violations.
Julian
Am 29.04.19, 13:20 schrieb "Justin Mclean" <ju...@classsoftware.com>:
Hi,
> But, I guess we get rid of the discussion if we do the following
> - generate a PLC4X branch where all the development takes place (=> everything is ASF licensed)
> - keep the paper source in the separate repo and, if necessary, reference code snippets from PLC4X Repo (with clear license)
Works for me.
> The point where I disagree is that the paper writing should be in the open for several reasons
> - it is not a paper of the PLC4X Community, it is a paper highlighting thinks strongly related to PLC4X
Which is also fine, the Apache license allows that, the only minor issue is to respect Apache’s trademarks and branding.
Thanks,
Justin
Re: [generation] Another approach to generated drivers
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
> But, I guess we get rid of the discussion if we do the following
> - generate a PLC4X branch where all the development takes place (=> everything is ASF licensed)
> - keep the paper source in the separate repo and, if necessary, reference code snippets from PLC4X Repo (with clear license)
Works for me.
> The point where I disagree is that the paper writing should be in the open for several reasons
> - it is not a paper of the PLC4X Community, it is a paper highlighting thinks strongly related to PLC4X
Which is also fine, the Apache license allows that, the only minor issue is to respect Apache’s trademarks and branding.
Thanks,
Justin
Re: [generation] Another approach to generated drivers
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hi Justin,
I see your concerns and I agree that I still have to learn some things about the "Apache Way" and this "do it all in the open" philosophy, thus this is a good discussion.
But, I guess we get rid of the discussion if we do the following
- generate a PLC4X branch where all the development takes place (=> everything is ASF licensed)
- keep the paper source in the separate repo and, if necessary, reference code snippets from PLC4X Repo (with clear license)
The point where I disagree is that the paper writing should be in the open for several reasons
- it is not a paper of the PLC4X Community, it is a paper highlighting thinks strongly related to PLC4X
- the paper is written by several individuals which may or may not be of the community
- I want to invite everybody with interest to participate (chris already has access) but no one is automatically part of this separate thing due to being PMC or something
- It is absolutely usual to do paper writing in the private and upload the results (artefacts) on ArXiv and submit them to the journal otherwise you always risk someone copying stuff from you and submitting it without jour notice (which may be an issue from the license side but which should be pretty hard and complicated to proof and which would almost surely damage your paper aswell)
On other project lists sometimes someone shares a preprint of a related Paper when its in the preprint stage.
As we are a small and very vital community I wanted to offer everybody to participate in this, to not feel like "stealing" ideas that others expressed on these lists but to give them the credit they deserve.
I hope that we can all agree on something like that?
Julian
Am 29.04.19, 12:34 schrieb "Justin Mclean" <ju...@classsoftware.com>:
Hi,
> some clarification... this repo currently hosts two things... some code snippets (really messy) and the draft version for the paper (like really really early draft).
If any of the code appears in the paper and is not clearly licensed under a compatible license then it’s unlikely PLC4X can use it.
> And I also want to avoid that someone just takes the paper and submits it somewhere with his own name as author…
And the problem with that is? Even under the most permissive licenses they generally need to acknowledge the original copyright owners.
> But I wanted to share also this Paper effort with the community to give everybody the chance to participate in this paper as it is so strongly related to plc4x.
IMO Putting it in a private repo is not sharing it with the community but the choice to do so is yours.
Thanks,
Justin
Re: [generation] Another approach to generated drivers
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
> some clarification... this repo currently hosts two things... some code snippets (really messy) and the draft version for the paper (like really really early draft).
If any of the code appears in the paper and is not clearly licensed under a compatible license then it’s unlikely PLC4X can use it.
> And I also want to avoid that someone just takes the paper and submits it somewhere with his own name as author…
And the problem with that is? Even under the most permissive licenses they generally need to acknowledge the original copyright owners.
> But I wanted to share also this Paper effort with the community to give everybody the chance to participate in this paper as it is so strongly related to plc4x.
IMO Putting it in a private repo is not sharing it with the community but the choice to do so is yours.
Thanks,
Justin
Re: [generation] Another approach to generated drivers
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hi Justin,
some clarification... this repo currently hosts two things... some code snippets (really messy) and the draft version for the paper (like really really early draft).
The paper code is what is (for me) unsure license related but the code will of course go to the PLC4X repo (once it is more than just a mess) and will be Apache Licensed.
And I also want to avoid that someone just takes the paper and submits it somewhere with his own name as author...
But I wanted to share also this Paper effort with the community to give everybody the chance to participate in this paper as it is so strongly related to plc4x.
Julian
Am 29.04.19, 11:16 schrieb "Justin Mclean" <ju...@classsoftware.com>:
Hi,
> And for the private Repo... It also felt wrong for me to write this. And strictly speaking its not a PLC4X Paper but a Paper which "talks about" PLC4X. The main reason for the private repo are license concerns as the authors usually have to hand over several rights to the Publisher which would not be possible in case of the Apache License. You don’t have to like that but that’s how it works.
If that the case it very unlikely it could even be used in an Apache project, so I can’t see how there could be a future discussion or vote on which approach is better if it doesn’t have a comparable license. Unless that it, the publisher agreed to relicenses it.
BTW this is not impossible one of the most draconian contract I ever signed was with a publisher, but I still managed to get them to agree to make all of the code available under an Apache license, otherwise no one would be able to use for anything useful until 70 years after my death.
Thanks,
Justin
Re: [generation] Another approach to generated drivers
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
> And for the private Repo... It also felt wrong for me to write this. And strictly speaking its not a PLC4X Paper but a Paper which "talks about" PLC4X. The main reason for the private repo are license concerns as the authors usually have to hand over several rights to the Publisher which would not be possible in case of the Apache License. You don’t have to like that but that’s how it works.
If that the case it very unlikely it could even be used in an Apache project, so I can’t see how there could be a future discussion or vote on which approach is better if it doesn’t have a comparable license. Unless that it, the publisher agreed to relicenses it.
BTW this is not impossible one of the most draconian contract I ever signed was with a publisher, but I still managed to get them to agree to make all of the code available under an Apache license, otherwise no one would be able to use for anything useful until 70 years after my death.
Thanks,
Justin
Re: [generation] Another approach to generated drivers
Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hi Chris,
Hi Niclas,
@Niclas:
Basically Chris answered all those questions as I would. I also looked at Meta Languages like Haxe but there usually where some drawbacks.
But what we discussed was the idea to generate the DSL or the AST from python code (very limited python code), so that our DSL would be equivalent to a subset of Python (which would allow easy and efficient debugging and such).
@Chris:
I wrote this email rather longish to ensure, that you don’t get me wrong. I see both approaches simply as two contenders for the best one, and indeed we should evolve both and then discuss (and finally VOTE) which route to follow, this is the way to do it.
And regarding the "in-between" I agree with you. I can imagine that DFDL is the way to go for the messages and types and that we replace the SCXML with something which comes out of our approach.
And with "head exploding" we did not mention some of your works but simply the thought to develop a SCXML description of a complex protocol, so please don’t take this as any offence.
To the two other statements... I do not want to couple project stuff with students work its just a possibility to explore things more in depth and e.g. develop some tooling which is not strictly necessary but nice to have.
And for the private Repo... It also felt wrong for me to write this. And strictly speaking its not a PLC4X Paper but a Paper which "talks about" PLC4X. The main reason for the private repo are license concerns as the authors usually have to hand over several rights to the Publisher which would not be possible in case of the Apache License. You don’t have to like that but that’s how it works.
Currently, I see it as an research engagement of Matthias and myself but want to invite everybody to participate but (also for the reason above) would not make it a central concern of the project.
I hope you can agree on that?
Julian
Am 29.04.19, 10:05 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi All,
to answer Niclas' question first ... yes a lot of tools allow writing code in for example Java and have that compiled to c-like programs.
I had a look at a lot of these options before starting to work on PLC4X and continued since then.
I was never quite happy with the results. While the performance of the original is ... well ... 100% you usually have great losses on the cross compiled outputs. Also there is the problem that the cross-compiled version usually needs quite a lot more resources as the original and especially as the native implementation.
For example when writing Raw-Socket code in java, we use some extra hoops to do so, in C, C++, C# (maybe even python) we could directly use the network hardware. So depending on the language you use and the platform you build for the drivers could be quite different.
And regarding the API: We would be translating an API of one language to others ... what would we use for C support? Mapping Java to C isn't trivial and the result would be quite evil looking, I bet. The way we were currently planning on doing things, would be to manually implement an API module that perfectly fits for the language we are targeting and to generate the drivers for that language.
Regarding Julian's proposal:
Well if you wouldn't be spawning new initiatives that bind me with preparing the build all the time, I would have a chance to actually work on my proposal ;-) (Haven't worked on that for weeks)
Also didn't I ever have the chance to explain it to anyone yet, so calling it a "mind detonator" is a little unfair, cause I think it's a lot simpler than our current hand-written drivers. I do agree that a specially hand-crafted and well implemented DSL for exactly this use-case will probably be easier, but you would have to implement the DSL first and then the tooling (Keep in mind with Daffodil/DFDL we get a full blown toolkit to write unit-tests and document the protocol). I was trying to use, what's already out there, cause I knew that inventing something that achieves 90% of what you are doing is super-easy, however to get to 100% you need to put in great efforts.
I would be the last to object in general, but I wouldn't want to rely on a student project where we have to wait till someone else finishes something and hope that he does. Especially if the key-person in charge of this initiative seems to have great problems communicating on this list (Me "waiving with a fence pole" here :-P). I would only call it a PLC4X Paper, if it's actually done by involving the community ... and doing that in a private repo isn't quite the way we do things at Apache. So for this to be a real Apache initiative ... do it in public and do discussions on the list.
So if I finally manage to clean up and prepare the build for the multi-language part ... please at least give me the chance to work on my proposal a little more and at least explain what I had planned.
I would suggest you work on the other option and then we let the community decide which path to take. I am really fond of the DFDL part, but not so overwhelmed by the SCXML part of my proposal... perhaps the ideal solution would be something in-between.
Chris
Am 29.04.19, 08:53 schrieb "Niclas Hedhman" <ni...@hedhman.org>:
Outsider looking in; Many language runtimes allows embedding of other
languages' runtimes within. So why not pick a languages that is reasonably
easy to integrate into other languages, and then write the drivers in a
fully fledged programming language, rather than the DFDL abstraction or
creating a new DSL with a whole slew of consequences later on.
Languages that might be suitable; JavaScript, Lua, Forth, microPython or
even C...
Niclas
On Mon, Apr 29, 2019 at 2:20 PM Julian Feinauer <
j.feinauer@pragmaticminds.de> wrote:
> Hi all,
>
> just wanted to sum up some talks and discussions we had off-list about the
> whole topic of driver generation / providing drivers in other languages.
> Currently, there are the following two approaches going on:
>
> Driver Generation based on DFDL by Chris:
> Chris already shared his branch and is working on the generation of
> drivers based on the specification of the messages and a state machine.
> This should then generate Code based on Freemarker Templates.
>
> Proxy Drivers by me, Chris and also others:
> We have a thrift based server / client spec.
> A simple Java Server is implemented as Interop Server and we work on
> providing client in other languages.
> This is a separate feature (as Proxy) but in a mode, where the Client
> itself starts the server in the background, this is an intermediate
> solution to already provide other language bindings (although at a cost).
> All work is done in the PLC4X-111 branch and I hope that we will be able
> to Merge that soon (Chris spend a lot of effort to include all the new
> stuff in our build).
>
> And there is a new thing coming currently which is mostly Driven by
> Matthias and myself regarding the driver code generation.
> We went over Chris example code (and the xmls) and our heads nearly
> exploded as it is so abstract.
> And as Matthias does a lot with model based code generation we had a long
> discussion about using a model based approach (probably with a DSL).
> So we currently try to investigate that a bit but also with a focus on
> research. In fact, we have the potential that we can engage some students
> from his institute to participate at the work.
> In fact, we even started a private Repo where we prepare a Paper to
> discuss the matter.
> As this would be our first “PLC4X Paper” everybody is invited to join us
> and should simply ping me (with github credentials), to get repo access.
> If we make it through we will will of course list everybody who
> contributed as author.
>
> To make it clear, this work, the DSL based driver generation tries to
> achieve the same as Chris approach based on DFDL just through another way.
> And right now I’m unable to say which one is better, could be better and
> where are the drawbacks and advantages. So we want to investigate that to
> have a basis for a discussion and decision. In fact, both approaches should
> be equally powerful, so one could be able to translate one to the other and
> vice versa, in theory.
> I consider it highly important to have a good and easy way to develop and
> maintain drivers as this is the crucial thing for the future of PLC4X.
>
> So please feel free to comment or discuss, if you feel like : )
>
> Julian
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
RE: [generation] Another approach to generated drivers
Posted by "Strljic, Matthias Milan" <ma...@isw.uni-stuttgart.de>.
Hi all,
i see there some chris waving 😃 I always need some provocation to act sry for that.
@Chris I agree with you that such a sub-project like the attempt of a template based code generation(TbCG) should be also done in the Apache way and it will be. But Julian and me just wanted first to work out a rapid prototype on which we can then discuss some pro and cons of the attempt over a DSL. Julian is there a bit better in communicating the actual state then me and we discussed there some possibilities where we can push forward and include the the research areas of the ISW in a efficient way. The ideas with the students was more to have roadmap of the code generation idea and plan there some tooling around the TbCG which could be build in parallel around the core pipeline, when it is available(The full DFDL or the DSL).
In my opinion there have both approaches pros and cons. The DFDL approach supports a well implemented tool chain in a declarative manor and the DSL idea targets more on a programming like evolution of a DSL, there was the idea of a more practical attempt of restricting a sub-set of a programming language like python so that it fulfills all of our requirements, regarding supported communication patterns, messages structures and behavior model.
@Niclas I would try to avoid language bindings or the integration of interpreter engines. There you will always have a strict coupling to the implementation of the source language and each change will propagate through every platform. So it would promise fast language coverage but tight coupling to the core implementation.
Greetings the most silent
Matthias Strljic, M.Sc.
Universität Stuttgart
Institut für Steuerungstechnik der Werkzeugmaschinen und Fertigungseinrichtungen (ISW)
Seidenstraße 36
70174 Stuttgart
GERMANY
Tel: +49 711 685-84530
Fax: +49 711 685-74530
E-Mail: matthias.strljic@isw.uni-stuttgart.de
Web: http://www.isw.uni-stuttgart.de
-----Original Message-----
From: Christofer Dutz <ch...@c-ware.de>
Sent: Monday, April 29, 2019 10:06 AM
To: dev@plc4x.apache.org
Subject: Re: [generation] Another approach to generated drivers
Hi All,
to answer Niclas' question first ... yes a lot of tools allow writing code in for example Java and have that compiled to c-like programs.
I had a look at a lot of these options before starting to work on PLC4X and continued since then.
I was never quite happy with the results. While the performance of the original is ... well ... 100% you usually have great losses on the cross compiled outputs. Also there is the problem that the cross-compiled version usually needs quite a lot more resources as the original and especially as the native implementation.
For example when writing Raw-Socket code in java, we use some extra hoops to do so, in C, C++, C# (maybe even python) we could directly use the network hardware. So depending on the language you use and the platform you build for the drivers could be quite different.
And regarding the API: We would be translating an API of one language to others ... what would we use for C support? Mapping Java to C isn't trivial and the result would be quite evil looking, I bet. The way we were currently planning on doing things, would be to manually implement an API module that perfectly fits for the language we are targeting and to generate the drivers for that language.
Regarding Julian's proposal:
Well if you wouldn't be spawning new initiatives that bind me with preparing the build all the time, I would have a chance to actually work on my proposal ;-) (Haven't worked on that for weeks)
Also didn't I ever have the chance to explain it to anyone yet, so calling it a "mind detonator" is a little unfair, cause I think it's a lot simpler than our current hand-written drivers. I do agree that a specially hand-crafted and well implemented DSL for exactly this use-case will probably be easier, but you would have to implement the DSL first and then the tooling (Keep in mind with Daffodil/DFDL we get a full blown toolkit to write unit-tests and document the protocol). I was trying to use, what's already out there, cause I knew that inventing something that achieves 90% of what you are doing is super-easy, however to get to 100% you need to put in great efforts.
I would be the last to object in general, but I wouldn't want to rely on a student project where we have to wait till someone else finishes something and hope that he does. Especially if the key-person in charge of this initiative seems to have great problems communicating on this list (Me "waiving with a fence pole" here :-P). I would only call it a PLC4X Paper, if it's actually done by involving the community ... and doing that in a private repo isn't quite the way we do things at Apache. So for this to be a real Apache initiative ... do it in public and do discussions on the list.
So if I finally manage to clean up and prepare the build for the multi-language part ... please at least give me the chance to work on my proposal a little more and at least explain what I had planned.
I would suggest you work on the other option and then we let the community decide which path to take. I am really fond of the DFDL part, but not so overwhelmed by the SCXML part of my proposal... perhaps the ideal solution would be something in-between.
Chris
Am 29.04.19, 08:53 schrieb "Niclas Hedhman" <ni...@hedhman.org>:
Outsider looking in; Many language runtimes allows embedding of other
languages' runtimes within. So why not pick a languages that is reasonably
easy to integrate into other languages, and then write the drivers in a
fully fledged programming language, rather than the DFDL abstraction or
creating a new DSL with a whole slew of consequences later on.
Languages that might be suitable; JavaScript, Lua, Forth, microPython or
even C...
Niclas
On Mon, Apr 29, 2019 at 2:20 PM Julian Feinauer <
j.feinauer@pragmaticminds.de> wrote:
> Hi all,
>
> just wanted to sum up some talks and discussions we had off-list about the
> whole topic of driver generation / providing drivers in other languages.
> Currently, there are the following two approaches going on:
>
> Driver Generation based on DFDL by Chris:
> Chris already shared his branch and is working on the generation of
> drivers based on the specification of the messages and a state machine.
> This should then generate Code based on Freemarker Templates.
>
> Proxy Drivers by me, Chris and also others:
> We have a thrift based server / client spec.
> A simple Java Server is implemented as Interop Server and we work on
> providing client in other languages.
> This is a separate feature (as Proxy) but in a mode, where the Client
> itself starts the server in the background, this is an intermediate
> solution to already provide other language bindings (although at a cost).
> All work is done in the PLC4X-111 branch and I hope that we will be able
> to Merge that soon (Chris spend a lot of effort to include all the new
> stuff in our build).
>
> And there is a new thing coming currently which is mostly Driven by
> Matthias and myself regarding the driver code generation.
> We went over Chris example code (and the xmls) and our heads nearly
> exploded as it is so abstract.
> And as Matthias does a lot with model based code generation we had a long
> discussion about using a model based approach (probably with a DSL).
> So we currently try to investigate that a bit but also with a focus on
> research. In fact, we have the potential that we can engage some students
> from his institute to participate at the work.
> In fact, we even started a private Repo where we prepare a Paper to
> discuss the matter.
> As this would be our first “PLC4X Paper” everybody is invited to join us
> and should simply ping me (with github credentials), to get repo access.
> If we make it through we will will of course list everybody who
> contributed as author.
>
> To make it clear, this work, the DSL based driver generation tries to
> achieve the same as Chris approach based on DFDL just through another way.
> And right now I’m unable to say which one is better, could be better and
> where are the drawbacks and advantages. So we want to investigate that to
> have a basis for a discussion and decision. In fact, both approaches should
> be equally powerful, so one could be able to translate one to the other and
> vice versa, in theory.
> I consider it highly important to have a good and easy way to develop and
> maintain drivers as this is the crucial thing for the future of PLC4X.
>
> So please feel free to comment or discuss, if you feel like : )
>
> Julian
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: [generation] Another approach to generated drivers
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi All,
to answer Niclas' question first ... yes a lot of tools allow writing code in for example Java and have that compiled to c-like programs.
I had a look at a lot of these options before starting to work on PLC4X and continued since then.
I was never quite happy with the results. While the performance of the original is ... well ... 100% you usually have great losses on the cross compiled outputs. Also there is the problem that the cross-compiled version usually needs quite a lot more resources as the original and especially as the native implementation.
For example when writing Raw-Socket code in java, we use some extra hoops to do so, in C, C++, C# (maybe even python) we could directly use the network hardware. So depending on the language you use and the platform you build for the drivers could be quite different.
And regarding the API: We would be translating an API of one language to others ... what would we use for C support? Mapping Java to C isn't trivial and the result would be quite evil looking, I bet. The way we were currently planning on doing things, would be to manually implement an API module that perfectly fits for the language we are targeting and to generate the drivers for that language.
Regarding Julian's proposal:
Well if you wouldn't be spawning new initiatives that bind me with preparing the build all the time, I would have a chance to actually work on my proposal ;-) (Haven't worked on that for weeks)
Also didn't I ever have the chance to explain it to anyone yet, so calling it a "mind detonator" is a little unfair, cause I think it's a lot simpler than our current hand-written drivers. I do agree that a specially hand-crafted and well implemented DSL for exactly this use-case will probably be easier, but you would have to implement the DSL first and then the tooling (Keep in mind with Daffodil/DFDL we get a full blown toolkit to write unit-tests and document the protocol). I was trying to use, what's already out there, cause I knew that inventing something that achieves 90% of what you are doing is super-easy, however to get to 100% you need to put in great efforts.
I would be the last to object in general, but I wouldn't want to rely on a student project where we have to wait till someone else finishes something and hope that he does. Especially if the key-person in charge of this initiative seems to have great problems communicating on this list (Me "waiving with a fence pole" here :-P). I would only call it a PLC4X Paper, if it's actually done by involving the community ... and doing that in a private repo isn't quite the way we do things at Apache. So for this to be a real Apache initiative ... do it in public and do discussions on the list.
So if I finally manage to clean up and prepare the build for the multi-language part ... please at least give me the chance to work on my proposal a little more and at least explain what I had planned.
I would suggest you work on the other option and then we let the community decide which path to take. I am really fond of the DFDL part, but not so overwhelmed by the SCXML part of my proposal... perhaps the ideal solution would be something in-between.
Chris
Am 29.04.19, 08:53 schrieb "Niclas Hedhman" <ni...@hedhman.org>:
Outsider looking in; Many language runtimes allows embedding of other
languages' runtimes within. So why not pick a languages that is reasonably
easy to integrate into other languages, and then write the drivers in a
fully fledged programming language, rather than the DFDL abstraction or
creating a new DSL with a whole slew of consequences later on.
Languages that might be suitable; JavaScript, Lua, Forth, microPython or
even C...
Niclas
On Mon, Apr 29, 2019 at 2:20 PM Julian Feinauer <
j.feinauer@pragmaticminds.de> wrote:
> Hi all,
>
> just wanted to sum up some talks and discussions we had off-list about the
> whole topic of driver generation / providing drivers in other languages.
> Currently, there are the following two approaches going on:
>
> Driver Generation based on DFDL by Chris:
> Chris already shared his branch and is working on the generation of
> drivers based on the specification of the messages and a state machine.
> This should then generate Code based on Freemarker Templates.
>
> Proxy Drivers by me, Chris and also others:
> We have a thrift based server / client spec.
> A simple Java Server is implemented as Interop Server and we work on
> providing client in other languages.
> This is a separate feature (as Proxy) but in a mode, where the Client
> itself starts the server in the background, this is an intermediate
> solution to already provide other language bindings (although at a cost).
> All work is done in the PLC4X-111 branch and I hope that we will be able
> to Merge that soon (Chris spend a lot of effort to include all the new
> stuff in our build).
>
> And there is a new thing coming currently which is mostly Driven by
> Matthias and myself regarding the driver code generation.
> We went over Chris example code (and the xmls) and our heads nearly
> exploded as it is so abstract.
> And as Matthias does a lot with model based code generation we had a long
> discussion about using a model based approach (probably with a DSL).
> So we currently try to investigate that a bit but also with a focus on
> research. In fact, we have the potential that we can engage some students
> from his institute to participate at the work.
> In fact, we even started a private Repo where we prepare a Paper to
> discuss the matter.
> As this would be our first “PLC4X Paper” everybody is invited to join us
> and should simply ping me (with github credentials), to get repo access.
> If we make it through we will will of course list everybody who
> contributed as author.
>
> To make it clear, this work, the DSL based driver generation tries to
> achieve the same as Chris approach based on DFDL just through another way.
> And right now I’m unable to say which one is better, could be better and
> where are the drawbacks and advantages. So we want to investigate that to
> have a basis for a discussion and decision. In fact, both approaches should
> be equally powerful, so one could be able to translate one to the other and
> vice versa, in theory.
> I consider it highly important to have a good and easy way to develop and
> maintain drivers as this is the crucial thing for the future of PLC4X.
>
> So please feel free to comment or discuss, if you feel like : )
>
> Julian
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: [generation] Another approach to generated drivers
Posted by Niclas Hedhman <ni...@hedhman.org>.
Outsider looking in; Many language runtimes allows embedding of other
languages' runtimes within. So why not pick a languages that is reasonably
easy to integrate into other languages, and then write the drivers in a
fully fledged programming language, rather than the DFDL abstraction or
creating a new DSL with a whole slew of consequences later on.
Languages that might be suitable; JavaScript, Lua, Forth, microPython or
even C...
Niclas
On Mon, Apr 29, 2019 at 2:20 PM Julian Feinauer <
j.feinauer@pragmaticminds.de> wrote:
> Hi all,
>
> just wanted to sum up some talks and discussions we had off-list about the
> whole topic of driver generation / providing drivers in other languages.
> Currently, there are the following two approaches going on:
>
> Driver Generation based on DFDL by Chris:
> Chris already shared his branch and is working on the generation of
> drivers based on the specification of the messages and a state machine.
> This should then generate Code based on Freemarker Templates.
>
> Proxy Drivers by me, Chris and also others:
> We have a thrift based server / client spec.
> A simple Java Server is implemented as Interop Server and we work on
> providing client in other languages.
> This is a separate feature (as Proxy) but in a mode, where the Client
> itself starts the server in the background, this is an intermediate
> solution to already provide other language bindings (although at a cost).
> All work is done in the PLC4X-111 branch and I hope that we will be able
> to Merge that soon (Chris spend a lot of effort to include all the new
> stuff in our build).
>
> And there is a new thing coming currently which is mostly Driven by
> Matthias and myself regarding the driver code generation.
> We went over Chris example code (and the xmls) and our heads nearly
> exploded as it is so abstract.
> And as Matthias does a lot with model based code generation we had a long
> discussion about using a model based approach (probably with a DSL).
> So we currently try to investigate that a bit but also with a focus on
> research. In fact, we have the potential that we can engage some students
> from his institute to participate at the work.
> In fact, we even started a private Repo where we prepare a Paper to
> discuss the matter.
> As this would be our first “PLC4X Paper” everybody is invited to join us
> and should simply ping me (with github credentials), to get repo access.
> If we make it through we will will of course list everybody who
> contributed as author.
>
> To make it clear, this work, the DSL based driver generation tries to
> achieve the same as Chris approach based on DFDL just through another way.
> And right now I’m unable to say which one is better, could be better and
> where are the drawbacks and advantages. So we want to investigate that to
> have a basis for a discussion and decision. In fact, both approaches should
> be equally powerful, so one could be able to translate one to the other and
> vice versa, in theory.
> I consider it highly important to have a good and easy way to develop and
> maintain drivers as this is the crucial thing for the future of PLC4X.
>
> So please feel free to comment or discuss, if you feel like : )
>
> Julian
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java