You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by "Interrante, John A (GE Research, US)" <Jo...@ge.com> on 2021/03/10 21:24:51 UTC
Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
Daffodil's Confluence describes Runtime2's design here:
https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
* Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
* Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are welcome:
1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
3. Which of the Runtime2 ToDos need to be done before the merge as well?
Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
AW: [OFFTOPIC] Re: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by Christofer Dutz <ch...@c-ware.de>.
Aaaaahh ...
Think I'll renew my Office 365 subscripton and pass on that one ;-)
Duck and run ...
Chris
-----Ursprüngliche Nachricht-----
Von: Dave Fisher <wa...@apache.org>
Gesendet: Donnerstag, 11. März 2021 21:55
An: dev@daffodil.apache.org
Betreff: [OFFTOPIC] Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Hi Chris,
Try building OpenOffice … macOS, Windows, Linux, and downstream FreeBSD, OS/2 …. Older c++, two different build tools, patched dependencies.
;-)
Regards,
Dave
> On Mar 11, 2021, at 12:51 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Hi John,
>
> the script is here:
> https://github.com/apache/plc4x/blob/develop/src/main/script/prerequisiteCheck.groovy
>
> And it's called from here:
> https://github.com/apache/plc4x/blob/develop/pom.xml#L644
>
> Hope it helps ... I just know as soon as you leave the Java eco-system software-development really gets tricky regarding checking all the stuff that could be or be configured wrong, so I tend to automate as much as possible.
> It increases the user acceptance and it reduces the questions from annoyed users ;.)
>
> Chris
>
>
> -----Ursprüngliche Nachricht-----
> Von: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Gesendet: Donnerstag, 11. März 2021 20:51
> An: dev@daffodil.apache.org
> Betreff: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Chris,
>
> Yes, I think it would be nice if sbt called a Groovy or Scala script to check that the C setup is correct. Will you please tell me where to find PLC4X's script and where it gets called from?
>
> Thanks,
> John
>
> -----Original Message-----
> From: Christofer Dutz <ch...@c-ware.de>
> Sent: Thursday, March 11, 2021 11:54 AM
> To: dev@daffodil.apache.org
> Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Hi all,
>
> I would strongly suggest having some tooling to check the setup is correct.
> In PLC4X we have a groovy script that is executed by the build. Depending on which parts you are building, it checks if required tools are installed in sufficient versions. Not sure if sbt allows similar checks, but perhaps our setup in PLC4X could help?
>
> Otherwise with all these C, C++ tools it's really hard for newbies to find what's going wrong.
>
> Chris
>
> -----Ursprüngliche Nachricht-----
> Von: Beckerle, Mike <mb...@owlcyberdefense.com>
> Gesendet: Donnerstag, 11. März 2021 17:38
> An: dev@daffodil.apache.org
> Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Setup of the C-toolchain as well as scala shoulnd't be a problem so long as these are all no-cost tools.
> ________________________________
> From: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Sent: Thursday, March 11, 2021 11:34 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Is that all? :-). I would add some criteria testing runtime2's conformance to the DFDL 1.0 specification as well. Here goes...
>
> 1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
>
> Yup. We have 7 schemas in daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" examples of messages with sufficient complexity to indicate that other real messages should be possible.
>
> 2) contains built-in tests for one or several such examples, showing each supported aspect working.
>
> Yup. We have TDML tests and binary data/XML infoset files for each of these 7 schemas.
>
> 3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
>
> Yup. We have Scala test classes for each of these 7 schemas' TDML tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
>
> 4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
>
> Needs work. The top-level README lists daffodil-runtime2's build requirements under Build Requirements and has a single sentence under Getting Started telling developers that they will need a C toolchain in order to build daffodil-runtime2. However, I didn't make it clear in the README that developers must setup a C toolchain as well as a Java/Scala toolchain and I need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step instructions showing developers how to setup a C toolchain on Linux and Windows. I hope developers won't mind if we make it mandatory to setup both Java/Scala and C toolchains before building Daffodil. I don't like the idea of some developers building Daffodil without checking that the C files compile successfully and the runtime2 tests pass successfully. I think it's sufficient to guarantee that end-users of Daffodil never have to install a C toolchain to use Daffodil. When end-users install Daffodil, they will get a pure Scala Daffodil just like before. The only user-visible change is that this Daffodil will know how to generate, compile, and run C code on the end-user's system if the user asks Daffodil to do that.
>
> I also would add another criteria:
>
> 5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests modified to run on runtime2 as well as runtime1 during 'sbt test'.
>
> We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 1.0 specification. We should be able to find a subset of these tests that should pass on runtime2 as well and verify that they do keep passing when run on both runtime1 and runtime2. This step would go a long way to ensure that people working on other aspects of Daffodil do not break the C backend inadvertently without knowing it. Note that this makes it even more mandatory that Daffodil developers setup both Java/Scala and C toolchains. What do you all think?
>
>
> -----Original Message-----
> From: Beckerle, Mike <mb...@owlcyberdefense.com>
> Sent: Thursday, March 11, 2021 9:27 AM
> To: dev@daffodil.apache.org
> Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Acceptance criteria to me:
>
> 1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
>
> 2) contains built-in tests for one or several such examples, showing each supported aspect working.
>
> 3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
>
> 4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
>
> I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
> ________________________________
> From: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Sent: Wednesday, March 10, 2021 4:24 PM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
>
> https://daffodil.apache.org/dev/design-notes/runtime2-todos/
>
> I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
>
> Daffodil's Confluence describes Runtime2's design here:
>
> https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
>
> In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
>
> * Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
> * Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
> I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
> Here is an initial set of discussion points - more questions and criteria are welcome:
>
>
> 1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
> 2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
> 3. Which of the Runtime2 ToDos need to be done before the merge as well?
>
> Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
[OFFTOPIC] Re: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by Dave Fisher <wa...@apache.org>.
Hi Chris,
Try building OpenOffice … macOS, Windows, Linux, and downstream FreeBSD, OS/2 …. Older c++, two different build tools, patched dependencies.
;-)
Regards,
Dave
> On Mar 11, 2021, at 12:51 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Hi John,
>
> the script is here:
> https://github.com/apache/plc4x/blob/develop/src/main/script/prerequisiteCheck.groovy
>
> And it's called from here:
> https://github.com/apache/plc4x/blob/develop/pom.xml#L644
>
> Hope it helps ... I just know as soon as you leave the Java eco-system software-development really gets tricky regarding checking all the stuff that could be or be configured wrong, so I tend to automate as much as possible.
> It increases the user acceptance and it reduces the questions from annoyed users ;.)
>
> Chris
>
>
> -----Ursprüngliche Nachricht-----
> Von: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Gesendet: Donnerstag, 11. März 2021 20:51
> An: dev@daffodil.apache.org
> Betreff: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Chris,
>
> Yes, I think it would be nice if sbt called a Groovy or Scala script to check that the C setup is correct. Will you please tell me where to find PLC4X's script and where it gets called from?
>
> Thanks,
> John
>
> -----Original Message-----
> From: Christofer Dutz <ch...@c-ware.de>
> Sent: Thursday, March 11, 2021 11:54 AM
> To: dev@daffodil.apache.org
> Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Hi all,
>
> I would strongly suggest having some tooling to check the setup is correct.
> In PLC4X we have a groovy script that is executed by the build. Depending on which parts you are building, it checks if required tools are installed in sufficient versions. Not sure if sbt allows similar checks, but perhaps our setup in PLC4X could help?
>
> Otherwise with all these C, C++ tools it's really hard for newbies to find what's going wrong.
>
> Chris
>
> -----Ursprüngliche Nachricht-----
> Von: Beckerle, Mike <mb...@owlcyberdefense.com>
> Gesendet: Donnerstag, 11. März 2021 17:38
> An: dev@daffodil.apache.org
> Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Setup of the C-toolchain as well as scala shoulnd't be a problem so long as these are all no-cost tools.
> ________________________________
> From: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Sent: Thursday, March 11, 2021 11:34 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Is that all? :-). I would add some criteria testing runtime2's conformance to the DFDL 1.0 specification as well. Here goes...
>
> 1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
>
> Yup. We have 7 schemas in daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" examples of messages with sufficient complexity to indicate that other real messages should be possible.
>
> 2) contains built-in tests for one or several such examples, showing each supported aspect working.
>
> Yup. We have TDML tests and binary data/XML infoset files for each of these 7 schemas.
>
> 3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
>
> Yup. We have Scala test classes for each of these 7 schemas' TDML tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
>
> 4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
>
> Needs work. The top-level README lists daffodil-runtime2's build requirements under Build Requirements and has a single sentence under Getting Started telling developers that they will need a C toolchain in order to build daffodil-runtime2. However, I didn't make it clear in the README that developers must setup a C toolchain as well as a Java/Scala toolchain and I need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step instructions showing developers how to setup a C toolchain on Linux and Windows. I hope developers won't mind if we make it mandatory to setup both Java/Scala and C toolchains before building Daffodil. I don't like the idea of some developers building Daffodil without checking that the C files compile successfully and the runtime2 tests pass successfully. I think it's sufficient to guarantee that end-users of Daffodil never have to install a C toolchain to use Daffodil. When end-users install Daffodil, they will get a pure Scala Daffodil just like before. The only user-visible change is that this Daffodil will know how to generate, compile, and run C code on the end-user's system if the user asks Daffodil to do that.
>
> I also would add another criteria:
>
> 5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests modified to run on runtime2 as well as runtime1 during 'sbt test'.
>
> We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 1.0 specification. We should be able to find a subset of these tests that should pass on runtime2 as well and verify that they do keep passing when run on both runtime1 and runtime2. This step would go a long way to ensure that people working on other aspects of Daffodil do not break the C backend inadvertently without knowing it. Note that this makes it even more mandatory that Daffodil developers setup both Java/Scala and C toolchains. What do you all think?
>
>
> -----Original Message-----
> From: Beckerle, Mike <mb...@owlcyberdefense.com>
> Sent: Thursday, March 11, 2021 9:27 AM
> To: dev@daffodil.apache.org
> Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Acceptance criteria to me:
>
> 1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
>
> 2) contains built-in tests for one or several such examples, showing each supported aspect working.
>
> 3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
>
> 4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
>
> I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
> ________________________________
> From: Interrante, John A (GE Research, US) <Jo...@ge.com>
> Sent: Wednesday, March 10, 2021 4:24 PM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
>
> Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
>
> https://daffodil.apache.org/dev/design-notes/runtime2-todos/
>
> I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
>
> Daffodil's Confluence describes Runtime2's design here:
>
> https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
>
> In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
>
> * Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
> * Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
> I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
> Here is an initial set of discussion points - more questions and criteria are welcome:
>
>
> 1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
> 2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
> 3. Which of the Runtime2 ToDos need to be done before the merge as well?
>
> Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
AW: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi John,
the script is here:
https://github.com/apache/plc4x/blob/develop/src/main/script/prerequisiteCheck.groovy
And it's called from here:
https://github.com/apache/plc4x/blob/develop/pom.xml#L644
Hope it helps ... I just know as soon as you leave the Java eco-system software-development really gets tricky regarding checking all the stuff that could be or be configured wrong, so I tend to automate as much as possible.
It increases the user acceptance and it reduces the questions from annoyed users ;.)
Chris
-----Ursprüngliche Nachricht-----
Von: Interrante, John A (GE Research, US) <Jo...@ge.com>
Gesendet: Donnerstag, 11. März 2021 20:51
An: dev@daffodil.apache.org
Betreff: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Chris,
Yes, I think it would be nice if sbt called a Groovy or Scala script to check that the C setup is correct. Will you please tell me where to find PLC4X's script and where it gets called from?
Thanks,
John
-----Original Message-----
From: Christofer Dutz <ch...@c-ware.de>
Sent: Thursday, March 11, 2021 11:54 AM
To: dev@daffodil.apache.org
Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Hi all,
I would strongly suggest having some tooling to check the setup is correct.
In PLC4X we have a groovy script that is executed by the build. Depending on which parts you are building, it checks if required tools are installed in sufficient versions. Not sure if sbt allows similar checks, but perhaps our setup in PLC4X could help?
Otherwise with all these C, C++ tools it's really hard for newbies to find what's going wrong.
Chris
-----Ursprüngliche Nachricht-----
Von: Beckerle, Mike <mb...@owlcyberdefense.com>
Gesendet: Donnerstag, 11. März 2021 17:38
An: dev@daffodil.apache.org
Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Setup of the C-toolchain as well as scala shoulnd't be a problem so long as these are all no-cost tools.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Is that all? :-). I would add some criteria testing runtime2's conformance to the DFDL 1.0 specification as well. Here goes...
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
Yup. We have 7 schemas in daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" examples of messages with sufficient complexity to indicate that other real messages should be possible.
2) contains built-in tests for one or several such examples, showing each supported aspect working.
Yup. We have TDML tests and binary data/XML infoset files for each of these 7 schemas.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
Yup. We have Scala test classes for each of these 7 schemas' TDML tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
Needs work. The top-level README lists daffodil-runtime2's build requirements under Build Requirements and has a single sentence under Getting Started telling developers that they will need a C toolchain in order to build daffodil-runtime2. However, I didn't make it clear in the README that developers must setup a C toolchain as well as a Java/Scala toolchain and I need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step instructions showing developers how to setup a C toolchain on Linux and Windows. I hope developers won't mind if we make it mandatory to setup both Java/Scala and C toolchains before building Daffodil. I don't like the idea of some developers building Daffodil without checking that the C files compile successfully and the runtime2 tests pass successfully. I think it's sufficient to guarantee that end-users of Daffodil never have to install a C toolchain to use Daffodil. When end-users install Daffodil, they will get a pure Scala Daffodil just like before. The only user-visible change is that this Daffodil will know how to generate, compile, and run C code on the end-user's system if the user asks Daffodil to do that.
I also would add another criteria:
5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests modified to run on runtime2 as well as runtime1 during 'sbt test'.
We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 1.0 specification. We should be able to find a subset of these tests that should pass on runtime2 as well and verify that they do keep passing when run on both runtime1 and runtime2. This step would go a long way to ensure that people working on other aspects of Daffodil do not break the C backend inadvertently without knowing it. Note that this makes it even more mandatory that Daffodil developers setup both Java/Scala and C toolchains. What do you all think?
-----Original Message-----
From: Beckerle, Mike <mb...@owlcyberdefense.com>
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Acceptance criteria to me:
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
2) contains built-in tests for one or several such examples, showing each supported aspect working.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
Daffodil's Confluence describes Runtime2's design here:
https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
* Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
* Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are welcome:
1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
3. Which of the Runtime2 ToDos need to be done before the merge as well?
Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
RE: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by "Interrante, John A (GE Research, US)" <Jo...@ge.com>.
Chris,
Yes, I think it would be nice if sbt called a Groovy or Scala script to check that the C setup is correct. Will you please tell me where to find PLC4X's script and where it gets called from?
Thanks,
John
-----Original Message-----
From: Christofer Dutz <ch...@c-ware.de>
Sent: Thursday, March 11, 2021 11:54 AM
To: dev@daffodil.apache.org
Subject: EXT: AW: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Hi all,
I would strongly suggest having some tooling to check the setup is correct.
In PLC4X we have a groovy script that is executed by the build. Depending on which parts you are building, it checks if required tools are installed in sufficient versions. Not sure if sbt allows similar checks, but perhaps our setup in PLC4X could help?
Otherwise with all these C, C++ tools it's really hard for newbies to find what's going wrong.
Chris
-----Ursprüngliche Nachricht-----
Von: Beckerle, Mike <mb...@owlcyberdefense.com>
Gesendet: Donnerstag, 11. März 2021 17:38
An: dev@daffodil.apache.org
Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Setup of the C-toolchain as well as scala shoulnd't be a problem so long as these are all no-cost tools.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Is that all? :-). I would add some criteria testing runtime2's conformance to the DFDL 1.0 specification as well. Here goes...
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
Yup. We have 7 schemas in daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" examples of messages with sufficient complexity to indicate that other real messages should be possible.
2) contains built-in tests for one or several such examples, showing each supported aspect working.
Yup. We have TDML tests and binary data/XML infoset files for each of these 7 schemas.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
Yup. We have Scala test classes for each of these 7 schemas' TDML tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
Needs work. The top-level README lists daffodil-runtime2's build requirements under Build Requirements and has a single sentence under Getting Started telling developers that they will need a C toolchain in order to build daffodil-runtime2. However, I didn't make it clear in the README that developers must setup a C toolchain as well as a Java/Scala toolchain and I need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step instructions showing developers how to setup a C toolchain on Linux and Windows. I hope developers won't mind if we make it mandatory to setup both Java/Scala and C toolchains before building Daffodil. I don't like the idea of some developers building Daffodil without checking that the C files compile successfully and the runtime2 tests pass successfully. I think it's sufficient to guarantee that end-users of Daffodil never have to install a C toolchain to use Daffodil. When end-users install Daffodil, they will get a pure Scala Daffodil just like before. The only user-visible change is that this Daffodil will know how to generate, compile, and run C code on the end-user's system if the user asks Daffodil to do that.
I also would add another criteria:
5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests modified to run on runtime2 as well as runtime1 during 'sbt test'.
We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 1.0 specification. We should be able to find a subset of these tests that should pass on runtime2 as well and verify that they do keep passing when run on both runtime1 and runtime2. This step would go a long way to ensure that people working on other aspects of Daffodil do not break the C backend inadvertently without knowing it. Note that this makes it even more mandatory that Daffodil developers setup both Java/Scala and C toolchains. What do you all think?
-----Original Message-----
From: Beckerle, Mike <mb...@owlcyberdefense.com>
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Acceptance criteria to me:
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
2) contains built-in tests for one or several such examples, showing each supported aspect working.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
Daffodil's Confluence describes Runtime2's design here:
https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
* Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
* Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are welcome:
1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
3. Which of the Runtime2 ToDos need to be done before the merge as well?
Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
AW: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi all,
I would strongly suggest having some tooling to check the setup is correct.
In PLC4X we have a groovy script that is executed by the build. Depending on which parts you are building, it checks if required tools are installed in sufficient versions. Not sure if sbt allows similar checks, but perhaps our setup in PLC4X could help?
Otherwise with all these C, C++ tools it's really hard for newbies to find what's going wrong.
Chris
-----Ursprüngliche Nachricht-----
Von: Beckerle, Mike <mb...@owlcyberdefense.com>
Gesendet: Donnerstag, 11. März 2021 17:38
An: dev@daffodil.apache.org
Betreff: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Setup of the C-toolchain as well as scala shoulnd't be a problem so long as these are all no-cost tools.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Is that all? :-). I would add some criteria testing runtime2's conformance to the DFDL 1.0 specification as well. Here goes...
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
Yup. We have 7 schemas in daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" examples of messages with sufficient complexity to indicate that other real messages should be possible.
2) contains built-in tests for one or several such examples, showing each supported aspect working.
Yup. We have TDML tests and binary data/XML infoset files for each of these 7 schemas.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
Yup. We have Scala test classes for each of these 7 schemas' TDML tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
Needs work. The top-level README lists daffodil-runtime2's build requirements under Build Requirements and has a single sentence under Getting Started telling developers that they will need a C toolchain in order to build daffodil-runtime2. However, I didn't make it clear in the README that developers must setup a C toolchain as well as a Java/Scala toolchain and I need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step instructions showing developers how to setup a C toolchain on Linux and Windows. I hope developers won't mind if we make it mandatory to setup both Java/Scala and C toolchains before building Daffodil. I don't like the idea of some developers building Daffodil without checking that the C files compile successfully and the runtime2 tests pass successfully. I think it's sufficient to guarantee that end-users of Daffodil never have to install a C toolchain to use Daffodil. When end-users install Daffodil, they will get a pure Scala Daffodil just like before. The only user-visible change is that this Daffodil will know how to generate, compile, and run C code on the end-user's system if the user asks Daffodil to do that.
I also would add another criteria:
5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests modified to run on runtime2 as well as runtime1 during 'sbt test'.
We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 1.0 specification. We should be able to find a subset of these tests that should pass on runtime2 as well and verify that they do keep passing when run on both runtime1 and runtime2. This step would go a long way to ensure that people working on other aspects of Daffodil do not break the C backend inadvertently without knowing it. Note that this makes it even more mandatory that Daffodil developers setup both Java/Scala and C toolchains. What do you all think?
-----Original Message-----
From: Beckerle, Mike <mb...@owlcyberdefense.com>
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Acceptance criteria to me:
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
2) contains built-in tests for one or several such examples, showing each supported aspect working.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
Daffodil's Confluence describes Runtime2's design here:
https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
* Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
* Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are welcome:
1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
3. Which of the Runtime2 ToDos need to be done before the merge as well?
Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
Re: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
Setup of the C-toolchain as well as scala shoulnd't be a problem so long as these are all no-cost tools.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Thursday, March 11, 2021 11:34 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: RE: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Is that all? :-). I would add some criteria testing runtime2's conformance to the DFDL 1.0 specification as well. Here goes...
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
Yup. We have 7 schemas in daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" examples of messages with sufficient complexity to indicate that other real messages should be possible.
2) contains built-in tests for one or several such examples, showing each supported aspect working.
Yup. We have TDML tests and binary data/XML infoset files for each of these 7 schemas.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
Yup. We have Scala test classes for each of these 7 schemas' TDML tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
Needs work. The top-level README lists daffodil-runtime2's build requirements under Build Requirements and has a single sentence under Getting Started telling developers that they will need a C toolchain in order to build daffodil-runtime2. However, I didn't make it clear in the README that developers must setup a C toolchain as well as a Java/Scala toolchain and I need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step instructions showing developers how to setup a C toolchain on Linux and Windows. I hope developers won't mind if we make it mandatory to setup both Java/Scala and C toolchains before building Daffodil. I don't like the idea of some developers building Daffodil without checking that the C files compile successfully and the runtime2 tests pass successfully. I think it's sufficient to guarantee that end-users of Daffodil never have to install a C toolchain to use Daffodil. When end-users install Daffodil, they will get a pure Scala Daffodil just like before. The only user-visible change is that this Daffodil will know how to generate, compile, and run C code on the end-user's system if the user asks Daffodil to do that.
I also would add another criteria:
5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests modified to run on runtime2 as well as runtime1 during 'sbt test'.
We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 1.0 specification. We should be able to find a subset of these tests that should pass on runtime2 as well and verify that they do keep passing when run on both runtime1 and runtime2. This step would go a long way to ensure that people working on other aspects of Daffodil do not break the C backend inadvertently without knowing it. Note that this makes it even more mandatory that Daffodil developers setup both Java/Scala and C toolchains. What do you all think?
-----Original Message-----
From: Beckerle, Mike <mb...@owlcyberdefense.com>
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Acceptance criteria to me:
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
2) contains built-in tests for one or several such examples, showing each supported aspect working.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
Daffodil's Confluence describes Runtime2's design here:
https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
* Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
* Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are welcome:
1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
3. Which of the Runtime2 ToDos need to be done before the merge as well?
Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
RE: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by "Interrante, John A (GE Research, US)" <Jo...@ge.com>.
Is that all? :-). I would add some criteria testing runtime2's conformance to the DFDL 1.0 specification as well. Here goes...
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
Yup. We have 7 schemas in daffodil-test/src/test/resources/org/apache/daffodil/runtime2 that have "real" examples of messages with sufficient complexity to indicate that other real messages should be possible.
2) contains built-in tests for one or several such examples, showing each supported aspect working.
Yup. We have TDML tests and binary data/XML infoset files for each of these 7 schemas.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
Yup. We have Scala test classes for each of these 7 schemas' TDML tests in daffodil-test/src/test/scala/org/apache/daffodil/runtime2.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
Needs work. The top-level README lists daffodil-runtime2's build requirements under Build Requirements and has a single sentence under Getting Started telling developers that they will need a C toolchain in order to build daffodil-runtime2. However, I didn't make it clear in the README that developers must setup a C toolchain as well as a Java/Scala toolchain and I need to add a "C Setup and Notes" page to the Confluence Wiki with step-by-step instructions showing developers how to setup a C toolchain on Linux and Windows. I hope developers won't mind if we make it mandatory to setup both Java/Scala and C toolchains before building Daffodil. I don't like the idea of some developers building Daffodil without checking that the C files compile successfully and the runtime2 tests pass successfully. I think it's sufficient to guarantee that end-users of Daffodil never have to install a C toolchain to use Daffodil. When end-users install Daffodil, they will get a pure Scala Daffodil just like before. The only user-visible change is that this Daffodil will know how to generate, compile, and run C code on the end-user's system if the user asks Daffodil to do that.
I also would add another criteria:
5) A subset of Daffodil's DFDL 1.0 specification conformance TDML tests modified to run on runtime2 as well as runtime1 during 'sbt test'.
We have lots and lots of TDML tests checking that Daffodil conforms to the DFDL 1.0 specification. We should be able to find a subset of these tests that should pass on runtime2 as well and verify that they do keep passing when run on both runtime1 and runtime2. This step would go a long way to ensure that people working on other aspects of Daffodil do not break the C backend inadvertently without knowing it. Note that this makes it even more mandatory that Daffodil developers setup both Java/Scala and C toolchains. What do you all think?
-----Original Message-----
From: Beckerle, Mike <mb...@owlcyberdefense.com>
Sent: Thursday, March 11, 2021 9:27 AM
To: dev@daffodil.apache.org
Subject: EXT: Re: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Acceptance criteria to me:
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
2) contains built-in tests for one or several such examples, showing each supported aspect working.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
Daffodil's Confluence describes Runtime2's design here:
https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
* Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
* Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are welcome:
1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
3. Which of the Runtime2 ToDos need to be done before the merge as well?
Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.
Re: Acceptance criteria for merging DFDL-to-C backend
(runtime2-2202)?
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
Acceptance criteria to me:
1) Sufficient functionality to describe at least one example, of sufficient message complexity to indicate that other similar "real" examples should be possible
2) contains built-in tests for one or several such examples, showing each supported aspect working.
3) the tests are fully integrated - they run every time 'sbt test' is run on daffodil.
4) setup instructions for developers are there so that people know how to insure the required C tool-chain elements are there. These need to work on Linux and Windows.
I believe these things insure that people working on other aspects of Daffodil will not be inadvertently breaking the C backend indetectably. That's one major concern I would have.
________________________________
From: Interrante, John A (GE Research, US) <Jo...@ge.com>
Sent: Wednesday, March 10, 2021 4:24 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Acceptance criteria for merging DFDL-to-C backend (runtime2-2202)?
Thanks to Mike's suggestion, I have moved the Runtime2 ToDos (changes requested by reviewers) from the DAFFODIL-2202<https://issues.apache.org/jira/browse/DAFFODIL-2202> issue to an AsciiDoc document in the dev/design-notes subtree of the Daffodil website which you can read here:
https://daffodil.apache.org/dev/design-notes/runtime2-todos/
I would like to discuss which acceptance criteria the runtime2-2202 development branch must meet before I can submit a pull request to merge the DFDL-to-C backend and code generator into the main branch. I plan to address the Runtime2 ToDos and I especially want to run some of Daffodil's TDML tests on the new runtime2 backend as well as the runtime1 backend by adding defaultImplementations="daffodil daffodil-runtime2" to certain TDML tests' attributes. (Although I suggest we use the shorter name "daf-c" or "runtime2" because "daffodil daffodil-runtime2" is a lot of characters to put into the defaultImplementations attribute.)
Daffodil's Confluence describes Runtime2's design here:
https://cwiki.apache.org/confluence/display/DAFFODIL/WIP:+Daffodil+Runtime+2
In particular, it suggests we divide the implementation of runtime2 into two distinct phases:
* Phase 1 (aka Runtime2P1): No expressions. All lengths are fixed. All arrays have fixed length.
* Phase 2 (aka Runtime2P2): Adding the DFDL expression language, lengthKind 'explicit', occursCountKind 'expression'.
I think phase 1 is almost done but we need to run a subset of Daffodil's TDML tests on runtime2 before we can really say for sure.
Here is an initial set of discussion points - more questions and criteria are welcome:
1. Which Daffodil TDML tests do we need to run on runtime2 to assert that phase 1 is complete?
2. Can we merge runtime2 when these tests pass and then build out phase 2 in the main branch, hopefully with help from other developers once they see how useful phase 1 is?
3. Which of the Runtime2 ToDos need to be done before the merge as well?
Once we agree on a minimal set of acceptance criteria for the merge, I'll copy the criteria to the JIRA issue.