You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by "Interrante, John A (GE Research, US)" <in...@research.ge.com> on 2020/10/05 13:28:35 UTC
Subproject names proposed for discussion
Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
What do devs think about us renaming some subprojects like this?
rename daffodil-core to daffodil-schema-compiler
leave daffodil-lib alone
rename daffodil-runtime1 to daffodil-backend-parser-scala
rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
rename daffodil-runtime2 to daffodil-backend-generator-c
Re: Subproject names proposed for discussion
Posted by "Sloane, Brandon" <bs...@owlcyberdefense.com>.
+1
________________________________
From: Beckerle, Mike <mb...@owlcyberdefense.com>
Sent: Monday, October 5, 2020 10:23 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: Subproject names proposed for discussion
+1 from me.
________________________________
From: Interrante, John A (GE Research, US) <in...@research.ge.com>
Sent: Monday, October 5, 2020 9:28 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Subproject names proposed for discussion
Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
What do devs think about us renaming some subprojects like this?
rename daffodil-core to daffodil-schema-compiler
leave daffodil-lib alone
rename daffodil-runtime1 to daffodil-backend-parser-scala
rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
rename daffodil-runtime2 to daffodil-backend-generator-c
Re: Subproject names proposed for discussion
Posted by Steve Lawrence <sl...@apache.org>.
One thing to maybe consider related to project names. In order to run an
sbt command for a subproject, you use sbt <subproject>/command. So for
example, to build the CLI:
sbt daffodil-cli/packageBin
If we remove the prefix from directory names, it makes sense to also
remove the prefix from sbt subproject names. So the above becomes
sbt cli/packageBin
This seems perfectly reasonable. One minor concern is the daffodil-test
project becomes just "test", which is perhaps a bit confusing since "sbt
test" is a normal and very common command. So to run tests in just the
test subproject, you would run
sbt test/test
That isn't awful, but does a bit weird? Maybe a new name for the "test"
project would also make this more clear, and could also describe what
these tests are more about? Similar to test-ibm1 or test-stdLayout?
On 10/5/20 11:59 AM, Steve Lawrence wrote:
> I'm fine with just removing it completely. As I said, I don't think the
> reasons I listed are strong reasons for keeping a prefix.
>
> On 10/5/20 11:36 AM, Beckerle, Mike wrote:
>> A shorter prefix like just "daf-" would bother me less.
>>
>> It literally becomes screen-real-estate using IDEs because the project trees extend from the left of the screen, and they just go too far right before you get to the actual files contained.
>> ________________________________
>> From: Steve Lawrence <sl...@apache.org>
>> Sent: Monday, October 5, 2020 11:25 AM
>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>> Subject: Re: Subproject names proposed for discussion
>>
>> Your suggestion seems reasonable to me. Adds a few more top level
>> directories by splitting lib and schema-compiler which I'm not a huge
>> fan of, but it is logical and reduces the size the lib jars.
>>
>> The one thing I like about the "daffodil-" prefix is it makes the code
>> directories stand out a bit more. For example, the containers, project,
>> target, tutorials, etc. directories or more about infrastructure. It
>> makes it eaiser to ignore those when scanning through directories (of
>> which this change would add quite a few more). I think this is also
>> pretty common among java projects, but I don't think they all do it. But
>> I'm not sure if either of those are strong reasons to keep it. We have
>> had problems with long paths in the past on windows, so removing the
>> prefix could help with that...
>>
>>
>> On 10/5/20 11:11 AM, Beckerle, Mike wrote:
>>> Some reponses inline below
>>>
>>> ________________________________
>>> From: Steve Lawrence <sl...@apache.org>
>>> Sent: Monday, October 5, 2020 10:48 AM
>>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>>> Subject: Re: Subproject names proposed for discussion
>>>
>>> A handful of unrelated thoughts, maybe overthinking things and I don't
>>> feel strongly about anything below, but renaming is always pain so it'd
>>> be nice to ensure we have something future proof.
>>>
>>> 1) Is there any benefit organizationally to having all backends being in
>>> the same directory?
>>>
>>> I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
>>>
>>> 2) From a sorting perspective, it'd be nice if the scala projects were
>>> together, so having it be scala-parser and scala-unparser rather than
>>> parser-scala and unparser-scala has advantages.
>>> Agree.
>>>
>>> 3) Maybe the scala parser/unparser should be considered the same "scala"
>>> runtime, and so parser/unparser should be subdirectories of a
>>> "daffodil-backend-scala" subdirectory?
>>>
>>> There is no reason to keep the scala parser and unparser separate.
>>>
>>> That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
>>>
>>> 4) Is there even a benefit to separating parser/unparser into separatejars? There's so much shared logic between the two, and there's even a
>>> bunch of unparsing stuff in the parser jar. Should we just combine them
>>> under the same backend?
>>>
>>> Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
>>>
>>> Taking all of the above into account, perhaps something like this:
>>>
>>> ...
>>> |-- daffodil-backends
>>>
>>> I would not add the backends parent.
>>>
>>> daffodil-backend-scala
>>>
>>> | |-- daffodil-scala
>>> | | `-- src
>>> | `-- daffodil-generator-c
>>>
>>> I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
>>>
>>> While we're at it, can't we drop the "daffodil-" prefix on all these?
>>>
>>> | `-- src
>>> |-- daffodil-lib
>>> | `-- src
>>> |-- daffodil-schema-compiler
>>>
>>> This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
>>>
>>> This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
>>>
>>> | `-- src
>>> ...
>>>
>>> 5) Is there something better than "backend" for describing these. I
>>> can't think of anything. Does the DFDL spec have a concept of this?
>>>
>>> There is no concept of this in the DFDL spec to borrow from.
>>>
>>> 6) Are there any benefits to using "codenames". My thinking is maybe
>>> someday there could be multiple "scala" backends with different
>>> goals/extensions, and so "daffodil-scala" is too generic. Codenames
>>> would be more like what we have today, except real code names might be
>>> easier to remember than "runtime1" and "runtime2". Disadvantage is
>>> there's less discoverability, but a README could be added with short
>>> descriptions about what the backends try to accomplish. Not sure I like
>>> this, but thought I'd throw it out there.
>>>
>>> Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
>>>
>>> Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
>>>
>>> My summary:
>>>
>>> daffodil
>>> backend-scala-schema-compiler
>>> core-schema-compiler
>>> backend-c-generator-schema-compiler
>>> lib
>>> backend-scala-parser-lib
>>> backend-scala-unparser-lib
>>> backend-c-lib
>>> tdml-lib
>>> tdml-processor
>>> .....
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 10/5/20 10:23 AM, Beckerle, Mike wrote:
>>>> +1 from me.
>>>>
>>>> ________________________________
>>>> From: Interrante, John A (GE Research, US) <in...@research.ge.com>
>>>> Sent: Monday, October 5, 2020 9:28 AM
>>>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>>>> Subject: Subproject names proposed for discussion
>>>>
>>>> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>>>>
>>>> What do devs think about us renaming some subprojects like this?
>>>>
>>>> rename daffodil-core to daffodil-schema-compiler
>>>> leave daffodil-lib alone
>>>> rename daffodil-runtime1 to daffodil-backend-parser-scala
>>>> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
>>>> rename daffodil-runtime2 to daffodil-backend-generator-c
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
Re: Subproject names proposed for discussion
Posted by Steve Lawrence <sl...@apache.org>.
I'd prefer to not add a number to the name. It doesn't help to make any
distinction if we add a suite2. If we do decide we want a second test
suite, we can just rename them so that the separation is more clear.
Also, I think another reason test-stdayout is separate is because it
follows a very different layout than our normal tests. It doesn't have
any org/apache stuff, and includes schemas in src/main that reference
one another and uses namespacing to ensure there are no conflicts
between the two schemas.
I do think "test-suite" is a reasonable name.
On 10/5/20 5:15 PM, Beckerle, Mike wrote:
> The problematic "daffodil-test" module is much too big, that's why the tests for the std-layout were broken out as a separate module, and test-ibm1 was kept separate also.
>
> The notion was to stop piling all testing into the daffodil-test module.
>
> I would suggest that daffodil-test be renamed "daf-test" or if we're dropping "daf-" then "test-suite1" would be a good name.
>
>
> ________________________________
> From: Interrante, John A (GE Research, US) <in...@research.ge.com>
> Sent: Monday, October 5, 2020 1:29 PM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Subproject names proposed for discussion
>
> OK, I'll summarize the discussion to see if we have a consensus now. Right now we have the following subprojects:
>
> - containers
> - daffodil-cli
> - daffodil-core
> - daffodil-io
> - daffodil-lib
> - daffodil-macro-lib
> - daffodil-propgen
> - daffodil-runtime1
> - daffodil-runtime1-unparser
> - daffodil-runtime2
> - daffodil-sapi
> - daffodil-tdml-lib
> - daffodil-tdml-processor
> - daffodil-test
> - daffodil-test-ibm
> - daffodil-udf
> - project
> - test-stdLayout
> - tutorials
>
> Now we might have the following new names:
>
> - containers
> - daf-backend-c-generator
> - daf-backend-scala-parser (to be merged as daf-backend-scala)
> - daf-backend-scala -unparser (to be merged as daf-backend-scala)
> - daf-cli
> - daf-io
> - daf-lib
> - daf-macro-lib
> - daf-propgen
> - daf-sapi
> - daf-schema-compiler
> - daf-tdml-lib
> - daf-tdml-processor
> - daf-test
> - daf-test-ibm
> - daf-udf
> - project
> - test-stdLayout
> - tutorials
>
> If we dropped the daf- prefix, we might need another name for test-stdLayout (it would not be clear to me what the difference between test and test-stdLayout is). I see Steve just commented that the sbt commands for subprojects also would need to drop their "daffodil-" prefix and this would introduce a confusion with the "sbt test" command. I'll let devs decide how to make the names of test subprojects and sbt commands clearer.
>
> However, Mike just pointed out we should build another set of "c-cli" C source files (the C source files used for CLI parsing and conversion to/from XML to support TDML tests) separately from the "common runtime" C source files (the primitive parsers/unparsers called from the generated code) and the "generated code" C source files (the result of compiling the DFDL schema). He also said in a review comment:
>
> Eventually, this "static code" is going to become a rather large library. As big as both runtime1 parser and unparser together. Bigger given that this is C code, in terms of Lines of code of course. Think in terms of 50K lines of code kind of library. So I think it needs to be built as a static library.
>
> These comments motivate me to say that we eventually might need several subprojects for the C runtime2 backend, not just one, since one subproject should build one library, correct? I'm torn a bit, though. We can put C source files in subdirectories like packages and generate multiple libraries from these multiple directories using the sbt C compiler plugin without any difficulty too. (I know the topic of whether to compile C source files in the sbt build and distribute C-code libraries with daffodil warrants its own discussion on the dev list too.)
>
> If we keep going without any further changes, then after two more pull requests, we would end up with:
>
> - containers
> - daf-backend-c-generator
> - daf-backend-scala
> - daf-cli
> - daf-io
> - daf-lib
> - daf-macro-lib
> - daf-propgen
> - daf-sapi
> - daf-schema-compiler
> - daf-tdml-lib
> - daf-tdml-processor
> - daf-test
> - daf-test-ibm
> - daf-udf
> - project
> - test-stdLayout
> - tutorials
>
> -----Original Message-----
> From: Steve Lawrence <sl...@apache.org>
> Sent: Monday, October 5, 2020 12:00 PM
> To: dev@daffodil.apache.org
> Subject: EXT: Re: Subproject names proposed for discussion
>
> I'm fine with just removing it completely. As I said, I don't think the reasons I listed are strong reasons for keeping a prefix.
>
> On 10/5/20 11:36 AM, Beckerle, Mike wrote:
>> A shorter prefix like just "daf-" would bother me less.
>>
>> It literally becomes screen-real-estate using IDEs because the project trees extend from the left of the screen, and they just go too far right before you get to the actual files contained.
>> ________________________________
>> From: Steve Lawrence <sl...@apache.org>
>> Sent: Monday, October 5, 2020 11:25 AM
>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>> Subject: Re: Subproject names proposed for discussion
>>
>> Your suggestion seems reasonable to me. Adds a few more top level
>> directories by splitting lib and schema-compiler which I'm not a huge
>> fan of, but it is logical and reduces the size the lib jars.
>>
>> The one thing I like about the "daffodil-" prefix is it makes the code
>> directories stand out a bit more. For example, the containers,
>> project, target, tutorials, etc. directories or more about
>> infrastructure. It makes it eaiser to ignore those when scanning
>> through directories (of which this change would add quite a few more).
>> I think this is also pretty common among java projects, but I don't
>> think they all do it. But I'm not sure if either of those are strong
>> reasons to keep it. We have had problems with long paths in the past
>> on windows, so removing the prefix could help with that...
>>
>>
>> On 10/5/20 11:11 AM, Beckerle, Mike wrote:
>>> Some reponses inline below
>>>
>>> ________________________________
>>> From: Steve Lawrence <sl...@apache.org>
>>> Sent: Monday, October 5, 2020 10:48 AM
>>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>>> Subject: Re: Subproject names proposed for discussion
>>>
>>> A handful of unrelated thoughts, maybe overthinking things and I
>>> don't feel strongly about anything below, but renaming is always pain
>>> so it'd be nice to ensure we have something future proof.
>>>
>>> 1) Is there any benefit organizationally to having all backends being
>>> in the same directory?
>>>
>>> I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
>>>
>>> 2) From a sorting perspective, it'd be nice if the scala projects
>>> were together, so having it be scala-parser and scala-unparser rather
>>> than parser-scala and unparser-scala has advantages.
>>> Agree.
>>>
>>> 3) Maybe the scala parser/unparser should be considered the same "scala"
>>> runtime, and so parser/unparser should be subdirectories of a
>>> "daffodil-backend-scala" subdirectory?
>>>
>>> There is no reason to keep the scala parser and unparser separate.
>>>
>>> That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
>>>
>>> 4) Is there even a benefit to separating parser/unparser into
>>> separatejars? There's so much shared logic between the two, and
>>> there's even a bunch of unparsing stuff in the parser jar. Should we just combine them under the same backend?
>>>
>>> Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
>>>
>>> Taking all of the above into account, perhaps something like this:
>>>
>>> ...
>>> |-- daffodil-backends
>>>
>>> I would not add the backends parent.
>>>
>>> daffodil-backend-scala
>>>
>>> | |-- daffodil-scala
>>> | | `-- src
>>> | `-- daffodil-generator-c
>>>
>>> I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
>>>
>>> While we're at it, can't we drop the "daffodil-" prefix on all these?
>>>
>>> | `-- src
>>> |-- daffodil-lib
>>> | `-- src
>>> |-- daffodil-schema-compiler
>>>
>>> This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
>>>
>>> This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
>>>
>>> | `-- src
>>> ...
>>>
>>> 5) Is there something better than "backend" for describing these. I
>>> can't think of anything. Does the DFDL spec have a concept of this?
>>>
>>> There is no concept of this in the DFDL spec to borrow from.
>>>
>>> 6) Are there any benefits to using "codenames". My thinking is maybe
>>> someday there could be multiple "scala" backends with different
>>> goals/extensions, and so "daffodil-scala" is too generic. Codenames
>>> would be more like what we have today, except real code names might
>>> be easier to remember than "runtime1" and "runtime2". Disadvantage is
>>> there's less discoverability, but a README could be added with short
>>> descriptions about what the backends try to accomplish. Not sure I
>>> like this, but thought I'd throw it out there.
>>>
>>> Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
>>>
>>> Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
>>>
>>> My summary:
>>>
>>> daffodil
>>> backend-scala-schema-compiler
>>> core-schema-compiler
>>> backend-c-generator-schema-compiler
>>> lib
>>> backend-scala-parser-lib
>>> backend-scala-unparser-lib
>>> backend-c-lib
>>> tdml-lib
>>> tdml-processor
>>> .....
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 10/5/20 10:23 AM, Beckerle, Mike wrote:
>>>> +1 from me.
>>>>
>>>> ________________________________
>>>> From: Interrante, John A (GE Research, US)
>>>> <in...@research.ge.com>
>>>> Sent: Monday, October 5, 2020 9:28 AM
>>>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>>>> Subject: Subproject names proposed for discussion
>>>>
>>>> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>>>>
>>>> What do devs think about us renaming some subprojects like this?
>>>>
>>>> rename daffodil-core to daffodil-schema-compiler
>>>> leave daffodil-lib alone
>>>> rename daffodil-runtime1 to daffodil-backend-parser-scala
>>>> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
>>>> rename daffodil-runtime2 to daffodil-backend-generator-c
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Subproject names proposed for discussion
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
The problematic "daffodil-test" module is much too big, that's why the tests for the std-layout were broken out as a separate module, and test-ibm1 was kept separate also.
The notion was to stop piling all testing into the daffodil-test module.
I would suggest that daffodil-test be renamed "daf-test" or if we're dropping "daf-" then "test-suite1" would be a good name.
________________________________
From: Interrante, John A (GE Research, US) <in...@research.ge.com>
Sent: Monday, October 5, 2020 1:29 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Subproject names proposed for discussion
OK, I'll summarize the discussion to see if we have a consensus now. Right now we have the following subprojects:
- containers
- daffodil-cli
- daffodil-core
- daffodil-io
- daffodil-lib
- daffodil-macro-lib
- daffodil-propgen
- daffodil-runtime1
- daffodil-runtime1-unparser
- daffodil-runtime2
- daffodil-sapi
- daffodil-tdml-lib
- daffodil-tdml-processor
- daffodil-test
- daffodil-test-ibm
- daffodil-udf
- project
- test-stdLayout
- tutorials
Now we might have the following new names:
- containers
- daf-backend-c-generator
- daf-backend-scala-parser (to be merged as daf-backend-scala)
- daf-backend-scala -unparser (to be merged as daf-backend-scala)
- daf-cli
- daf-io
- daf-lib
- daf-macro-lib
- daf-propgen
- daf-sapi
- daf-schema-compiler
- daf-tdml-lib
- daf-tdml-processor
- daf-test
- daf-test-ibm
- daf-udf
- project
- test-stdLayout
- tutorials
If we dropped the daf- prefix, we might need another name for test-stdLayout (it would not be clear to me what the difference between test and test-stdLayout is). I see Steve just commented that the sbt commands for subprojects also would need to drop their "daffodil-" prefix and this would introduce a confusion with the "sbt test" command. I'll let devs decide how to make the names of test subprojects and sbt commands clearer.
However, Mike just pointed out we should build another set of "c-cli" C source files (the C source files used for CLI parsing and conversion to/from XML to support TDML tests) separately from the "common runtime" C source files (the primitive parsers/unparsers called from the generated code) and the "generated code" C source files (the result of compiling the DFDL schema). He also said in a review comment:
Eventually, this "static code" is going to become a rather large library. As big as both runtime1 parser and unparser together. Bigger given that this is C code, in terms of Lines of code of course. Think in terms of 50K lines of code kind of library. So I think it needs to be built as a static library.
These comments motivate me to say that we eventually might need several subprojects for the C runtime2 backend, not just one, since one subproject should build one library, correct? I'm torn a bit, though. We can put C source files in subdirectories like packages and generate multiple libraries from these multiple directories using the sbt C compiler plugin without any difficulty too. (I know the topic of whether to compile C source files in the sbt build and distribute C-code libraries with daffodil warrants its own discussion on the dev list too.)
If we keep going without any further changes, then after two more pull requests, we would end up with:
- containers
- daf-backend-c-generator
- daf-backend-scala
- daf-cli
- daf-io
- daf-lib
- daf-macro-lib
- daf-propgen
- daf-sapi
- daf-schema-compiler
- daf-tdml-lib
- daf-tdml-processor
- daf-test
- daf-test-ibm
- daf-udf
- project
- test-stdLayout
- tutorials
-----Original Message-----
From: Steve Lawrence <sl...@apache.org>
Sent: Monday, October 5, 2020 12:00 PM
To: dev@daffodil.apache.org
Subject: EXT: Re: Subproject names proposed for discussion
I'm fine with just removing it completely. As I said, I don't think the reasons I listed are strong reasons for keeping a prefix.
On 10/5/20 11:36 AM, Beckerle, Mike wrote:
> A shorter prefix like just "daf-" would bother me less.
>
> It literally becomes screen-real-estate using IDEs because the project trees extend from the left of the screen, and they just go too far right before you get to the actual files contained.
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Monday, October 5, 2020 11:25 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: Subproject names proposed for discussion
>
> Your suggestion seems reasonable to me. Adds a few more top level
> directories by splitting lib and schema-compiler which I'm not a huge
> fan of, but it is logical and reduces the size the lib jars.
>
> The one thing I like about the "daffodil-" prefix is it makes the code
> directories stand out a bit more. For example, the containers,
> project, target, tutorials, etc. directories or more about
> infrastructure. It makes it eaiser to ignore those when scanning
> through directories (of which this change would add quite a few more).
> I think this is also pretty common among java projects, but I don't
> think they all do it. But I'm not sure if either of those are strong
> reasons to keep it. We have had problems with long paths in the past
> on windows, so removing the prefix could help with that...
>
>
> On 10/5/20 11:11 AM, Beckerle, Mike wrote:
>> Some reponses inline below
>>
>> ________________________________
>> From: Steve Lawrence <sl...@apache.org>
>> Sent: Monday, October 5, 2020 10:48 AM
>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>> Subject: Re: Subproject names proposed for discussion
>>
>> A handful of unrelated thoughts, maybe overthinking things and I
>> don't feel strongly about anything below, but renaming is always pain
>> so it'd be nice to ensure we have something future proof.
>>
>> 1) Is there any benefit organizationally to having all backends being
>> in the same directory?
>>
>> I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
>>
>> 2) From a sorting perspective, it'd be nice if the scala projects
>> were together, so having it be scala-parser and scala-unparser rather
>> than parser-scala and unparser-scala has advantages.
>> Agree.
>>
>> 3) Maybe the scala parser/unparser should be considered the same "scala"
>> runtime, and so parser/unparser should be subdirectories of a
>> "daffodil-backend-scala" subdirectory?
>>
>> There is no reason to keep the scala parser and unparser separate.
>>
>> That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
>>
>> 4) Is there even a benefit to separating parser/unparser into
>> separatejars? There's so much shared logic between the two, and
>> there's even a bunch of unparsing stuff in the parser jar. Should we just combine them under the same backend?
>>
>> Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
>>
>> Taking all of the above into account, perhaps something like this:
>>
>> ...
>> |-- daffodil-backends
>>
>> I would not add the backends parent.
>>
>> daffodil-backend-scala
>>
>> | |-- daffodil-scala
>> | | `-- src
>> | `-- daffodil-generator-c
>>
>> I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
>>
>> While we're at it, can't we drop the "daffodil-" prefix on all these?
>>
>> | `-- src
>> |-- daffodil-lib
>> | `-- src
>> |-- daffodil-schema-compiler
>>
>> This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
>>
>> This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
>>
>> | `-- src
>> ...
>>
>> 5) Is there something better than "backend" for describing these. I
>> can't think of anything. Does the DFDL spec have a concept of this?
>>
>> There is no concept of this in the DFDL spec to borrow from.
>>
>> 6) Are there any benefits to using "codenames". My thinking is maybe
>> someday there could be multiple "scala" backends with different
>> goals/extensions, and so "daffodil-scala" is too generic. Codenames
>> would be more like what we have today, except real code names might
>> be easier to remember than "runtime1" and "runtime2". Disadvantage is
>> there's less discoverability, but a README could be added with short
>> descriptions about what the backends try to accomplish. Not sure I
>> like this, but thought I'd throw it out there.
>>
>> Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
>>
>> Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
>>
>> My summary:
>>
>> daffodil
>> backend-scala-schema-compiler
>> core-schema-compiler
>> backend-c-generator-schema-compiler
>> lib
>> backend-scala-parser-lib
>> backend-scala-unparser-lib
>> backend-c-lib
>> tdml-lib
>> tdml-processor
>> .....
>>
>>
>>
>>
>>
>>
>> On 10/5/20 10:23 AM, Beckerle, Mike wrote:
>>> +1 from me.
>>>
>>> ________________________________
>>> From: Interrante, John A (GE Research, US)
>>> <in...@research.ge.com>
>>> Sent: Monday, October 5, 2020 9:28 AM
>>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>>> Subject: Subproject names proposed for discussion
>>>
>>> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>>>
>>> What do devs think about us renaming some subprojects like this?
>>>
>>> rename daffodil-core to daffodil-schema-compiler
>>> leave daffodil-lib alone
>>> rename daffodil-runtime1 to daffodil-backend-parser-scala
>>> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
>>> rename daffodil-runtime2 to daffodil-backend-generator-c
>>>
>>>
>>
>>
>>
>
>
Subproject names proposed for discussion
Posted by "Interrante, John A (GE Research, US)" <in...@research.ge.com>.
OK, I'll summarize the discussion to see if we have a consensus now. Right now we have the following subprojects:
- containers
- daffodil-cli
- daffodil-core
- daffodil-io
- daffodil-lib
- daffodil-macro-lib
- daffodil-propgen
- daffodil-runtime1
- daffodil-runtime1-unparser
- daffodil-runtime2
- daffodil-sapi
- daffodil-tdml-lib
- daffodil-tdml-processor
- daffodil-test
- daffodil-test-ibm
- daffodil-udf
- project
- test-stdLayout
- tutorials
Now we might have the following new names:
- containers
- daf-backend-c-generator
- daf-backend-scala-parser (to be merged as daf-backend-scala)
- daf-backend-scala -unparser (to be merged as daf-backend-scala)
- daf-cli
- daf-io
- daf-lib
- daf-macro-lib
- daf-propgen
- daf-sapi
- daf-schema-compiler
- daf-tdml-lib
- daf-tdml-processor
- daf-test
- daf-test-ibm
- daf-udf
- project
- test-stdLayout
- tutorials
If we dropped the daf- prefix, we might need another name for test-stdLayout (it would not be clear to me what the difference between test and test-stdLayout is). I see Steve just commented that the sbt commands for subprojects also would need to drop their "daffodil-" prefix and this would introduce a confusion with the "sbt test" command. I'll let devs decide how to make the names of test subprojects and sbt commands clearer.
However, Mike just pointed out we should build another set of "c-cli" C source files (the C source files used for CLI parsing and conversion to/from XML to support TDML tests) separately from the "common runtime" C source files (the primitive parsers/unparsers called from the generated code) and the "generated code" C source files (the result of compiling the DFDL schema). He also said in a review comment:
Eventually, this "static code" is going to become a rather large library. As big as both runtime1 parser and unparser together. Bigger given that this is C code, in terms of Lines of code of course. Think in terms of 50K lines of code kind of library. So I think it needs to be built as a static library.
These comments motivate me to say that we eventually might need several subprojects for the C runtime2 backend, not just one, since one subproject should build one library, correct? I'm torn a bit, though. We can put C source files in subdirectories like packages and generate multiple libraries from these multiple directories using the sbt C compiler plugin without any difficulty too. (I know the topic of whether to compile C source files in the sbt build and distribute C-code libraries with daffodil warrants its own discussion on the dev list too.)
If we keep going without any further changes, then after two more pull requests, we would end up with:
- containers
- daf-backend-c-generator
- daf-backend-scala
- daf-cli
- daf-io
- daf-lib
- daf-macro-lib
- daf-propgen
- daf-sapi
- daf-schema-compiler
- daf-tdml-lib
- daf-tdml-processor
- daf-test
- daf-test-ibm
- daf-udf
- project
- test-stdLayout
- tutorials
-----Original Message-----
From: Steve Lawrence <sl...@apache.org>
Sent: Monday, October 5, 2020 12:00 PM
To: dev@daffodil.apache.org
Subject: EXT: Re: Subproject names proposed for discussion
I'm fine with just removing it completely. As I said, I don't think the reasons I listed are strong reasons for keeping a prefix.
On 10/5/20 11:36 AM, Beckerle, Mike wrote:
> A shorter prefix like just "daf-" would bother me less.
>
> It literally becomes screen-real-estate using IDEs because the project trees extend from the left of the screen, and they just go too far right before you get to the actual files contained.
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Monday, October 5, 2020 11:25 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: Subproject names proposed for discussion
>
> Your suggestion seems reasonable to me. Adds a few more top level
> directories by splitting lib and schema-compiler which I'm not a huge
> fan of, but it is logical and reduces the size the lib jars.
>
> The one thing I like about the "daffodil-" prefix is it makes the code
> directories stand out a bit more. For example, the containers,
> project, target, tutorials, etc. directories or more about
> infrastructure. It makes it eaiser to ignore those when scanning
> through directories (of which this change would add quite a few more).
> I think this is also pretty common among java projects, but I don't
> think they all do it. But I'm not sure if either of those are strong
> reasons to keep it. We have had problems with long paths in the past
> on windows, so removing the prefix could help with that...
>
>
> On 10/5/20 11:11 AM, Beckerle, Mike wrote:
>> Some reponses inline below
>>
>> ________________________________
>> From: Steve Lawrence <sl...@apache.org>
>> Sent: Monday, October 5, 2020 10:48 AM
>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>> Subject: Re: Subproject names proposed for discussion
>>
>> A handful of unrelated thoughts, maybe overthinking things and I
>> don't feel strongly about anything below, but renaming is always pain
>> so it'd be nice to ensure we have something future proof.
>>
>> 1) Is there any benefit organizationally to having all backends being
>> in the same directory?
>>
>> I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
>>
>> 2) From a sorting perspective, it'd be nice if the scala projects
>> were together, so having it be scala-parser and scala-unparser rather
>> than parser-scala and unparser-scala has advantages.
>> Agree.
>>
>> 3) Maybe the scala parser/unparser should be considered the same "scala"
>> runtime, and so parser/unparser should be subdirectories of a
>> "daffodil-backend-scala" subdirectory?
>>
>> There is no reason to keep the scala parser and unparser separate.
>>
>> That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
>>
>> 4) Is there even a benefit to separating parser/unparser into
>> separatejars? There's so much shared logic between the two, and
>> there's even a bunch of unparsing stuff in the parser jar. Should we just combine them under the same backend?
>>
>> Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
>>
>> Taking all of the above into account, perhaps something like this:
>>
>> ...
>> |-- daffodil-backends
>>
>> I would not add the backends parent.
>>
>> daffodil-backend-scala
>>
>> | |-- daffodil-scala
>> | | `-- src
>> | `-- daffodil-generator-c
>>
>> I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
>>
>> While we're at it, can't we drop the "daffodil-" prefix on all these?
>>
>> | `-- src
>> |-- daffodil-lib
>> | `-- src
>> |-- daffodil-schema-compiler
>>
>> This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
>>
>> This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
>>
>> | `-- src
>> ...
>>
>> 5) Is there something better than "backend" for describing these. I
>> can't think of anything. Does the DFDL spec have a concept of this?
>>
>> There is no concept of this in the DFDL spec to borrow from.
>>
>> 6) Are there any benefits to using "codenames". My thinking is maybe
>> someday there could be multiple "scala" backends with different
>> goals/extensions, and so "daffodil-scala" is too generic. Codenames
>> would be more like what we have today, except real code names might
>> be easier to remember than "runtime1" and "runtime2". Disadvantage is
>> there's less discoverability, but a README could be added with short
>> descriptions about what the backends try to accomplish. Not sure I
>> like this, but thought I'd throw it out there.
>>
>> Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
>>
>> Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
>>
>> My summary:
>>
>> daffodil
>> backend-scala-schema-compiler
>> core-schema-compiler
>> backend-c-generator-schema-compiler
>> lib
>> backend-scala-parser-lib
>> backend-scala-unparser-lib
>> backend-c-lib
>> tdml-lib
>> tdml-processor
>> .....
>>
>>
>>
>>
>>
>>
>> On 10/5/20 10:23 AM, Beckerle, Mike wrote:
>>> +1 from me.
>>>
>>> ________________________________
>>> From: Interrante, John A (GE Research, US)
>>> <in...@research.ge.com>
>>> Sent: Monday, October 5, 2020 9:28 AM
>>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>>> Subject: Subproject names proposed for discussion
>>>
>>> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>>>
>>> What do devs think about us renaming some subprojects like this?
>>>
>>> rename daffodil-core to daffodil-schema-compiler
>>> leave daffodil-lib alone
>>> rename daffodil-runtime1 to daffodil-backend-parser-scala
>>> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
>>> rename daffodil-runtime2 to daffodil-backend-generator-c
>>>
>>>
>>
>>
>>
>
>
Re: Subproject names proposed for discussion
Posted by Steve Lawrence <sl...@apache.org>.
I'm fine with just removing it completely. As I said, I don't think the
reasons I listed are strong reasons for keeping a prefix.
On 10/5/20 11:36 AM, Beckerle, Mike wrote:
> A shorter prefix like just "daf-" would bother me less.
>
> It literally becomes screen-real-estate using IDEs because the project trees extend from the left of the screen, and they just go too far right before you get to the actual files contained.
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Monday, October 5, 2020 11:25 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: Subproject names proposed for discussion
>
> Your suggestion seems reasonable to me. Adds a few more top level
> directories by splitting lib and schema-compiler which I'm not a huge
> fan of, but it is logical and reduces the size the lib jars.
>
> The one thing I like about the "daffodil-" prefix is it makes the code
> directories stand out a bit more. For example, the containers, project,
> target, tutorials, etc. directories or more about infrastructure. It
> makes it eaiser to ignore those when scanning through directories (of
> which this change would add quite a few more). I think this is also
> pretty common among java projects, but I don't think they all do it. But
> I'm not sure if either of those are strong reasons to keep it. We have
> had problems with long paths in the past on windows, so removing the
> prefix could help with that...
>
>
> On 10/5/20 11:11 AM, Beckerle, Mike wrote:
>> Some reponses inline below
>>
>> ________________________________
>> From: Steve Lawrence <sl...@apache.org>
>> Sent: Monday, October 5, 2020 10:48 AM
>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>> Subject: Re: Subproject names proposed for discussion
>>
>> A handful of unrelated thoughts, maybe overthinking things and I don't
>> feel strongly about anything below, but renaming is always pain so it'd
>> be nice to ensure we have something future proof.
>>
>> 1) Is there any benefit organizationally to having all backends being in
>> the same directory?
>>
>> I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
>>
>> 2) From a sorting perspective, it'd be nice if the scala projects were
>> together, so having it be scala-parser and scala-unparser rather than
>> parser-scala and unparser-scala has advantages.
>> Agree.
>>
>> 3) Maybe the scala parser/unparser should be considered the same "scala"
>> runtime, and so parser/unparser should be subdirectories of a
>> "daffodil-backend-scala" subdirectory?
>>
>> There is no reason to keep the scala parser and unparser separate.
>>
>> That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
>>
>> 4) Is there even a benefit to separating parser/unparser into separatejars? There's so much shared logic between the two, and there's even a
>> bunch of unparsing stuff in the parser jar. Should we just combine them
>> under the same backend?
>>
>> Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
>>
>> Taking all of the above into account, perhaps something like this:
>>
>> ...
>> |-- daffodil-backends
>>
>> I would not add the backends parent.
>>
>> daffodil-backend-scala
>>
>> | |-- daffodil-scala
>> | | `-- src
>> | `-- daffodil-generator-c
>>
>> I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
>>
>> While we're at it, can't we drop the "daffodil-" prefix on all these?
>>
>> | `-- src
>> |-- daffodil-lib
>> | `-- src
>> |-- daffodil-schema-compiler
>>
>> This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
>>
>> This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
>>
>> | `-- src
>> ...
>>
>> 5) Is there something better than "backend" for describing these. I
>> can't think of anything. Does the DFDL spec have a concept of this?
>>
>> There is no concept of this in the DFDL spec to borrow from.
>>
>> 6) Are there any benefits to using "codenames". My thinking is maybe
>> someday there could be multiple "scala" backends with different
>> goals/extensions, and so "daffodil-scala" is too generic. Codenames
>> would be more like what we have today, except real code names might be
>> easier to remember than "runtime1" and "runtime2". Disadvantage is
>> there's less discoverability, but a README could be added with short
>> descriptions about what the backends try to accomplish. Not sure I like
>> this, but thought I'd throw it out there.
>>
>> Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
>>
>> Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
>>
>> My summary:
>>
>> daffodil
>> backend-scala-schema-compiler
>> core-schema-compiler
>> backend-c-generator-schema-compiler
>> lib
>> backend-scala-parser-lib
>> backend-scala-unparser-lib
>> backend-c-lib
>> tdml-lib
>> tdml-processor
>> .....
>>
>>
>>
>>
>>
>>
>> On 10/5/20 10:23 AM, Beckerle, Mike wrote:
>>> +1 from me.
>>>
>>> ________________________________
>>> From: Interrante, John A (GE Research, US) <in...@research.ge.com>
>>> Sent: Monday, October 5, 2020 9:28 AM
>>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>>> Subject: Subproject names proposed for discussion
>>>
>>> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>>>
>>> What do devs think about us renaming some subprojects like this?
>>>
>>> rename daffodil-core to daffodil-schema-compiler
>>> leave daffodil-lib alone
>>> rename daffodil-runtime1 to daffodil-backend-parser-scala
>>> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
>>> rename daffodil-runtime2 to daffodil-backend-generator-c
>>>
>>>
>>
>>
>>
>
>
Re: Subproject names proposed for discussion
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
A shorter prefix like just "daf-" would bother me less.
It literally becomes screen-real-estate using IDEs because the project trees extend from the left of the screen, and they just go too far right before you get to the actual files contained.
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Monday, October 5, 2020 11:25 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: Subproject names proposed for discussion
Your suggestion seems reasonable to me. Adds a few more top level
directories by splitting lib and schema-compiler which I'm not a huge
fan of, but it is logical and reduces the size the lib jars.
The one thing I like about the "daffodil-" prefix is it makes the code
directories stand out a bit more. For example, the containers, project,
target, tutorials, etc. directories or more about infrastructure. It
makes it eaiser to ignore those when scanning through directories (of
which this change would add quite a few more). I think this is also
pretty common among java projects, but I don't think they all do it. But
I'm not sure if either of those are strong reasons to keep it. We have
had problems with long paths in the past on windows, so removing the
prefix could help with that...
On 10/5/20 11:11 AM, Beckerle, Mike wrote:
> Some reponses inline below
>
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Monday, October 5, 2020 10:48 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: Subproject names proposed for discussion
>
> A handful of unrelated thoughts, maybe overthinking things and I don't
> feel strongly about anything below, but renaming is always pain so it'd
> be nice to ensure we have something future proof.
>
> 1) Is there any benefit organizationally to having all backends being in
> the same directory?
>
> I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
>
> 2) From a sorting perspective, it'd be nice if the scala projects were
> together, so having it be scala-parser and scala-unparser rather than
> parser-scala and unparser-scala has advantages.
> Agree.
>
> 3) Maybe the scala parser/unparser should be considered the same "scala"
> runtime, and so parser/unparser should be subdirectories of a
> "daffodil-backend-scala" subdirectory?
>
> There is no reason to keep the scala parser and unparser separate.
>
> That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
>
> 4) Is there even a benefit to separating parser/unparser into separatejars? There's so much shared logic between the two, and there's even a
> bunch of unparsing stuff in the parser jar. Should we just combine them
> under the same backend?
>
> Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
>
> Taking all of the above into account, perhaps something like this:
>
> ...
> |-- daffodil-backends
>
> I would not add the backends parent.
>
> daffodil-backend-scala
>
> | |-- daffodil-scala
> | | `-- src
> | `-- daffodil-generator-c
>
> I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
>
> While we're at it, can't we drop the "daffodil-" prefix on all these?
>
> | `-- src
> |-- daffodil-lib
> | `-- src
> |-- daffodil-schema-compiler
>
> This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
>
> This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
>
> | `-- src
> ...
>
> 5) Is there something better than "backend" for describing these. I
> can't think of anything. Does the DFDL spec have a concept of this?
>
> There is no concept of this in the DFDL spec to borrow from.
>
> 6) Are there any benefits to using "codenames". My thinking is maybe
> someday there could be multiple "scala" backends with different
> goals/extensions, and so "daffodil-scala" is too generic. Codenames
> would be more like what we have today, except real code names might be
> easier to remember than "runtime1" and "runtime2". Disadvantage is
> there's less discoverability, but a README could be added with short
> descriptions about what the backends try to accomplish. Not sure I like
> this, but thought I'd throw it out there.
>
> Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
>
> Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
>
> My summary:
>
> daffodil
> backend-scala-schema-compiler
> core-schema-compiler
> backend-c-generator-schema-compiler
> lib
> backend-scala-parser-lib
> backend-scala-unparser-lib
> backend-c-lib
> tdml-lib
> tdml-processor
> .....
>
>
>
>
>
>
> On 10/5/20 10:23 AM, Beckerle, Mike wrote:
>> +1 from me.
>>
>> ________________________________
>> From: Interrante, John A (GE Research, US) <in...@research.ge.com>
>> Sent: Monday, October 5, 2020 9:28 AM
>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>> Subject: Subproject names proposed for discussion
>>
>> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>>
>> What do devs think about us renaming some subprojects like this?
>>
>> rename daffodil-core to daffodil-schema-compiler
>> leave daffodil-lib alone
>> rename daffodil-runtime1 to daffodil-backend-parser-scala
>> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
>> rename daffodil-runtime2 to daffodil-backend-generator-c
>>
>>
>
>
>
Re: Subproject names proposed for discussion
Posted by Steve Lawrence <sl...@apache.org>.
Your suggestion seems reasonable to me. Adds a few more top level
directories by splitting lib and schema-compiler which I'm not a huge
fan of, but it is logical and reduces the size the lib jars.
The one thing I like about the "daffodil-" prefix is it makes the code
directories stand out a bit more. For example, the containers, project,
target, tutorials, etc. directories or more about infrastructure. It
makes it eaiser to ignore those when scanning through directories (of
which this change would add quite a few more). I think this is also
pretty common among java projects, but I don't think they all do it. But
I'm not sure if either of those are strong reasons to keep it. We have
had problems with long paths in the past on windows, so removing the
prefix could help with that...
On 10/5/20 11:11 AM, Beckerle, Mike wrote:
> Some reponses inline below
>
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Monday, October 5, 2020 10:48 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: Subproject names proposed for discussion
>
> A handful of unrelated thoughts, maybe overthinking things and I don't
> feel strongly about anything below, but renaming is always pain so it'd
> be nice to ensure we have something future proof.
>
> 1) Is there any benefit organizationally to having all backends being in
> the same directory?
>
> I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
>
> 2) From a sorting perspective, it'd be nice if the scala projects were
> together, so having it be scala-parser and scala-unparser rather than
> parser-scala and unparser-scala has advantages.
> Agree.
>
> 3) Maybe the scala parser/unparser should be considered the same "scala"
> runtime, and so parser/unparser should be subdirectories of a
> "daffodil-backend-scala" subdirectory?
>
> There is no reason to keep the scala parser and unparser separate.
>
> That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
>
> 4) Is there even a benefit to separating parser/unparser into separatejars? There's so much shared logic between the two, and there's even a
> bunch of unparsing stuff in the parser jar. Should we just combine them
> under the same backend?
>
> Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
>
> Taking all of the above into account, perhaps something like this:
>
> ...
> |-- daffodil-backends
>
> I would not add the backends parent.
>
> daffodil-backend-scala
>
> | |-- daffodil-scala
> | | `-- src
> | `-- daffodil-generator-c
>
> I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
>
> While we're at it, can't we drop the "daffodil-" prefix on all these?
>
> | `-- src
> |-- daffodil-lib
> | `-- src
> |-- daffodil-schema-compiler
>
> This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
>
> This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
>
> | `-- src
> ...
>
> 5) Is there something better than "backend" for describing these. I
> can't think of anything. Does the DFDL spec have a concept of this?
>
> There is no concept of this in the DFDL spec to borrow from.
>
> 6) Are there any benefits to using "codenames". My thinking is maybe
> someday there could be multiple "scala" backends with different
> goals/extensions, and so "daffodil-scala" is too generic. Codenames
> would be more like what we have today, except real code names might be
> easier to remember than "runtime1" and "runtime2". Disadvantage is
> there's less discoverability, but a README could be added with short
> descriptions about what the backends try to accomplish. Not sure I like
> this, but thought I'd throw it out there.
>
> Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
>
> Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
>
> My summary:
>
> daffodil
> backend-scala-schema-compiler
> core-schema-compiler
> backend-c-generator-schema-compiler
> lib
> backend-scala-parser-lib
> backend-scala-unparser-lib
> backend-c-lib
> tdml-lib
> tdml-processor
> .....
>
>
>
>
>
>
> On 10/5/20 10:23 AM, Beckerle, Mike wrote:
>> +1 from me.
>>
>> ________________________________
>> From: Interrante, John A (GE Research, US) <in...@research.ge.com>
>> Sent: Monday, October 5, 2020 9:28 AM
>> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
>> Subject: Subproject names proposed for discussion
>>
>> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>>
>> What do devs think about us renaming some subprojects like this?
>>
>> rename daffodil-core to daffodil-schema-compiler
>> leave daffodil-lib alone
>> rename daffodil-runtime1 to daffodil-backend-parser-scala
>> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
>> rename daffodil-runtime2 to daffodil-backend-generator-c
>>
>>
>
>
>
Re: Subproject names proposed for discussion
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
Some reponses inline below
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Monday, October 5, 2020 10:48 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: Subproject names proposed for discussion
A handful of unrelated thoughts, maybe overthinking things and I don't
feel strongly about anything below, but renaming is always pain so it'd
be nice to ensure we have something future proof.
1) Is there any benefit organizationally to having all backends being in
the same directory?
I think the opposite. I think they should be separate and top level in fact, so that we're not tied to say, sbt build system as we are now.
2) From a sorting perspective, it'd be nice if the scala projects were
together, so having it be scala-parser and scala-unparser rather than
parser-scala and unparser-scala has advantages.
Agree.
3) Maybe the scala parser/unparser should be considered the same "scala"
runtime, and so parser/unparser should be subdirectories of a
"daffodil-backend-scala" subdirectory?
There is no reason to keep the scala parser and unparser separate.
That was done at one time because compilation time was so long that we wanted to be able to close the unparser project (on eclipse) and not be recompiling it if just doing work involving the parser. This is less an issue now.
4) Is there even a benefit to separating parser/unparser into separatejars? There's so much shared logic between the two, and there's even a
bunch of unparsing stuff in the parser jar. Should we just combine them
under the same backend?
Yes, but one step at a time. First I would do the renaming. I'd do the merging of parser/unparser into one lib separately.
Taking all of the above into account, perhaps something like this:
...
|-- daffodil-backends
I would not add the backends parent.
daffodil-backend-scala
| |-- daffodil-scala
| | `-- src
| `-- daffodil-generator-c
I would call the above daffodil-c-generator to be consistent. It is important that this is identified as a generator.
While we're at it, can't we drop the "daffodil-" prefix on all these?
| `-- src
|-- daffodil-lib
| `-- src
|-- daffodil-schema-compiler
This is problematic because each back-end is going to have its part of the schema compiler that converts grammar objects into back-end primitives code by generation of code that calls libraries.
This should be distinguished from the 100% static code part of the back-end which is statically written and testable independently.
| `-- src
...
5) Is there something better than "backend" for describing these. I
can't think of anything. Does the DFDL spec have a concept of this?
There is no concept of this in the DFDL spec to borrow from.
6) Are there any benefits to using "codenames". My thinking is maybe
someday there could be multiple "scala" backends with different
goals/extensions, and so "daffodil-scala" is too generic. Codenames
would be more like what we have today, except real code names might be
easier to remember than "runtime1" and "runtime2". Disadvantage is
there's less discoverability, but a README could be added with short
descriptions about what the backends try to accomplish. Not sure I like
this, but thought I'd throw it out there.
Naming backends after "famous runtimes" as in marathons and other road races, .... except those just end up being named after places or people. Might as well use the name or home-town of the person who wrote the first line of code in whatever backend it is.
Personally, I find separating them into backend scala, vs. backend c-generator works fine, and if it splits below that, numbering will be sufficient.
My summary:
daffodil
backend-scala-schema-compiler
core-schema-compiler
backend-c-generator-schema-compiler
lib
backend-scala-parser-lib
backend-scala-unparser-lib
backend-c-lib
tdml-lib
tdml-processor
.....
On 10/5/20 10:23 AM, Beckerle, Mike wrote:
> +1 from me.
>
> ________________________________
> From: Interrante, John A (GE Research, US) <in...@research.ge.com>
> Sent: Monday, October 5, 2020 9:28 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Subproject names proposed for discussion
>
> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>
> What do devs think about us renaming some subprojects like this?
>
> rename daffodil-core to daffodil-schema-compiler
> leave daffodil-lib alone
> rename daffodil-runtime1 to daffodil-backend-parser-scala
> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
> rename daffodil-runtime2 to daffodil-backend-generator-c
>
>
Re: Subproject names proposed for discussion
Posted by Steve Lawrence <sl...@apache.org>.
A handful of unrelated thoughts, maybe overthinking things and I don't
feel strongly about anything below, but renaming is always pain so it'd
be nice to ensure we have something future proof.
1) Is there any benefit organizationally to having all backends being in
the same directory?
2) From a sorting perspective, it'd be nice if the scala projects were
together, so having it be scala-parser and scala-unparser rather than
parser-scala and unparser-scala has advantages.
3) Maybe the scala parser/unparser should be considered the same "scala"
runtime, and so parser/unparser should be subdirectories of a
"daffodil-backend-scala" subdirectory?
4) Is there even a benefit to separating parser/unparser into separate
jars? There's so much shared logic between the two, and there's even a
bunch of unparsing stuff in the parser jar. Should we just combine them
under the same backend?
Taking all of the above into account, perhaps something like this:
...
|-- daffodil-backends
| |-- daffodil-scala
| | `-- src
| `-- daffodil-generator-c
| `-- src
|-- daffodil-lib
| `-- src
|-- daffodil-schema-compiler
| `-- src
...
5) Is there something better than "backend" for describing these. I
can't think of anything. Does the DFDL spec have a concept of this?
6) Are there any benefits to using "codenames". My thinking is maybe
someday there could be multiple "scala" backends with different
goals/extensions, and so "daffodil-scala" is too generic. Codenames
would be more like what we have today, except real code names might be
easier to remember than "runtime1" and "runtime2". Disadvantage is
there's less discoverability, but a README could be added with short
descriptions about what the backends try to accomplish. Not sure I like
this, but thought I'd throw it out there.
On 10/5/20 10:23 AM, Beckerle, Mike wrote:
> +1 from me.
>
> ________________________________
> From: Interrante, John A (GE Research, US) <in...@research.ge.com>
> Sent: Monday, October 5, 2020 9:28 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Subproject names proposed for discussion
>
> Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
>
> What do devs think about us renaming some subprojects like this?
>
> rename daffodil-core to daffodil-schema-compiler
> leave daffodil-lib alone
> rename daffodil-runtime1 to daffodil-backend-parser-scala
> rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
> rename daffodil-runtime2 to daffodil-backend-generator-c
>
>
Re: Subproject names proposed for discussion
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
+1 from me.
________________________________
From: Interrante, John A (GE Research, US) <in...@research.ge.com>
Sent: Monday, October 5, 2020 9:28 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Subproject names proposed for discussion
Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
What do devs think about us renaming some subprojects like this?
rename daffodil-core to daffodil-schema-compiler
leave daffodil-lib alone
rename daffodil-runtime1 to daffodil-backend-parser-scala
rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
rename daffodil-runtime2 to daffodil-backend-generator-c
Re: Subproject names proposed for discussion
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
Compilers are commonly divided into "front-end" - concerned with the language itself and its syntax, and "back end" - concerned with code-generation.
Sometimes back-ends are language independent as in the same compiler back-end can be shared using front-ends for different languages (C and Pascal for example).
The term "middle end" also comes up to mean the part that is neither front-end nor back-end, specific. In our case there are numerous optimizations we do inside Daffodil that are back-end independent. I suppose they're still DFDL-specific, but as they're not specifically concerned with the dialect/syntax of the DFDL language we distinguish them somewhat. These operate on the same grammar objects as the backends, but do their work in a backend-independent manner.
________________________________
From: Steinberger, Rick <rs...@owlcyberdefense.com>
Sent: Monday, October 5, 2020 10:46 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: RE: Subproject names proposed for discussion
What does the notation 'backend' refer to?
-----Original Message-----
From: Interrante, John A (GE Research, US) <in...@research.ge.com>
Sent: Monday, October 5, 2020 9:29 AM
To: dev@daffodil.apache.org
Subject: Subproject names proposed for discussion
Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
What do devs think about us renaming some subprojects like this?
rename daffodil-core to daffodil-schema-compiler
leave daffodil-lib alone
rename daffodil-runtime1 to daffodil-backend-parser-scala
rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
rename daffodil-runtime2 to daffodil-backend-generator-c
RE: Subproject names proposed for discussion
Posted by "Steinberger, Rick" <rs...@owlcyberdefense.com>.
What does the notation 'backend' refer to?
-----Original Message-----
From: Interrante, John A (GE Research, US) <in...@research.ge.com>
Sent: Monday, October 5, 2020 9:29 AM
To: dev@daffodil.apache.org
Subject: Subproject names proposed for discussion
Steve Lawrence and I would like to bring a topic to the dev list for discussion since not everyone is paying attention to the review of my runtime2 push request. Steve suggested, and I agree, that renaming some of the Daffodil subprojects might make their meanings more obvious to newcomer devs. If we do rename some subprojects after discussing it on this list, we will do it immediately in its own pull request since mixing changes with renames makes it difficult to see which changes are just renames instead of actual changes.
What do devs think about us renaming some subprojects like this?
rename daffodil-core to daffodil-schema-compiler
leave daffodil-lib alone
rename daffodil-runtime1 to daffodil-backend-parser-scala
rename daffodil-runtime1-unparser to daffodil-backend-unparser-scala
rename daffodil-runtime2 to daffodil-backend-generator-c