You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by John Wass <jw...@gmail.com> on 2021/01/08 21:26:45 UTC
rfc: sbt-tdml
https://github.com/jw3/sbt-tdml
I needed to get smarter on SBT plugins for a task and tinkered with
something in Daffodil to do it.
The goal here was to remove the TDML test boilerplate. I threw this at the
daffodil-test project with surpsisingly sane results on the first try. See
the bottom of the readme.
The learning process is pretty much complete, so not sure where this goes,
but wanted see if there are any comments.
Re: rfc: sbt-tdml
Posted by John Wass <jw...@gmail.com>.
Thanks everyone for the feedback. Appreciate it.
One thing to note for now, that ties into the most comments, is that an IDE
plugin was envisioned to provide proper integration with developer
workflows. The sbt plugin had to come first to preserve the headless test
functionality for CI, etc.
I'll have to take another read over all the feedback to respond in detail,
and while doing that I'll create some issues on the repo to capture the
actionable ideas.
On Mon, Jan 11, 2021 at 12:58 PM Beckerle, Mike <
mbeckerle@owlcyberdefense.com> wrote:
> Can we split this into
>
> 1) test code boilerplate - I have big issues with removing this
>
> 2) capturing tests that are expected to fail due to issues in TDML vs. in
> the scala test driver code. - I'm on board with this.
>
> With respect to (1) the test code boilerplate....
>
> The ability to start from a unit test one-liner in the IDE and debug a
> test is not a nice-to-have. It is 100% the way I do all my work on
> Daffodil. Furthermore, it is the way I *want* to work on Daffodil. It keeps
> the emphasis on test-driven development, etc.
>
> Furthermore, I depend on running large bulk test runs, within the IDE, and
> seeing the one-by-one little check boxes (or capture of the individual test
> error and console output) for each individual test. So it's not just can I
> run them easily. It's can I run 1000+ of them, and then use the IDE feature
> to click through the 15 that failed one by one.
>
> It is completely worth keeping all the test boilerplate in order to have
> this. This is a big part of where I get personal "joy" from working on
> Daffodil. You take this away, and it's no longer anywhere near as much fun
> to work on.
>
> Mess with peoples' fun at your peril 🙂
>
> If you can come up with a way for me to get exactly that behavior - where
> a TDML file of tests behaves exactly like a file of JUnit tests in the IDE,
> then I could deal with that.
>
> E.g., a pre-processor step like 'sbt setupTests' that scrapes all the TDML
> files and outputs all the scala test JUnit files automatically would be one
> way to maybe do that. The test driver files could then not be checked-in I
> think.
>
> But the goal here should not be to "accommodate IDE users" somehow. The
> goal should be perfect seamless execution that follows ordinary IDE idioms
> in the ordinary way, like developing any other library software with unit
> tests in it.
>
> A few one-time setup hiccups are tolerable. Once things are setup, then it
> has to be "IDE-ordinary/JUnit-ordinary" for the
> test-edit-debug-compile-repeat cycle. Without that I believe we exclude
> developers who want powerful IDE support. Like me.
>
> Both Eclipse and IntelliJ currently achieve this in my view. IntelliJ
> simply has better Scala-language support, so we seem to prefer it. But
> Eclipse works fine, for the degree of scala support it offers.
>
>
> For issue (2): capturing tests that are supposed to fail in the TDML not
> in the scala test driver code.
>
> This sounds really really good.
>
> I think moving all parameters currently expressed in scala test driver
> files onto the TDML as additional attributes on the testSuite element or on
> the parserTestCase and unparserTestCase elements is great.
>
> Some scala-test-suite parameters specify how the TDML file XML is to be
> read, such as with or without validation on. Capturing that flag on the
> TDML testSuite element requires reading every TDML file twice - once
> without validation, then once again to validate it once you rule out the
> use of the no-validate flag. This latter would be the normal case, not the
> exceptional case. But the overhead here might be trivial in the grand
> scheme of things.
>
> I think tests that are marked as expected to fail (hopefully with a JIRA
> ticket marking that), I think they should be run anyway for that
> implementation, and the fact that they fail verified. This insures they are
> run periodically and so will capture and characterize failing behavior.
>
> Some bugs fix themselves (as part of library upgrades, or as part of
> fixing other bugs), and this would identify those for us. When I comb
> through the bugs periodically there's often the notion "I think that may
> already be fixed." It would be great for the test suite to tell us.
>
> They do need to be somehow identified so when they fail as expected,
> that's not counted as a passing test, but as a known failing test that is
> still expected to fail. JUnit can't distinguish this I think, but a test
> script could, or a message. Though we have thousands of tests, dozens of
> which might be these "expected to fail" variety, and we don't want too much
> output noise. One line per saying "Expected Failure DAFFODIL-9876" would
> perhaps be ok.
>
> I would suggest the properties not combine implementation and failure
> documentation into one attribute, but rather since such a test can be a
> negative or positive test case, we need a place to put strings that are
> expected to find in the failure. We'd have to invent a new child element to
> express those like:
>
> <tdml:bug implementation="daffodil" info="DAFFODIL-9876">
> <tdml:expect>some string here</tdml:expect>
> </tdml:bug>
>
> Shorter without the expect info:
>
> <tdml:bug info="DAFFODIL-9876"/>
>
> Notice here I inferred the implementation attribute from the info
> attribute (first token).
>
> We should allow multiple such bug child elements to report about different
> implementations, or if there are multiple tickets to highlight.
>
>
>
>
>
>
> ________________________________
> From: Steve Lawrence <sl...@apache.org>
> Sent: Monday, January 11, 2021 8:58 AM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: rfc: sbt-tdml
>
> This is great! Additional comments inline:
>
> On 1/9/21 12:20 PM, Interrante, John A (GE Research, US) wrote:
> > I like the idea of removing the Scala TDML test boilerplate as much as
> possible. Does your sbt plugin handle the use case where some Scala test
> files define customized or multiple TDML runners in the same file? For
> example, see
> daffodil-test/src/test/scala/org/apache/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala
> where two runners are defined for the same TDML file (expressions.tdml, one
> standard runner and one customized runner with validateDFDLSchemas = false)
> and several other customized runners are defined (expressions2.tdml with
> compileAllTopLevel = true). Even if your sbt plugin can define only a
> standard runner for each TDML file, it would be nice to remove most of the
> Scala TDML test code except for the code using customized runners. I'm
> sure there will be some edge cases where the sbt plugin must not run a TDML
> file with a standard runner because it works only with a customized runner,
> but the ability to specify configuration files in TDML or tell the sbt
> plugin to skip some TDML files may help somewhat.
>
> I suspect if we ever need the same TDML file to be run with different
> TDML Runner configurations, then we would just need two different TDML
> files. It's maybe a bit of a headache, but certainly much cleaner, and
> is a pretty uncommon occurrence.
>
> Having this plugin read the TDML file for attributes defined on the
> testSuite would be a good way to configure the runner.
>
> > We need to make sure we preserve the ability to skip some TDML test
> cases with a string explaining why they are failing (usually a JIRA bug
> number). If we comment out failing test cases directly in the TDML file
> then we take the risk that these tests will bitrot faster once the TDML
> processor no longer parses them. I looked at
> daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd and I
> think we could modify the TDML schema to support skipping failing test
> cases. Both parserTestCase and unparserTestCase use the same
> attributeGroup (testCaseAttribs). We could add a new optional string-type
> attribute "reasonForSkipping" to testCaseAttribs and modify the sbt plugin
> or the TDML processor to automatically skip such test cases (perhaps
> printing their names and/or the reasons for skipping them, but we don't
> want to make the test output too big).
>
> Agreed, parser/unparserTestCases definitely need a way to specify that
> this test is expected to fail and should be skipped.
>
> We sortof already have this capability with the "implementations"
> attribute. If that attribute doesn't contain "daffodil" that it is
> skipped when testing Daffodil. So you could do something like
> implementations="" or implementations="ibm" (if a test only works in IBM).
>
> Unfortunately, this doesn't required that bug information be included,
> which is pretty important. It would maybe be nice if a test would only
> be skipped if bug information were included.
>
> I wonder if maybe we need to rethink the "implementations" attribute? We
> might consider instead something like
> <implementation>-failure="details". So for example, a test that fails on
> Daffodil might include
>
> daffodil-failure="DAFFODIL-1234"
>
> This way, in order to have the TDML runner skip a test, you MUST include
> this attribute which can provide details. This detail can then be
> included in the output when a test is skipped.
>
> Additionally it makes it easy to grep for failed tests and related bug
> numbers.
>
> > Nice idea, it will be good to see how far it can reduce the size of the
> Scala test boilerplate code! Oh, it occurs to me that sometimes it's
> useful to debug a TDML test case directly from the IDE by running a Scala
> test method under the debugger. How often do developers need to debug
> existing or new TDML test cases, which would require them to write
> throw-away Scala test methods by hand whenever necessary? Maybe we can
> code up some sort of universal Scala entry point we can run from the IDE
> which can run any desired TDML test case by editing (at most) one or two
> lines of code.
>
> Yeah, I think this is probably the the most important functionality to
> have, and probably isn't viable if IDE support isn't there. How does
> this integrate with IDEs? Maybe it "just works" for IDEs that support
> sbt? What about if an IDE supports BSP since our current version of SBT
> supports BSP now? But what about IDE's like eclipse. I'm not sure if any
> devs use Eclipse anymore since IntelliJ seems much better, but maybe
> something to consider? Can we claim we only support IDE's that support
> SBT/BSP?
>
> Also, SBT has the testOnly command to run a single testsuite/test. Is
> that supported with this? For example, currently we can run
>
> sbt testOnly org.apache.daffodil....TestSuite -- --tests=test_regex
>
> So it takes a test suite and an optional regex of tests in that suite to
> run. Having this similar functionality would be pretty critical.
> Presumably, it would be very similar to the above, but maybe just a path
> to the TDML file instead, e.g.
>
> sbt testOnly org/apache/daffodil/.../TestSuite.tdml -- --tests=test_regex
>
> Using the path to the tdml file as the test suite name avoids issues
> with duplicate test suite names in TDML files. Also makes for easily
> copy/pasting a path when you want to test a tdml file.
>
>
> Some other thoughts:
>
> Does anything output when a test fails? Or is that still to be implemented?
>
>
> I'm also curious how this works regarding Daffodil versions. Presumably
> this depends on a certain Daffodil version? What happens if a repo has a
> different Daffodil dependency? Does this override that? Or does this
> just expect that Daffodil is already a dependency and uses that version?
> And hopefully our TDMLRUnner API is stable enough that things just work?
> We definitely have cases where we need to run tests with different
> Daffodil versions, it would be nice if we didn't have to also change the
> plugin version to match.
>
>
> We also need to consider how this gets published. Do you plan to
> maintain and publish this separately as an outside plugin that Daffodil
> just depends on an uses? Or do you plant to have it merged into a
> Daffodil repo? If so, I wonder if it lives in the Daffodil repo so
> Daffodil always has the latest, and then this is published when there
> are Daffodil releases? Or maybe it's a separate Apache repo that follows
> the same Apache release process/requirements, but can be on a different
> schedule than Daffodil?
>
>
> If all the above issue can potentially be resolved, I think there's a
> good chance this could replace all of our boiler plate, which I'm in big
> favor of.
>
>
> > -----Original Message-----
> > From: John Wass <jw...@gmail.com>
> > Sent: Friday, January 8, 2021 4:27 PM
> > To: dev@daffodil.apache.org
> > Subject: EXT: rfc: sbt-tdml
> >
> > https://github.com/jw3/sbt-tdml
> > I needed to get smarter on SBT plugins for a task and tinkered with
> something in Daffodil to do it.
> >
> > The goal here was to remove the TDML test boilerplate. I threw this at
> the daffodil-test project with surpsisingly sane results on the first try.
> See the bottom of the readme.
> >
> > The learning process is pretty much complete, so not sure where this
> goes, but wanted see if there are any comments.
> >
>
>
Re: rfc: sbt-tdml
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
Can we split this into
1) test code boilerplate - I have big issues with removing this
2) capturing tests that are expected to fail due to issues in TDML vs. in the scala test driver code. - I'm on board with this.
With respect to (1) the test code boilerplate....
The ability to start from a unit test one-liner in the IDE and debug a test is not a nice-to-have. It is 100% the way I do all my work on Daffodil. Furthermore, it is the way I *want* to work on Daffodil. It keeps the emphasis on test-driven development, etc.
Furthermore, I depend on running large bulk test runs, within the IDE, and seeing the one-by-one little check boxes (or capture of the individual test error and console output) for each individual test. So it's not just can I run them easily. It's can I run 1000+ of them, and then use the IDE feature to click through the 15 that failed one by one.
It is completely worth keeping all the test boilerplate in order to have this. This is a big part of where I get personal "joy" from working on Daffodil. You take this away, and it's no longer anywhere near as much fun to work on.
Mess with peoples' fun at your peril 🙂
If you can come up with a way for me to get exactly that behavior - where a TDML file of tests behaves exactly like a file of JUnit tests in the IDE, then I could deal with that.
E.g., a pre-processor step like 'sbt setupTests' that scrapes all the TDML files and outputs all the scala test JUnit files automatically would be one way to maybe do that. The test driver files could then not be checked-in I think.
But the goal here should not be to "accommodate IDE users" somehow. The goal should be perfect seamless execution that follows ordinary IDE idioms in the ordinary way, like developing any other library software with unit tests in it.
A few one-time setup hiccups are tolerable. Once things are setup, then it has to be "IDE-ordinary/JUnit-ordinary" for the test-edit-debug-compile-repeat cycle. Without that I believe we exclude developers who want powerful IDE support. Like me.
Both Eclipse and IntelliJ currently achieve this in my view. IntelliJ simply has better Scala-language support, so we seem to prefer it. But Eclipse works fine, for the degree of scala support it offers.
For issue (2): capturing tests that are supposed to fail in the TDML not in the scala test driver code.
This sounds really really good.
I think moving all parameters currently expressed in scala test driver files onto the TDML as additional attributes on the testSuite element or on the parserTestCase and unparserTestCase elements is great.
Some scala-test-suite parameters specify how the TDML file XML is to be read, such as with or without validation on. Capturing that flag on the TDML testSuite element requires reading every TDML file twice - once without validation, then once again to validate it once you rule out the use of the no-validate flag. This latter would be the normal case, not the exceptional case. But the overhead here might be trivial in the grand scheme of things.
I think tests that are marked as expected to fail (hopefully with a JIRA ticket marking that), I think they should be run anyway for that implementation, and the fact that they fail verified. This insures they are run periodically and so will capture and characterize failing behavior.
Some bugs fix themselves (as part of library upgrades, or as part of fixing other bugs), and this would identify those for us. When I comb through the bugs periodically there's often the notion "I think that may already be fixed." It would be great for the test suite to tell us.
They do need to be somehow identified so when they fail as expected, that's not counted as a passing test, but as a known failing test that is still expected to fail. JUnit can't distinguish this I think, but a test script could, or a message. Though we have thousands of tests, dozens of which might be these "expected to fail" variety, and we don't want too much output noise. One line per saying "Expected Failure DAFFODIL-9876" would perhaps be ok.
I would suggest the properties not combine implementation and failure documentation into one attribute, but rather since such a test can be a negative or positive test case, we need a place to put strings that are expected to find in the failure. We'd have to invent a new child element to express those like:
<tdml:bug implementation="daffodil" info="DAFFODIL-9876">
<tdml:expect>some string here</tdml:expect>
</tdml:bug>
Shorter without the expect info:
<tdml:bug info="DAFFODIL-9876"/>
Notice here I inferred the implementation attribute from the info attribute (first token).
We should allow multiple such bug child elements to report about different implementations, or if there are multiple tickets to highlight.
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Monday, January 11, 2021 8:58 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: rfc: sbt-tdml
This is great! Additional comments inline:
On 1/9/21 12:20 PM, Interrante, John A (GE Research, US) wrote:
> I like the idea of removing the Scala TDML test boilerplate as much as possible. Does your sbt plugin handle the use case where some Scala test files define customized or multiple TDML runners in the same file? For example, see daffodil-test/src/test/scala/org/apache/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala where two runners are defined for the same TDML file (expressions.tdml, one standard runner and one customized runner with validateDFDLSchemas = false) and several other customized runners are defined (expressions2.tdml with compileAllTopLevel = true). Even if your sbt plugin can define only a standard runner for each TDML file, it would be nice to remove most of the Scala TDML test code except for the code using customized runners. I'm sure there will be some edge cases where the sbt plugin must not run a TDML file with a standard runner because it works only with a customized runner, but the ability to specify configuration files in TDML or tell the sbt plugin to skip some TDML files may help somewhat.
I suspect if we ever need the same TDML file to be run with different
TDML Runner configurations, then we would just need two different TDML
files. It's maybe a bit of a headache, but certainly much cleaner, and
is a pretty uncommon occurrence.
Having this plugin read the TDML file for attributes defined on the
testSuite would be a good way to configure the runner.
> We need to make sure we preserve the ability to skip some TDML test cases with a string explaining why they are failing (usually a JIRA bug number). If we comment out failing test cases directly in the TDML file then we take the risk that these tests will bitrot faster once the TDML processor no longer parses them. I looked at daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd and I think we could modify the TDML schema to support skipping failing test cases. Both parserTestCase and unparserTestCase use the same attributeGroup (testCaseAttribs). We could add a new optional string-type attribute "reasonForSkipping" to testCaseAttribs and modify the sbt plugin or the TDML processor to automatically skip such test cases (perhaps printing their names and/or the reasons for skipping them, but we don't want to make the test output too big).
Agreed, parser/unparserTestCases definitely need a way to specify that
this test is expected to fail and should be skipped.
We sortof already have this capability with the "implementations"
attribute. If that attribute doesn't contain "daffodil" that it is
skipped when testing Daffodil. So you could do something like
implementations="" or implementations="ibm" (if a test only works in IBM).
Unfortunately, this doesn't required that bug information be included,
which is pretty important. It would maybe be nice if a test would only
be skipped if bug information were included.
I wonder if maybe we need to rethink the "implementations" attribute? We
might consider instead something like
<implementation>-failure="details". So for example, a test that fails on
Daffodil might include
daffodil-failure="DAFFODIL-1234"
This way, in order to have the TDML runner skip a test, you MUST include
this attribute which can provide details. This detail can then be
included in the output when a test is skipped.
Additionally it makes it easy to grep for failed tests and related bug
numbers.
> Nice idea, it will be good to see how far it can reduce the size of the Scala test boilerplate code! Oh, it occurs to me that sometimes it's useful to debug a TDML test case directly from the IDE by running a Scala test method under the debugger. How often do developers need to debug existing or new TDML test cases, which would require them to write throw-away Scala test methods by hand whenever necessary? Maybe we can code up some sort of universal Scala entry point we can run from the IDE which can run any desired TDML test case by editing (at most) one or two lines of code.
Yeah, I think this is probably the the most important functionality to
have, and probably isn't viable if IDE support isn't there. How does
this integrate with IDEs? Maybe it "just works" for IDEs that support
sbt? What about if an IDE supports BSP since our current version of SBT
supports BSP now? But what about IDE's like eclipse. I'm not sure if any
devs use Eclipse anymore since IntelliJ seems much better, but maybe
something to consider? Can we claim we only support IDE's that support
SBT/BSP?
Also, SBT has the testOnly command to run a single testsuite/test. Is
that supported with this? For example, currently we can run
sbt testOnly org.apache.daffodil....TestSuite -- --tests=test_regex
So it takes a test suite and an optional regex of tests in that suite to
run. Having this similar functionality would be pretty critical.
Presumably, it would be very similar to the above, but maybe just a path
to the TDML file instead, e.g.
sbt testOnly org/apache/daffodil/.../TestSuite.tdml -- --tests=test_regex
Using the path to the tdml file as the test suite name avoids issues
with duplicate test suite names in TDML files. Also makes for easily
copy/pasting a path when you want to test a tdml file.
Some other thoughts:
Does anything output when a test fails? Or is that still to be implemented?
I'm also curious how this works regarding Daffodil versions. Presumably
this depends on a certain Daffodil version? What happens if a repo has a
different Daffodil dependency? Does this override that? Or does this
just expect that Daffodil is already a dependency and uses that version?
And hopefully our TDMLRUnner API is stable enough that things just work?
We definitely have cases where we need to run tests with different
Daffodil versions, it would be nice if we didn't have to also change the
plugin version to match.
We also need to consider how this gets published. Do you plan to
maintain and publish this separately as an outside plugin that Daffodil
just depends on an uses? Or do you plant to have it merged into a
Daffodil repo? If so, I wonder if it lives in the Daffodil repo so
Daffodil always has the latest, and then this is published when there
are Daffodil releases? Or maybe it's a separate Apache repo that follows
the same Apache release process/requirements, but can be on a different
schedule than Daffodil?
If all the above issue can potentially be resolved, I think there's a
good chance this could replace all of our boiler plate, which I'm in big
favor of.
> -----Original Message-----
> From: John Wass <jw...@gmail.com>
> Sent: Friday, January 8, 2021 4:27 PM
> To: dev@daffodil.apache.org
> Subject: EXT: rfc: sbt-tdml
>
> https://github.com/jw3/sbt-tdml
> I needed to get smarter on SBT plugins for a task and tinkered with something in Daffodil to do it.
>
> The goal here was to remove the TDML test boilerplate. I threw this at the daffodil-test project with surpsisingly sane results on the first try. See the bottom of the readme.
>
> The learning process is pretty much complete, so not sure where this goes, but wanted see if there are any comments.
>
Re: rfc: sbt-tdml
Posted by Steve Lawrence <sl...@apache.org>.
This is great! Additional comments inline:
On 1/9/21 12:20 PM, Interrante, John A (GE Research, US) wrote:
> I like the idea of removing the Scala TDML test boilerplate as much as possible. Does your sbt plugin handle the use case where some Scala test files define customized or multiple TDML runners in the same file? For example, see daffodil-test/src/test/scala/org/apache/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala where two runners are defined for the same TDML file (expressions.tdml, one standard runner and one customized runner with validateDFDLSchemas = false) and several other customized runners are defined (expressions2.tdml with compileAllTopLevel = true). Even if your sbt plugin can define only a standard runner for each TDML file, it would be nice to remove most of the Scala TDML test code except for the code using customized runners. I'm sure there will be some edge cases where the sbt plugin must not run a TDML file with a standard runner because it works only with a customized runner, but the ability to specify configuration files in TDML or tell the sbt plugin to skip some TDML files may help somewhat.
I suspect if we ever need the same TDML file to be run with different
TDML Runner configurations, then we would just need two different TDML
files. It's maybe a bit of a headache, but certainly much cleaner, and
is a pretty uncommon occurrence.
Having this plugin read the TDML file for attributes defined on the
testSuite would be a good way to configure the runner.
> We need to make sure we preserve the ability to skip some TDML test cases with a string explaining why they are failing (usually a JIRA bug number). If we comment out failing test cases directly in the TDML file then we take the risk that these tests will bitrot faster once the TDML processor no longer parses them. I looked at daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd and I think we could modify the TDML schema to support skipping failing test cases. Both parserTestCase and unparserTestCase use the same attributeGroup (testCaseAttribs). We could add a new optional string-type attribute "reasonForSkipping" to testCaseAttribs and modify the sbt plugin or the TDML processor to automatically skip such test cases (perhaps printing their names and/or the reasons for skipping them, but we don't want to make the test output too big).
Agreed, parser/unparserTestCases definitely need a way to specify that
this test is expected to fail and should be skipped.
We sortof already have this capability with the "implementations"
attribute. If that attribute doesn't contain "daffodil" that it is
skipped when testing Daffodil. So you could do something like
implementations="" or implementations="ibm" (if a test only works in IBM).
Unfortunately, this doesn't required that bug information be included,
which is pretty important. It would maybe be nice if a test would only
be skipped if bug information were included.
I wonder if maybe we need to rethink the "implementations" attribute? We
might consider instead something like
<implementation>-failure="details". So for example, a test that fails on
Daffodil might include
daffodil-failure="DAFFODIL-1234"
This way, in order to have the TDML runner skip a test, you MUST include
this attribute which can provide details. This detail can then be
included in the output when a test is skipped.
Additionally it makes it easy to grep for failed tests and related bug
numbers.
> Nice idea, it will be good to see how far it can reduce the size of the Scala test boilerplate code! Oh, it occurs to me that sometimes it's useful to debug a TDML test case directly from the IDE by running a Scala test method under the debugger. How often do developers need to debug existing or new TDML test cases, which would require them to write throw-away Scala test methods by hand whenever necessary? Maybe we can code up some sort of universal Scala entry point we can run from the IDE which can run any desired TDML test case by editing (at most) one or two lines of code.
Yeah, I think this is probably the the most important functionality to
have, and probably isn't viable if IDE support isn't there. How does
this integrate with IDEs? Maybe it "just works" for IDEs that support
sbt? What about if an IDE supports BSP since our current version of SBT
supports BSP now? But what about IDE's like eclipse. I'm not sure if any
devs use Eclipse anymore since IntelliJ seems much better, but maybe
something to consider? Can we claim we only support IDE's that support
SBT/BSP?
Also, SBT has the testOnly command to run a single testsuite/test. Is
that supported with this? For example, currently we can run
sbt testOnly org.apache.daffodil....TestSuite -- --tests=test_regex
So it takes a test suite and an optional regex of tests in that suite to
run. Having this similar functionality would be pretty critical.
Presumably, it would be very similar to the above, but maybe just a path
to the TDML file instead, e.g.
sbt testOnly org/apache/daffodil/.../TestSuite.tdml -- --tests=test_regex
Using the path to the tdml file as the test suite name avoids issues
with duplicate test suite names in TDML files. Also makes for easily
copy/pasting a path when you want to test a tdml file.
Some other thoughts:
Does anything output when a test fails? Or is that still to be implemented?
I'm also curious how this works regarding Daffodil versions. Presumably
this depends on a certain Daffodil version? What happens if a repo has a
different Daffodil dependency? Does this override that? Or does this
just expect that Daffodil is already a dependency and uses that version?
And hopefully our TDMLRUnner API is stable enough that things just work?
We definitely have cases where we need to run tests with different
Daffodil versions, it would be nice if we didn't have to also change the
plugin version to match.
We also need to consider how this gets published. Do you plan to
maintain and publish this separately as an outside plugin that Daffodil
just depends on an uses? Or do you plant to have it merged into a
Daffodil repo? If so, I wonder if it lives in the Daffodil repo so
Daffodil always has the latest, and then this is published when there
are Daffodil releases? Or maybe it's a separate Apache repo that follows
the same Apache release process/requirements, but can be on a different
schedule than Daffodil?
If all the above issue can potentially be resolved, I think there's a
good chance this could replace all of our boiler plate, which I'm in big
favor of.
> -----Original Message-----
> From: John Wass <jw...@gmail.com>
> Sent: Friday, January 8, 2021 4:27 PM
> To: dev@daffodil.apache.org
> Subject: EXT: rfc: sbt-tdml
>
> https://github.com/jw3/sbt-tdml
> I needed to get smarter on SBT plugins for a task and tinkered with something in Daffodil to do it.
>
> The goal here was to remove the TDML test boilerplate. I threw this at the daffodil-test project with surpsisingly sane results on the first try. See the bottom of the readme.
>
> The learning process is pretty much complete, so not sure where this goes, but wanted see if there are any comments.
>
RE: rfc: sbt-tdml
Posted by "Interrante, John A (GE Research, US)" <in...@research.ge.com>.
I like the idea of removing the Scala TDML test boilerplate as much as possible. Does your sbt plugin handle the use case where some Scala test files define customized or multiple TDML runners in the same file? For example, see daffodil-test/src/test/scala/org/apache/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala where two runners are defined for the same TDML file (expressions.tdml, one standard runner and one customized runner with validateDFDLSchemas = false) and several other customized runners are defined (expressions2.tdml with compileAllTopLevel = true). Even if your sbt plugin can define only a standard runner for each TDML file, it would be nice to remove most of the Scala TDML test code except for the code using customized runners. I'm sure there will be some edge cases where the sbt plugin must not run a TDML file with a standard runner because it works only with a customized runner, but the ability to specify configuration files in TDML or tell the sbt plugin to skip some TDML files may help somewhat.
We need to make sure we preserve the ability to skip some TDML test cases with a string explaining why they are failing (usually a JIRA bug number). If we comment out failing test cases directly in the TDML file then we take the risk that these tests will bitrot faster once the TDML processor no longer parses them. I looked at daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd and I think we could modify the TDML schema to support skipping failing test cases. Both parserTestCase and unparserTestCase use the same attributeGroup (testCaseAttribs). We could add a new optional string-type attribute "reasonForSkipping" to testCaseAttribs and modify the sbt plugin or the TDML processor to automatically skip such test cases (perhaps printing their names and/or the reasons for skipping them, but we don't want to make the test output too big).
Nice idea, it will be good to see how far it can reduce the size of the Scala test boilerplate code! Oh, it occurs to me that sometimes it's useful to debug a TDML test case directly from the IDE by running a Scala test method under the debugger. How often do developers need to debug existing or new TDML test cases, which would require them to write throw-away Scala test methods by hand whenever necessary? Maybe we can code up some sort of universal Scala entry point we can run from the IDE which can run any desired TDML test case by editing (at most) one or two lines of code.
-----Original Message-----
From: John Wass <jw...@gmail.com>
Sent: Friday, January 8, 2021 4:27 PM
To: dev@daffodil.apache.org
Subject: EXT: rfc: sbt-tdml
https://github.com/jw3/sbt-tdml
I needed to get smarter on SBT plugins for a task and tinkered with something in Daffodil to do it.
The goal here was to remove the TDML test boilerplate. I threw this at the daffodil-test project with surpsisingly sane results on the first try. See the bottom of the readme.
The learning process is pretty much complete, so not sure where this goes, but wanted see if there are any comments.