You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@daffodil.apache.org by Mike Beckerle <mb...@apache.org> on 2024/10/25 23:56:10 UTC
sbt-daffodil question - disable automatics dfdl schema compilation
So in IntelliJ when I run a main app that is part of src/test/scala, the
build process runs packageDaffodilBin and that recompiles the entire schema.
This takes *minutes* for this large schema.
Is there a setting I can use to tell it to only recompile the binary schema
when I specifically request it by 'sbt packageDaffodilBin' ?
I don't want to turn off use of the binary by TDML tests. I just want to
turn off all automatic such compilation.
If the compiled binary doesn't exist I want a diagnostic error from
whatever was looking for it, not an automatic recompile.
Re: sbt-daffodil question - disable automatics dfdl schema compilation
Posted by Steve Lawrence <sl...@apache.org>.
> When the TDML runner is called, and the model for the test case identifies a
> binary file, somehow the schema version and daffodil-version information are
> added to it.
>
> Where does the TDML runner get this information from? I would expect that
> information is available to SBT, but how can the TDML runner get it?
Are you asking this because packageDaffodilBin creates files like
"dfdl-foo-1.0-message-daffodil390.bin" and you're asking how the TDML runner
knows to use "1.0" and "daffodil390", and also the "message" classifier to find
the binary file since those aren't including in the model?
The answer is that it doesn't. When daffodilTdmlUsesPackageBin is true, the
plugin runs packageDaffodilBin to create the saved parsers you're familiar with.
But then it actually finds the saved parsers built for daffodilVersion and
copies them to the Test/resourceManaged directory (which is on the test
classpath for the TDML runner to find), and it copies the files so the names do
not include any version information. So for example, the above dfdl-foo saved
parser would be copied to "resource_managed/test/dfdl-foo-message.bin".
This way TDML files do not need to be updated each time the schema version or
daffodilVersion setting is changed. Only names or classifiers require TDML
changes, and those change pretty rarely.
Note that based on this, instead of step 4, you could copy the binary file to
src/test/resources/<name>-<classifer>.bin. Then you don't need to do step 2 and
can keep your TDML files the same. You just have to remember to not commit the
build.sbt or src/test/resources changes.
Also, now that I think about how SBT works, I'm not even sure SBT can support
what you're asking for. Support for conditional tasks in SBT is kindof limited.
You either do the task and all sub-tasks, or you do none of it. There's no way
to do one part of a task (e.g. copy saved parser if it exists and add to
classpath) without also doing another part (build saved parsers). Maybe it's
possible with some complicated reorganization of the plugin, but it's already
complicated enough. I think modifying daffodilTdmlUsesPackageBin might be the
best option to get what you want. If that becomes too much of a hassle I'm happy
to revisit this and see if there's a way to make it work, but I'm not sure there is.
On 2024-10-28 12:31 PM, Mike Beckerle wrote:
> I can think of many situations where I have a pre-compiled schema, I've made
> changes, but I still want to run using the old one, not wait 2 or 3 minutes for
> a new one to be created.
>
> Here's my workaround:
>
> 1. Temporarily set daffodilTdmlUsesPackageBin := false
> 2. Temporarily change model attribute of every test case to point at a tempdir
> location.
> 1. (Enhancement: a tdml:testSuite defaultModel attribute where one could
> specify the model to be used in one place, not in every test case. This
> makes modifying the test suite to point it at a specific location for
> the binary model file a one-line edit, not a giant multi-replacement)
> 3. sbt packageDaffodilBin
> 4. copy the binary file to the tmpdir location.
>
> Here is what I don't understand.
>
> When the TDML runner is called, and the model for the test case identifies a
> binary file, somehow the schema version and daffodil-version information are
> added to it.
>
> Where does the TDML runner get this information from? I would expect that
> information is available to SBT, but how can the TDML runner get it?
>
>
> On Mon, Oct 28, 2024 at 9:36 AM Steve Lawrence <slawrence@apache.org
> <ma...@apache.org>> wrote:
>
> There is not currently a setting that does that. We could probably add one
> pretty easily, but it feels like it could be a bit fragile?
>
> For example, say this new setting is enabled and you run packageDaffodilBin to
> build a saved parser and you run "sbt test" and all test pass. Then you make
> changes to the schema that accidentally break things, but forget to run
> "packgeDaffodilBin". If you run "sbt test" all the tests will pass because
> you're using an older saved parser even though they should fail. The current
> logic detects when schema changes have been made and will force a rebuild when
> you run tests.
>
> Would it work to temporarily change daffodilTdmlUsesPackageBin to false while
> you do things that don't need a binary and then change it back when your done?
> Or would there be too much switching?
>
> If that would be too much of a headache, we can definitely add setting, but it
> would have to come with a warning that it's up to the user to not forget to
> rebuild schemas when running tests. We might be able to output a warning if
> tests are run and the cache is old, but dealing with SBT cache is tricky, so it
> might not be possible.
>
>
> On 2024-10-25 07:56 PM, Mike Beckerle wrote:
> >
> > So in IntelliJ when I run a main app that is part of src/test/scala, the
> build
> > process runs packageDaffodilBin and that recompiles the entire schema.
> >
> > This takes *minutes* for this large schema.
> >
> > Is there a setting I can use to tell it to only recompile the binary
> schema when
> > I specifically request it by 'sbt packageDaffodilBin' ?
> >
> > I don't want to turn off use of the binary by TDML tests. I just want to
> turn
> > off all automatic such compilation.
> >
> > If the compiled binary doesn't exist I want a diagnostic error from
> whatever was
> > looking for it, not an automatic recompile.
> >
> >
> >
> >
> >
>
Re: sbt-daffodil question - disable automatics dfdl schema compilation
Posted by Mike Beckerle <mb...@apache.org>.
I can think of many situations where I have a pre-compiled schema, I've
made changes, but I still want to run using the old one, not wait 2 or 3
minutes for a new one to be created.
Here's my workaround:
1. Temporarily set daffodilTdmlUsesPackageBin := false
2. Temporarily change model attribute of every test case to point at a
tempdir location.
1. (Enhancement: a tdml:testSuite defaultModel attribute where one
could specify the model to be used in one place, not in every test case.
This makes modifying the test suite to point it at a specific
location for
the binary model file a one-line edit, not a giant multi-replacement)
3. sbt packageDaffodilBin
4. copy the binary file to the tmpdir location.
Here is what I don't understand.
When the TDML runner is called, and the model for the test case identifies
a binary file, somehow the schema version and daffodil-version information
are added to it.
Where does the TDML runner get this information from? I would expect that
information is available to SBT, but how can the TDML runner get it?
On Mon, Oct 28, 2024 at 9:36 AM Steve Lawrence <sl...@apache.org> wrote:
> There is not currently a setting that does that. We could probably add one
> pretty easily, but it feels like it could be a bit fragile?
>
> For example, say this new setting is enabled and you run
> packageDaffodilBin to
> build a saved parser and you run "sbt test" and all test pass. Then you
> make
> changes to the schema that accidentally break things, but forget to run
> "packgeDaffodilBin". If you run "sbt test" all the tests will pass because
> you're using an older saved parser even though they should fail. The
> current
> logic detects when schema changes have been made and will force a rebuild
> when
> you run tests.
>
> Would it work to temporarily change daffodilTdmlUsesPackageBin to false
> while
> you do things that don't need a binary and then change it back when your
> done?
> Or would there be too much switching?
>
> If that would be too much of a headache, we can definitely add setting,
> but it
> would have to come with a warning that it's up to the user to not forget
> to
> rebuild schemas when running tests. We might be able to output a warning
> if
> tests are run and the cache is old, but dealing with SBT cache is tricky,
> so it
> might not be possible.
>
>
> On 2024-10-25 07:56 PM, Mike Beckerle wrote:
> >
> > So in IntelliJ when I run a main app that is part of src/test/scala, the
> build
> > process runs packageDaffodilBin and that recompiles the entire schema.
> >
> > This takes *minutes* for this large schema.
> >
> > Is there a setting I can use to tell it to only recompile the binary
> schema when
> > I specifically request it by 'sbt packageDaffodilBin' ?
> >
> > I don't want to turn off use of the binary by TDML tests. I just want to
> turn
> > off all automatic such compilation.
> >
> > If the compiled binary doesn't exist I want a diagnostic error from
> whatever was
> > looking for it, not an automatic recompile.
> >
> >
> >
> >
> >
>
>
Re: sbt-daffodil question - disable automatics dfdl schema compilation
Posted by Steve Lawrence <sl...@apache.org>.
There is not currently a setting that does that. We could probably add one
pretty easily, but it feels like it could be a bit fragile?
For example, say this new setting is enabled and you run packageDaffodilBin to
build a saved parser and you run "sbt test" and all test pass. Then you make
changes to the schema that accidentally break things, but forget to run
"packgeDaffodilBin". If you run "sbt test" all the tests will pass because
you're using an older saved parser even though they should fail. The current
logic detects when schema changes have been made and will force a rebuild when
you run tests.
Would it work to temporarily change daffodilTdmlUsesPackageBin to false while
you do things that don't need a binary and then change it back when your done?
Or would there be too much switching?
If that would be too much of a headache, we can definitely add setting, but it
would have to come with a warning that it's up to the user to not forget to
rebuild schemas when running tests. We might be able to output a warning if
tests are run and the cache is old, but dealing with SBT cache is tricky, so it
might not be possible.
On 2024-10-25 07:56 PM, Mike Beckerle wrote:
>
> So in IntelliJ when I run a main app that is part of src/test/scala, the build
> process runs packageDaffodilBin and that recompiles the entire schema.
>
> This takes *minutes* for this large schema.
>
> Is there a setting I can use to tell it to only recompile the binary schema when
> I specifically request it by 'sbt packageDaffodilBin' ?
>
> I don't want to turn off use of the binary by TDML tests. I just want to turn
> off all automatic such compilation.
>
> If the compiled binary doesn't exist I want a diagnostic error from whatever was
> looking for it, not an automatic recompile.
>
>
>
>
>