You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by Mike Beckerle <mb...@apache.org> on 2021/10/06 17:47:55 UTC
Please review PR for - Re: Design discussion - dynamically loadable
layers - DAFFODIL-1927
I have done a bunch of improvements to the code for this layer loading
based on the prior PR.
It is now stabilized and worth it to examine/reexamine the PR at this point.
https://github.com/apache/daffodil/pull/651
I have also changed the goals here somewhat. I split out the
defining/creation of a Java API for layers, and true dynamic loading via
service loaders into a separate JIRA ticket DAFFODIL-2567.
I think that experience with what we have so far makes sense before
defining any Java API for doing this. I also think the need for true
dynamic detect and load behavior is far less critical than the basic
ability to define these in external jars that aren't part of daffodil.
This PR now includes just the modularization of layers, so they can be
written in scala and defined in a jar outside of Daffodil, An application
must register them for use by calling LayerCompilerRegistry.register(...).
All the built-in layers are in daffodil-runtime1-layers which is a new
module.
Other layers are defined in daffodil-test for testing that checksums/check
digits work. The example AIS payload encoding layer is also now defined and
tested in daffodil-test.
On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle <mb...@apache.org> wrote:
> Please see:
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
>
> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
>
> See related PR: https://github.com/apache/daffodil/pull/643 which is for
> https://issues.apache.org/jira/browse/DAFFODIL-2221
>
> Please add comments or directly edit that wiki page, or discuss in this
> email thread if you prefer.
>
>
Re: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Mike Beckerle <mb...@apache.org>.
Latest push to the PR adds true ServiceLoader feature to dynamically
discover and load the layer transformers.
This should now work with CLI, vscode, etc.
There is still no Java API for writing a layer transformer in Java. I don't
know that this is actually a requirement.
On Wed, Oct 6, 2021 at 5:40 PM Larry Barber <la...@nteligen.com>
wrote:
> It seems like it would need to work with the CLI, debugger, as well as the
> NiFi version
>
> -----Original Message-----
> From: Mike Beckerle <mb...@apache.org>
> Sent: Wednesday, October 6, 2021 4:38 PM
> To: dev@daffodil.apache.org
> Subject: Re: Please review PR for - Re: Design discussion - dynamically
> loadable layers - DAFFODIL-1927
>
> Good point. Won't work from the new debugger. I guess I have to figure out
> the service loader stuff.
>
> On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence <sl...@apache.org>
> wrote:
>
> > Ah, right, I didn't think about registering it from within the test
> > suite object.
> >
> > Though, that isn't possible for the CLI, and I'd guess it also isn't
> > possible from the new VSCode debugger.
> >
> >
> > On 10/6/21 4:09 PM, Mike Beckerle wrote:
> > > Much easier than that.
> > >
> > > Look at TestAIS.scala:
> > >
> > >
> > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a
> > a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91&data=04%7C01%7C
> > larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2
> > 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT
> > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > 6Mn0%3D%7C3000&sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS
> > gLpc%3D&reserved=0
> > >
> > > You just call LayerCompilerRegistry.Register(
> > > AISPayloadArmoringLayerCompiler)
> > > From the test object.
> > >
> > > That's all an application using Daffodil with a custom layer has to
> > > do also.
> > >
> > > Honestly, I almost prefer this to dynamic loading.
> > >
> > >
> > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence <sl...@apache.org>
> > wrote:
> > >
> > >> Don't those only work because they are built in to Daffodil and
> > >> pre-registered? If someone wants to keep their layer outside of
> > >> Daffodil, it needs to be manually registered, but that can't be
> > >> done by the TDML Runner or CLI?
> > >>
> > >> On 10/6/21 3:38 PM, Mike Beckerle wrote:
> > >>> The TDML runner invoked from JUnit works fine.
> > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4
> > >>> examples in daffodil-test work this way, and anyone's DFDL schema
> > >>> could do the same thing.
> > >>>
> > >>> I plan to update the ethernetIP and PCAP schemas to compute their
> > >> checksums
> > >>> this way.
> > >>>
> > >>> Only the CLI is left out.
> > >>>
> > >>>
> > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence
> > >>> <sl...@apache.org>
> > >> wrote:
> > >>>
> > >>>> If we don't support dynamic loading and instead require that
> > >>>> users manually register the layer, that means that externally
> > >>>> defined layers cannot be tested using the TDML runner, right? It
> > >>>> also means there's
> > no
> > >>>> way to run schemas with external layers using the CLI?
> > >>>>
> > >>>> I guess that's not an issue until there are external layers, so
> > >>>> maybe not critical, but still feels pretty important. Maybe
> > >>>> adding it to the next release is reasonable, since that's
> > >>>> probably when external layers might start showing up?
> > >>>>
> > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote:
> > >>>>> I have done a bunch of improvements to the code for this layer
> > loading
> > >>>>> based on the prior PR.
> > >>>>>
> > >>>>> It is now stabilized and worth it to examine/reexamine the PR at
> > >>>>> this
> > >>>> point.
> > >>>>>
> > >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F
> > >>>>> %2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F651&data=04%7C01%
> > >>>>> 7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc
> > >>>>> %7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839
> > >>>>> %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> > >>>>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7YEJPwd2M%2BaGjrgtu
> > >>>>> 0z3bQL%2BN%2Fi1fCSFnu4PMeKH9MY%3D&reserved=0
> > >>>>>
> > >>>>> I have also changed the goals here somewhat. I split out the
> > >>>>> defining/creation of a Java API for layers, and true dynamic
> > >>>>> loading
> > >> via
> > >>>>> service loaders into a separate JIRA ticket DAFFODIL-2567.
> > >>>>>
> > >>>>> I think that experience with what we have so far makes sense
> > >>>>> before defining any Java API for doing this. I also think the
> > >>>>> need for true dynamic detect and load behavior is far less
> > >>>>> critical than the basic ability to define these in external jars
> > >>>>> that aren't part of
> > daffodil.
> > >>>>>
> > >>>>> This PR now includes just the modularization of layers, so they
> > >>>>> can
> > be
> > >>>>> written in scala and defined in a jar outside of Daffodil, An
> > >> application
> > >>>>> must register them for use by calling
> > >>>> LayerCompilerRegistry.register(...).
> > >>>>>
> > >>>>> All the built-in layers are in daffodil-runtime1-layers which is
> > >>>>> a
> > new
> > >>>>> module.
> > >>>>>
> > >>>>> Other layers are defined in daffodil-test for testing that
> > >>>> checksums/check
> > >>>>> digits work. The example AIS payload encoding layer is also now
> > defined
> > >>>> and
> > >>>>> tested in daffodil-test.
> > >>>>>
> > >>>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle
> > >>>>> <mb...@apache.org>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Please see:
> > >>>>>>
> > >>>>
> > >>
> > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fcwi
> > ki.apache.org%2Fconfluence%2Fdisplay%2FDAFFODIL%2FProposal%253A%2BDyna
> > mically%2Bloading%2BLayer%2BTransformations&data=04%7C01%7Clarry.b
> > arber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c214c5c94
> > 4e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > %7C3000&sdata=VG0y4Wrh3rkI2Ou8DzMu0rEgSHA7Oy4yeE%2FD1Vu7RbQ%3D&
> > ;reserved=0
> > >>>>>>
> > >>>>>> Also:
> > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2
> > >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-1927&data=
> > >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d
> > >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149
> > >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
> > >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Dm9bfd0
> > >>>>>> yosgB22nvGRMsbYvpAhx1LqsnobSsZQ9DR%2Fg%3D&reserved=0
> > >>>>>>
> > >>>>>> See related PR:
> > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2
> > >>>>>> F%2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F643&data=04%7C0
> > >>>>>> 1%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d989092
> > >>>>>> 9fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149687189
> > >>>>>> 4833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> > >>>>>> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=zD%2FWa3zLDGU
> > >>>>>> XIohNzN6n6Ro2umRyKlfdUCT0lsXxyns%3D&reserved=0 which
> > is
> > >>>> for
> > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2
> > >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-2221&data=
> > >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d
> > >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149
> > >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
> > >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FPn%2
> > >>>>>> B76%2FJ4MdpkHE%2BEBRwaqyasOK%2B33TBpNW96l%2F7FeA%3D&reserve
> > >>>>>> d=0
> > >>>>>>
> > >>>>>> Please add comments or directly edit that wiki page, or discuss
> > >>>>>> in
> > >> this
> > >>>>>> email thread if you prefer.
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >
> >
> >
>
RE: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Larry Barber <la...@nteligen.com>.
It seems like it would need to work with the CLI, debugger, as well as the NiFi version
-----Original Message-----
From: Mike Beckerle <mb...@apache.org>
Sent: Wednesday, October 6, 2021 4:38 PM
To: dev@daffodil.apache.org
Subject: Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Good point. Won't work from the new debugger. I guess I have to figure out the service loader stuff.
On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence <sl...@apache.org> wrote:
> Ah, right, I didn't think about registering it from within the test
> suite object.
>
> Though, that isn't possible for the CLI, and I'd guess it also isn't
> possible from the new VSCode debugger.
>
>
> On 10/6/21 4:09 PM, Mike Beckerle wrote:
> > Much easier than that.
> >
> > Look at TestAIS.scala:
> >
> >
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit
> hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a
> a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91&data=04%7C01%7C
> larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2
> 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT
> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000&sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS
> gLpc%3D&reserved=0
> >
> > You just call LayerCompilerRegistry.Register(
> > AISPayloadArmoringLayerCompiler)
> > From the test object.
> >
> > That's all an application using Daffodil with a custom layer has to
> > do also.
> >
> > Honestly, I almost prefer this to dynamic loading.
> >
> >
> > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence <sl...@apache.org>
> wrote:
> >
> >> Don't those only work because they are built in to Daffodil and
> >> pre-registered? If someone wants to keep their layer outside of
> >> Daffodil, it needs to be manually registered, but that can't be
> >> done by the TDML Runner or CLI?
> >>
> >> On 10/6/21 3:38 PM, Mike Beckerle wrote:
> >>> The TDML runner invoked from JUnit works fine.
> >>> The AIS Payload Armoring, and the CheckDigits and the IPv4
> >>> examples in daffodil-test work this way, and anyone's DFDL schema
> >>> could do the same thing.
> >>>
> >>> I plan to update the ethernetIP and PCAP schemas to compute their
> >> checksums
> >>> this way.
> >>>
> >>> Only the CLI is left out.
> >>>
> >>>
> >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence
> >>> <sl...@apache.org>
> >> wrote:
> >>>
> >>>> If we don't support dynamic loading and instead require that
> >>>> users manually register the layer, that means that externally
> >>>> defined layers cannot be tested using the TDML runner, right? It
> >>>> also means there's
> no
> >>>> way to run schemas with external layers using the CLI?
> >>>>
> >>>> I guess that's not an issue until there are external layers, so
> >>>> maybe not critical, but still feels pretty important. Maybe
> >>>> adding it to the next release is reasonable, since that's
> >>>> probably when external layers might start showing up?
> >>>>
> >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote:
> >>>>> I have done a bunch of improvements to the code for this layer
> loading
> >>>>> based on the prior PR.
> >>>>>
> >>>>> It is now stabilized and worth it to examine/reexamine the PR at
> >>>>> this
> >>>> point.
> >>>>>
> >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F
> >>>>> %2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F651&data=04%7C01%
> >>>>> 7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc
> >>>>> %7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839
> >>>>> %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> >>>>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7YEJPwd2M%2BaGjrgtu
> >>>>> 0z3bQL%2BN%2Fi1fCSFnu4PMeKH9MY%3D&reserved=0
> >>>>>
> >>>>> I have also changed the goals here somewhat. I split out the
> >>>>> defining/creation of a Java API for layers, and true dynamic
> >>>>> loading
> >> via
> >>>>> service loaders into a separate JIRA ticket DAFFODIL-2567.
> >>>>>
> >>>>> I think that experience with what we have so far makes sense
> >>>>> before defining any Java API for doing this. I also think the
> >>>>> need for true dynamic detect and load behavior is far less
> >>>>> critical than the basic ability to define these in external jars
> >>>>> that aren't part of
> daffodil.
> >>>>>
> >>>>> This PR now includes just the modularization of layers, so they
> >>>>> can
> be
> >>>>> written in scala and defined in a jar outside of Daffodil, An
> >> application
> >>>>> must register them for use by calling
> >>>> LayerCompilerRegistry.register(...).
> >>>>>
> >>>>> All the built-in layers are in daffodil-runtime1-layers which is
> >>>>> a
> new
> >>>>> module.
> >>>>>
> >>>>> Other layers are defined in daffodil-test for testing that
> >>>> checksums/check
> >>>>> digits work. The example AIS payload encoding layer is also now
> defined
> >>>> and
> >>>>> tested in daffodil-test.
> >>>>>
> >>>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle
> >>>>> <mb...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> Please see:
> >>>>>>
> >>>>
> >>
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fcwi
> ki.apache.org%2Fconfluence%2Fdisplay%2FDAFFODIL%2FProposal%253A%2BDyna
> mically%2Bloading%2BLayer%2BTransformations&data=04%7C01%7Clarry.b
> arber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c214c5c94
> 4e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000&sdata=VG0y4Wrh3rkI2Ou8DzMu0rEgSHA7Oy4yeE%2FD1Vu7RbQ%3D&
> ;reserved=0
> >>>>>>
> >>>>>> Also:
> >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2
> >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-1927&data=
> >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d
> >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149
> >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
> >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Dm9bfd0
> >>>>>> yosgB22nvGRMsbYvpAhx1LqsnobSsZQ9DR%2Fg%3D&reserved=0
> >>>>>>
> >>>>>> See related PR:
> >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2
> >>>>>> F%2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F643&data=04%7C0
> >>>>>> 1%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d989092
> >>>>>> 9fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149687189
> >>>>>> 4833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> >>>>>> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=zD%2FWa3zLDGU
> >>>>>> XIohNzN6n6Ro2umRyKlfdUCT0lsXxyns%3D&reserved=0 which
> is
> >>>> for
> >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2
> >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-2221&data=
> >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d
> >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149
> >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
> >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FPn%2
> >>>>>> B76%2FJ4MdpkHE%2BEBRwaqyasOK%2B33TBpNW96l%2F7FeA%3D&reserve
> >>>>>> d=0
> >>>>>>
> >>>>>> Please add comments or directly edit that wiki page, or discuss
> >>>>>> in
> >> this
> >>>>>> email thread if you prefer.
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Mike Beckerle <mb...@apache.org>.
Good point. Won't work from the new debugger. I guess I have to figure out
the service loader stuff.
On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence <sl...@apache.org> wrote:
> Ah, right, I didn't think about registering it from within the test
> suite object.
>
> Though, that isn't possible for the CLI, and I'd guess it also isn't
> possible from the new VSCode debugger.
>
>
> On 10/6/21 4:09 PM, Mike Beckerle wrote:
> > Much easier than that.
> >
> > Look at TestAIS.scala:
> >
> >
> https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91
> >
> > You just call LayerCompilerRegistry.Register(
> > AISPayloadArmoringLayerCompiler)
> > From the test object.
> >
> > That's all an application using Daffodil with a custom layer has to do
> > also.
> >
> > Honestly, I almost prefer this to dynamic loading.
> >
> >
> > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence <sl...@apache.org>
> wrote:
> >
> >> Don't those only work because they are built in to Daffodil and
> >> pre-registered? If someone wants to keep their layer outside of
> >> Daffodil, it needs to be manually registered, but that can't be done by
> >> the TDML Runner or CLI?
> >>
> >> On 10/6/21 3:38 PM, Mike Beckerle wrote:
> >>> The TDML runner invoked from JUnit works fine.
> >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in
> >>> daffodil-test work this way, and anyone's DFDL schema could do the same
> >>> thing.
> >>>
> >>> I plan to update the ethernetIP and PCAP schemas to compute their
> >> checksums
> >>> this way.
> >>>
> >>> Only the CLI is left out.
> >>>
> >>>
> >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence <sl...@apache.org>
> >> wrote:
> >>>
> >>>> If we don't support dynamic loading and instead require that users
> >>>> manually register the layer, that means that externally defined layers
> >>>> cannot be tested using the TDML runner, right? It also means there's
> no
> >>>> way to run schemas with external layers using the CLI?
> >>>>
> >>>> I guess that's not an issue until there are external layers, so maybe
> >>>> not critical, but still feels pretty important. Maybe adding it to the
> >>>> next release is reasonable, since that's probably when external layers
> >>>> might start showing up?
> >>>>
> >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote:
> >>>>> I have done a bunch of improvements to the code for this layer
> loading
> >>>>> based on the prior PR.
> >>>>>
> >>>>> It is now stabilized and worth it to examine/reexamine the PR at this
> >>>> point.
> >>>>>
> >>>>> https://github.com/apache/daffodil/pull/651
> >>>>>
> >>>>> I have also changed the goals here somewhat. I split out the
> >>>>> defining/creation of a Java API for layers, and true dynamic loading
> >> via
> >>>>> service loaders into a separate JIRA ticket DAFFODIL-2567.
> >>>>>
> >>>>> I think that experience with what we have so far makes sense before
> >>>>> defining any Java API for doing this. I also think the need for true
> >>>>> dynamic detect and load behavior is far less critical than the basic
> >>>>> ability to define these in external jars that aren't part of
> daffodil.
> >>>>>
> >>>>> This PR now includes just the modularization of layers, so they can
> be
> >>>>> written in scala and defined in a jar outside of Daffodil, An
> >> application
> >>>>> must register them for use by calling
> >>>> LayerCompilerRegistry.register(...).
> >>>>>
> >>>>> All the built-in layers are in daffodil-runtime1-layers which is a
> new
> >>>>> module.
> >>>>>
> >>>>> Other layers are defined in daffodil-test for testing that
> >>>> checksums/check
> >>>>> digits work. The example AIS payload encoding layer is also now
> defined
> >>>> and
> >>>>> tested in daffodil-test.
> >>>>>
> >>>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle <mb...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> Please see:
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
> >>>>>>
> >>>>>> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
> >>>>>>
> >>>>>> See related PR: https://github.com/apache/daffodil/pull/643 which
> is
> >>>> for
> >>>>>> https://issues.apache.org/jira/browse/DAFFODIL-2221
> >>>>>>
> >>>>>> Please add comments or directly edit that wiki page, or discuss in
> >> this
> >>>>>> email thread if you prefer.
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Steve Lawrence <sl...@apache.org>.
Ah, right, I didn't think about registering it from within the test
suite object.
Though, that isn't possible for the CLI, and I'd guess it also isn't
possible from the new VSCode debugger.
On 10/6/21 4:09 PM, Mike Beckerle wrote:
> Much easier than that.
>
> Look at TestAIS.scala:
>
> https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91
>
> You just call LayerCompilerRegistry.Register(
> AISPayloadArmoringLayerCompiler)
> From the test object.
>
> That's all an application using Daffodil with a custom layer has to do
> also.
>
> Honestly, I almost prefer this to dynamic loading.
>
>
> On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence <sl...@apache.org> wrote:
>
>> Don't those only work because they are built in to Daffodil and
>> pre-registered? If someone wants to keep their layer outside of
>> Daffodil, it needs to be manually registered, but that can't be done by
>> the TDML Runner or CLI?
>>
>> On 10/6/21 3:38 PM, Mike Beckerle wrote:
>>> The TDML runner invoked from JUnit works fine.
>>> The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in
>>> daffodil-test work this way, and anyone's DFDL schema could do the same
>>> thing.
>>>
>>> I plan to update the ethernetIP and PCAP schemas to compute their
>> checksums
>>> this way.
>>>
>>> Only the CLI is left out.
>>>
>>>
>>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence <sl...@apache.org>
>> wrote:
>>>
>>>> If we don't support dynamic loading and instead require that users
>>>> manually register the layer, that means that externally defined layers
>>>> cannot be tested using the TDML runner, right? It also means there's no
>>>> way to run schemas with external layers using the CLI?
>>>>
>>>> I guess that's not an issue until there are external layers, so maybe
>>>> not critical, but still feels pretty important. Maybe adding it to the
>>>> next release is reasonable, since that's probably when external layers
>>>> might start showing up?
>>>>
>>>> On 10/6/21 1:47 PM, Mike Beckerle wrote:
>>>>> I have done a bunch of improvements to the code for this layer loading
>>>>> based on the prior PR.
>>>>>
>>>>> It is now stabilized and worth it to examine/reexamine the PR at this
>>>> point.
>>>>>
>>>>> https://github.com/apache/daffodil/pull/651
>>>>>
>>>>> I have also changed the goals here somewhat. I split out the
>>>>> defining/creation of a Java API for layers, and true dynamic loading
>> via
>>>>> service loaders into a separate JIRA ticket DAFFODIL-2567.
>>>>>
>>>>> I think that experience with what we have so far makes sense before
>>>>> defining any Java API for doing this. I also think the need for true
>>>>> dynamic detect and load behavior is far less critical than the basic
>>>>> ability to define these in external jars that aren't part of daffodil.
>>>>>
>>>>> This PR now includes just the modularization of layers, so they can be
>>>>> written in scala and defined in a jar outside of Daffodil, An
>> application
>>>>> must register them for use by calling
>>>> LayerCompilerRegistry.register(...).
>>>>>
>>>>> All the built-in layers are in daffodil-runtime1-layers which is a new
>>>>> module.
>>>>>
>>>>> Other layers are defined in daffodil-test for testing that
>>>> checksums/check
>>>>> digits work. The example AIS payload encoding layer is also now defined
>>>> and
>>>>> tested in daffodil-test.
>>>>>
>>>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle <mb...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Please see:
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
>>>>>>
>>>>>> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
>>>>>>
>>>>>> See related PR: https://github.com/apache/daffodil/pull/643 which is
>>>> for
>>>>>> https://issues.apache.org/jira/browse/DAFFODIL-2221
>>>>>>
>>>>>> Please add comments or directly edit that wiki page, or discuss in
>> this
>>>>>> email thread if you prefer.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Mike Beckerle <mb...@apache.org>.
Much easier than that.
Look at TestAIS.scala:
https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91
You just call LayerCompilerRegistry.Register(
AISPayloadArmoringLayerCompiler)
From the test object.
That's all an application using Daffodil with a custom layer has to do
also.
Honestly, I almost prefer this to dynamic loading.
On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence <sl...@apache.org> wrote:
> Don't those only work because they are built in to Daffodil and
> pre-registered? If someone wants to keep their layer outside of
> Daffodil, it needs to be manually registered, but that can't be done by
> the TDML Runner or CLI?
>
> On 10/6/21 3:38 PM, Mike Beckerle wrote:
> > The TDML runner invoked from JUnit works fine.
> > The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in
> > daffodil-test work this way, and anyone's DFDL schema could do the same
> > thing.
> >
> > I plan to update the ethernetIP and PCAP schemas to compute their
> checksums
> > this way.
> >
> > Only the CLI is left out.
> >
> >
> > On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence <sl...@apache.org>
> wrote:
> >
> >> If we don't support dynamic loading and instead require that users
> >> manually register the layer, that means that externally defined layers
> >> cannot be tested using the TDML runner, right? It also means there's no
> >> way to run schemas with external layers using the CLI?
> >>
> >> I guess that's not an issue until there are external layers, so maybe
> >> not critical, but still feels pretty important. Maybe adding it to the
> >> next release is reasonable, since that's probably when external layers
> >> might start showing up?
> >>
> >> On 10/6/21 1:47 PM, Mike Beckerle wrote:
> >>> I have done a bunch of improvements to the code for this layer loading
> >>> based on the prior PR.
> >>>
> >>> It is now stabilized and worth it to examine/reexamine the PR at this
> >> point.
> >>>
> >>> https://github.com/apache/daffodil/pull/651
> >>>
> >>> I have also changed the goals here somewhat. I split out the
> >>> defining/creation of a Java API for layers, and true dynamic loading
> via
> >>> service loaders into a separate JIRA ticket DAFFODIL-2567.
> >>>
> >>> I think that experience with what we have so far makes sense before
> >>> defining any Java API for doing this. I also think the need for true
> >>> dynamic detect and load behavior is far less critical than the basic
> >>> ability to define these in external jars that aren't part of daffodil.
> >>>
> >>> This PR now includes just the modularization of layers, so they can be
> >>> written in scala and defined in a jar outside of Daffodil, An
> application
> >>> must register them for use by calling
> >> LayerCompilerRegistry.register(...).
> >>>
> >>> All the built-in layers are in daffodil-runtime1-layers which is a new
> >>> module.
> >>>
> >>> Other layers are defined in daffodil-test for testing that
> >> checksums/check
> >>> digits work. The example AIS payload encoding layer is also now defined
> >> and
> >>> tested in daffodil-test.
> >>>
> >>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle <mb...@apache.org>
> >> wrote:
> >>>
> >>>> Please see:
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
> >>>>
> >>>> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
> >>>>
> >>>> See related PR: https://github.com/apache/daffodil/pull/643 which is
> >> for
> >>>> https://issues.apache.org/jira/browse/DAFFODIL-2221
> >>>>
> >>>> Please add comments or directly edit that wiki page, or discuss in
> this
> >>>> email thread if you prefer.
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Steve Lawrence <sl...@apache.org>.
Don't those only work because they are built in to Daffodil and
pre-registered? If someone wants to keep their layer outside of
Daffodil, it needs to be manually registered, but that can't be done by
the TDML Runner or CLI?
On 10/6/21 3:38 PM, Mike Beckerle wrote:
> The TDML runner invoked from JUnit works fine.
> The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in
> daffodil-test work this way, and anyone's DFDL schema could do the same
> thing.
>
> I plan to update the ethernetIP and PCAP schemas to compute their checksums
> this way.
>
> Only the CLI is left out.
>
>
> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence <sl...@apache.org> wrote:
>
>> If we don't support dynamic loading and instead require that users
>> manually register the layer, that means that externally defined layers
>> cannot be tested using the TDML runner, right? It also means there's no
>> way to run schemas with external layers using the CLI?
>>
>> I guess that's not an issue until there are external layers, so maybe
>> not critical, but still feels pretty important. Maybe adding it to the
>> next release is reasonable, since that's probably when external layers
>> might start showing up?
>>
>> On 10/6/21 1:47 PM, Mike Beckerle wrote:
>>> I have done a bunch of improvements to the code for this layer loading
>>> based on the prior PR.
>>>
>>> It is now stabilized and worth it to examine/reexamine the PR at this
>> point.
>>>
>>> https://github.com/apache/daffodil/pull/651
>>>
>>> I have also changed the goals here somewhat. I split out the
>>> defining/creation of a Java API for layers, and true dynamic loading via
>>> service loaders into a separate JIRA ticket DAFFODIL-2567.
>>>
>>> I think that experience with what we have so far makes sense before
>>> defining any Java API for doing this. I also think the need for true
>>> dynamic detect and load behavior is far less critical than the basic
>>> ability to define these in external jars that aren't part of daffodil.
>>>
>>> This PR now includes just the modularization of layers, so they can be
>>> written in scala and defined in a jar outside of Daffodil, An application
>>> must register them for use by calling
>> LayerCompilerRegistry.register(...).
>>>
>>> All the built-in layers are in daffodil-runtime1-layers which is a new
>>> module.
>>>
>>> Other layers are defined in daffodil-test for testing that
>> checksums/check
>>> digits work. The example AIS payload encoding layer is also now defined
>> and
>>> tested in daffodil-test.
>>>
>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle <mb...@apache.org>
>> wrote:
>>>
>>>> Please see:
>>>>
>> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
>>>>
>>>> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
>>>>
>>>> See related PR: https://github.com/apache/daffodil/pull/643 which is
>> for
>>>> https://issues.apache.org/jira/browse/DAFFODIL-2221
>>>>
>>>> Please add comments or directly edit that wiki page, or discuss in this
>>>> email thread if you prefer.
>>>>
>>>>
>>>
>>
>>
>
Re: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Mike Beckerle <mb...@apache.org>.
The TDML runner invoked from JUnit works fine.
The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in
daffodil-test work this way, and anyone's DFDL schema could do the same
thing.
I plan to update the ethernetIP and PCAP schemas to compute their checksums
this way.
Only the CLI is left out.
On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence <sl...@apache.org> wrote:
> If we don't support dynamic loading and instead require that users
> manually register the layer, that means that externally defined layers
> cannot be tested using the TDML runner, right? It also means there's no
> way to run schemas with external layers using the CLI?
>
> I guess that's not an issue until there are external layers, so maybe
> not critical, but still feels pretty important. Maybe adding it to the
> next release is reasonable, since that's probably when external layers
> might start showing up?
>
> On 10/6/21 1:47 PM, Mike Beckerle wrote:
> > I have done a bunch of improvements to the code for this layer loading
> > based on the prior PR.
> >
> > It is now stabilized and worth it to examine/reexamine the PR at this
> point.
> >
> > https://github.com/apache/daffodil/pull/651
> >
> > I have also changed the goals here somewhat. I split out the
> > defining/creation of a Java API for layers, and true dynamic loading via
> > service loaders into a separate JIRA ticket DAFFODIL-2567.
> >
> > I think that experience with what we have so far makes sense before
> > defining any Java API for doing this. I also think the need for true
> > dynamic detect and load behavior is far less critical than the basic
> > ability to define these in external jars that aren't part of daffodil.
> >
> > This PR now includes just the modularization of layers, so they can be
> > written in scala and defined in a jar outside of Daffodil, An application
> > must register them for use by calling
> LayerCompilerRegistry.register(...).
> >
> > All the built-in layers are in daffodil-runtime1-layers which is a new
> > module.
> >
> > Other layers are defined in daffodil-test for testing that
> checksums/check
> > digits work. The example AIS payload encoding layer is also now defined
> and
> > tested in daffodil-test.
> >
> > On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle <mb...@apache.org>
> wrote:
> >
> >> Please see:
> >>
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
> >>
> >> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
> >>
> >> See related PR: https://github.com/apache/daffodil/pull/643 which is
> for
> >> https://issues.apache.org/jira/browse/DAFFODIL-2221
> >>
> >> Please add comments or directly edit that wiki page, or discuss in this
> >> email thread if you prefer.
> >>
> >>
> >
>
>
Re: Please review PR for - Re: Design discussion - dynamically
loadable layers - DAFFODIL-1927
Posted by Steve Lawrence <sl...@apache.org>.
If we don't support dynamic loading and instead require that users
manually register the layer, that means that externally defined layers
cannot be tested using the TDML runner, right? It also means there's no
way to run schemas with external layers using the CLI?
I guess that's not an issue until there are external layers, so maybe
not critical, but still feels pretty important. Maybe adding it to the
next release is reasonable, since that's probably when external layers
might start showing up?
On 10/6/21 1:47 PM, Mike Beckerle wrote:
> I have done a bunch of improvements to the code for this layer loading
> based on the prior PR.
>
> It is now stabilized and worth it to examine/reexamine the PR at this point.
>
> https://github.com/apache/daffodil/pull/651
>
> I have also changed the goals here somewhat. I split out the
> defining/creation of a Java API for layers, and true dynamic loading via
> service loaders into a separate JIRA ticket DAFFODIL-2567.
>
> I think that experience with what we have so far makes sense before
> defining any Java API for doing this. I also think the need for true
> dynamic detect and load behavior is far less critical than the basic
> ability to define these in external jars that aren't part of daffodil.
>
> This PR now includes just the modularization of layers, so they can be
> written in scala and defined in a jar outside of Daffodil, An application
> must register them for use by calling LayerCompilerRegistry.register(...).
>
> All the built-in layers are in daffodil-runtime1-layers which is a new
> module.
>
> Other layers are defined in daffodil-test for testing that checksums/check
> digits work. The example AIS payload encoding layer is also now defined and
> tested in daffodil-test.
>
> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle <mb...@apache.org> wrote:
>
>> Please see:
>> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
>>
>> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
>>
>> See related PR: https://github.com/apache/daffodil/pull/643 which is for
>> https://issues.apache.org/jira/browse/DAFFODIL-2221
>>
>> Please add comments or directly edit that wiki page, or discuss in this
>> email thread if you prefer.
>>
>>
>