DAFFODIL-1927

[Sandeep Kumar <sn...@gmail.com>]

Hi Team,

I'd like to work on the following task :
https://issues.apache.org/jira/browse/DAFFODIL-1927
Can you please help with the initial steps for this?

Regards,
Sandeep

Re: DAFFODIL-1927

["Beckerle, Mike" <mb...@owlcyberdefense.com>]

This is an important change. There is design work to do here for how it should work.

There are two features in Daffodil that dynamically load extensions already: User-defined functions, and Validators.

So you should be able to look at those to figure out how to dynamically load layer implementations from the classpath.

One possible approach is to take one of the existing layering transforms, like the one for AIS payload armoring, and make that into a dynamically loaded one.

There are already tests for that, so you would be able to tell easily if your dynamically loaded version works the same without having to invent a lot of tests.

And AIS is an obscure and special-purpose thing that really should​ be outside of daffodil as a loadable layer transform.

Arguably, all of the layer transforms should be in external libraries that live in separate jars.

See daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/AISTransformer.scala

You'll see that these are "registered" in the file LayerTransformer.scala, and that mechanism is what must change to get the transformer from an external classpath/jar.

I am happy if the layer transformers have to be written in Scala for now. A later enhancement can be to make a polished Java API for these extensions. The UDF feature allows writing UDFs in either Java or Scala, but I don't think we need to do a Java API as yet for this layering stuff, or I'd start by just doing scala for now.

If a layer definition moves out to a separate jar, the testing for it also needs to be expressed outside as well.
The unit tests this is straightforward to to. But there are TDML-based tests in

daffodil-test/src/test/resources/org/apache/daffodil/layers/ais.tdml
daffodil-test/src/test/scala/org/apache/daffodil/layers/TestAIS.scala

The dynamic loading aspect of the user-defined function (UDF) feature is in

daffodil-runtime1/src/main/scala/org/apache/daffodil/udf/UserDefinedFunctionService.scala

It has TDML tests as well in:

daffodil-test/src/test/scala/org/apache/daffodil/udf/TestUdfsInSchemas.scala

but the UDFs used to test, are themselves defined in

daffodil-udf/src/test/scala/org/sgoodudfs/example/StringFunctions/StringFunctionsProvider.scala

Which is a separate module of daffodil, Note these are for testing, and so live under src/test in that module. So they're not delivered as part of Daffodil jars.

The primary developer of UDF was Olabusayo Kilo, and the primary developer of the dynamically loadable validations was John Wass (jw3)

________________________________
From: Sandeep Kumar <sn...@gmail.com>
Sent: Thursday, May 20, 2021 7:47 AM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: DAFFODIL-1927

Hi Team,

I'd like to work on the following task :
https://issues.apache.org/jira/browse/DAFFODIL-1927
Can you please help with the initial steps for this?

Regards,
Sandeep