dynamically loaded jars, and dependencies thereof

[Mike Beckerle <mb...@apache.org>]

So I wrote a layer transformer that unzips zip files in data.

It depends on apache commons compress library.

Daffodil recognizes that a schema requires this, and dynamically loads it.

But then things fail because apache commons compress library is not found.

We have to manually add apache commons compress jar to the daffodil
classpath.

Is this expected behavior?

Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense | www.owlcyberdefense.com

Re: dynamically loaded jars, and dependencies thereof

[Mike Beckerle <mb...@apache.org>]

Fat jars would need to implement Java-Modules/OGSI-style isolation,
otherwise a depencency jar contained in the fat jar could conflict with
some other dependency.

Do fat jars do that, or are they just a means of packaging these things all
together so you don't have to chase them down and add to the classpath
individually?

Alternatively both fat and regular (non-fat) jars would be needed so you
can use the fat jar, but if you get conflicts, go back to the non-fat and
doing things the hard way.

I definitely agree I don't want to add maven/ivy support to the CLI.

But in some sense the CLI's "parse --schema schemaFilePath" option is not
particularly helpful. As currently formulated it suggests that's all that
is needed to specify the schema to use. The reality is that when that
schema has dependencies, much more is needed. We're finding that most
"real" schemas have these dependencies.

Perhaps we start by just saying that plug-ins (of any kind, layers, UDFs)
should document that they have dependencies, and how to use sbt to list
them (in case the doc gets out of date), and how to add them to the
classpath via DAFFODIL_CLASSPATH.

On Wed, Jan 11, 2023 at 9:53 AM Steve Lawrence <sl...@apache.org> wrote:

> Ultimately all jars need to end up on the classpath somehow.
>
> Maybe one option is to add maven/ivy support to the Daffodil CLI and add
> way to specify a dependency, e.g
>
>    daffodil parse --dependency com.example.transforms/ZipLayer@1.2.0 ...
>
> Then Daffodil could download any dependencies (excluding Daffodil deps
> that it already has) and update the classpath, maybe with a custom
> classloader or maybe reexecing itself with a new claspath or something.
>
>
> The feels a bit extreme though--I'm not sure I want the CLI to have
> maven/ivy support.
>
> I think maybe a more reasonable option is to document how to create a
> fat jar for these kinds of things (with a way to exclude the Daffodil
> dependencies), and then users only need to add a single jar to their
> classpath. It still requires a little user intervention but it's not
> terrible.
>
> And it looks like the sbt-assembly plugin can already create a fat jar
> and exclude dependencies using the "provided" keyword which just a
> couple of sbt lines:
>
>    https://www.baeldung.com/scala/sbt
>
> This is nice too because this will include schemas, transformers, and
> dependencies (both Java and schema dependencies) all in a single jar, so
> it's a all in one bundle that can easily be put on the class path that
> contains everything you could need.
>
> We've also discussed a Daffodil/DFDL plugin in the past to simplify
> schema development. Having that plugin also configure these fat jars for
> releases might make a lot of sense. This way, all you would do is add
> the DFDL plugin to your project, define your dependencies the normal
> way, put your schemas and transforms in the right place, and run "sbt
> assembly", and you'll get a fat jar to put on the classpath.
>
>
> On 2023-01-10 05:52 PM, Mike Beckerle wrote:
> > So when I use daffodil by way of sbt, all the managed dependencies of the
> > zip-layer-transformer (really just on apache commons compress) are taken
> > care of for me.
> >
> > My main schema just declares a managed dependency on a specific version
> of
> > the zip-layer-transformer, and voila, the transitive dependency on the
> > apache commons compress takes care of itself.
> >
> > When I use the CLI, I have to do this stuff manually. It's amazingly hard
> > to get DAFFODIL_CLASSPATH complete and correct.
> >
> > 'sbt show dependencyClasspath' shows me the dependency on apache commons
> > compress library, of course it also has all the daffodil jars and all
> their
> > dependencies.
> >
> > How do we make the CLI or any environment for running Daffodil, as easy
> as
> > sbt?
> >
> >
> > On Tue, Jan 10, 2023 at 3:36 PM Steve Lawrence <sl...@apache.org>
> wrote:
> >
> >> I think so. Daffodil can't know what libraries a layer transformer might
> >> need, so we can't depend on them or include them in releases.
> >>
> >> So if a transformer plugin requires one or more libraries that don't
> >> already ship with Daffodil, then they should be marked as a dependency
> >> of the transformer and their jars downloaded and included in the
> >> classpath along with the transformer.
> >>
> >> You could maybe get around his by making your transformer a fat jar, but
> >> it might include all the Daffodil jars as well, which might cause some
> >> problems.
> >>
> >>
> >> On 2023-01-10 03:24 PM, Mike Beckerle wrote:
> >>> So I wrote a layer transformer that unzips zip files in data.
> >>>
> >>> It depends on apache commons compress library.
> >>>
> >>> Daffodil recognizes that a schema requires this, and dynamically loads
> >> it.
> >>>
> >>> But then things fail because apache commons compress library is not
> >> found.
> >>>
> >>> We have to manually add apache commons compress jar to the daffodil
> >>> classpath.
> >>>
> >>> Is this expected behavior?
> >>>
> >>> Mike Beckerle
> >>> Apache Daffodil PMC | daffodil.apache.org
> >>> OGF DFDL Workgroup Co-Chair |
> >> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> >>> Owl Cyber Defense | www.owlcyberdefense.com
> >>>
> >>
> >>
> >
>
>

Re: dynamically loaded jars, and dependencies thereof

[Steve Lawrence <sl...@apache.org>]

Ultimately all jars need to end up on the classpath somehow.

Maybe one option is to add maven/ivy support to the Daffodil CLI and add 
way to specify a dependency, e.g

   daffodil parse --dependency com.example.transforms/ZipLayer@1.2.0 ...

Then Daffodil could download any dependencies (excluding Daffodil deps 
that it already has) and update the classpath, maybe with a custom 
classloader or maybe reexecing itself with a new claspath or something.


The feels a bit extreme though--I'm not sure I want the CLI to have 
maven/ivy support.

I think maybe a more reasonable option is to document how to create a 
fat jar for these kinds of things (with a way to exclude the Daffodil 
dependencies), and then users only need to add a single jar to their 
classpath. It still requires a little user intervention but it's not 
terrible.

And it looks like the sbt-assembly plugin can already create a fat jar 
and exclude dependencies using the "provided" keyword which just a 
couple of sbt lines:

   https://www.baeldung.com/scala/sbt

This is nice too because this will include schemas, transformers, and 
dependencies (both Java and schema dependencies) all in a single jar, so 
it's a all in one bundle that can easily be put on the class path that 
contains everything you could need.

We've also discussed a Daffodil/DFDL plugin in the past to simplify 
schema development. Having that plugin also configure these fat jars for 
releases might make a lot of sense. This way, all you would do is add 
the DFDL plugin to your project, define your dependencies the normal 
way, put your schemas and transforms in the right place, and run "sbt 
assembly", and you'll get a fat jar to put on the classpath.


On 2023-01-10 05:52 PM, Mike Beckerle wrote:
> So when I use daffodil by way of sbt, all the managed dependencies of the
> zip-layer-transformer (really just on apache commons compress) are taken
> care of for me.
> 
> My main schema just declares a managed dependency on a specific version of
> the zip-layer-transformer, and voila, the transitive dependency on the
> apache commons compress takes care of itself.
> 
> When I use the CLI, I have to do this stuff manually. It's amazingly hard
> to get DAFFODIL_CLASSPATH complete and correct.
> 
> 'sbt show dependencyClasspath' shows me the dependency on apache commons
> compress library, of course it also has all the daffodil jars and all their
> dependencies.
> 
> How do we make the CLI or any environment for running Daffodil, as easy as
> sbt?
> 
> 
> On Tue, Jan 10, 2023 at 3:36 PM Steve Lawrence <sl...@apache.org> wrote:
> 
>> I think so. Daffodil can't know what libraries a layer transformer might
>> need, so we can't depend on them or include them in releases.
>>
>> So if a transformer plugin requires one or more libraries that don't
>> already ship with Daffodil, then they should be marked as a dependency
>> of the transformer and their jars downloaded and included in the
>> classpath along with the transformer.
>>
>> You could maybe get around his by making your transformer a fat jar, but
>> it might include all the Daffodil jars as well, which might cause some
>> problems.
>>
>>
>> On 2023-01-10 03:24 PM, Mike Beckerle wrote:
>>> So I wrote a layer transformer that unzips zip files in data.
>>>
>>> It depends on apache commons compress library.
>>>
>>> Daffodil recognizes that a schema requires this, and dynamically loads
>> it.
>>>
>>> But then things fail because apache commons compress library is not
>> found.
>>>
>>> We have to manually add apache commons compress jar to the daffodil
>>> classpath.
>>>
>>> Is this expected behavior?
>>>
>>> Mike Beckerle
>>> Apache Daffodil PMC | daffodil.apache.org
>>> OGF DFDL Workgroup Co-Chair |
>> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
>>> Owl Cyber Defense | www.owlcyberdefense.com
>>>
>>
>>
> 

Re: dynamically loaded jars, and dependencies thereof

[Mike Beckerle <mb...@apache.org>]

So when I use daffodil by way of sbt, all the managed dependencies of the
zip-layer-transformer (really just on apache commons compress) are taken
care of for me.

My main schema just declares a managed dependency on a specific version of
the zip-layer-transformer, and voila, the transitive dependency on the
apache commons compress takes care of itself.

When I use the CLI, I have to do this stuff manually. It's amazingly hard
to get DAFFODIL_CLASSPATH complete and correct.

'sbt show dependencyClasspath' shows me the dependency on apache commons
compress library, of course it also has all the daffodil jars and all their
dependencies.

How do we make the CLI or any environment for running Daffodil, as easy as
sbt?


On Tue, Jan 10, 2023 at 3:36 PM Steve Lawrence <sl...@apache.org> wrote:

> I think so. Daffodil can't know what libraries a layer transformer might
> need, so we can't depend on them or include them in releases.
>
> So if a transformer plugin requires one or more libraries that don't
> already ship with Daffodil, then they should be marked as a dependency
> of the transformer and their jars downloaded and included in the
> classpath along with the transformer.
>
> You could maybe get around his by making your transformer a fat jar, but
> it might include all the Daffodil jars as well, which might cause some
> problems.
>
>
> On 2023-01-10 03:24 PM, Mike Beckerle wrote:
> > So I wrote a layer transformer that unzips zip files in data.
> >
> > It depends on apache commons compress library.
> >
> > Daffodil recognizes that a schema requires this, and dynamically loads
> it.
> >
> > But then things fail because apache commons compress library is not
> found.
> >
> > We have to manually add apache commons compress jar to the daffodil
> > classpath.
> >
> > Is this expected behavior?
> >
> > Mike Beckerle
> > Apache Daffodil PMC | daffodil.apache.org
> > OGF DFDL Workgroup Co-Chair |
> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> > Owl Cyber Defense | www.owlcyberdefense.com
> >
>
>

Re: dynamically loaded jars, and dependencies thereof

[Steve Lawrence <sl...@apache.org>]

I think so. Daffodil can't know what libraries a layer transformer might 
need, so we can't depend on them or include them in releases.

So if a transformer plugin requires one or more libraries that don't 
already ship with Daffodil, then they should be marked as a dependency 
of the transformer and their jars downloaded and included in the 
classpath along with the transformer.

You could maybe get around his by making your transformer a fat jar, but 
it might include all the Daffodil jars as well, which might cause some 
problems.


On 2023-01-10 03:24 PM, Mike Beckerle wrote:
> So I wrote a layer transformer that unzips zip files in data.
> 
> It depends on apache commons compress library.
> 
> Daffodil recognizes that a schema requires this, and dynamically loads it.
> 
> But then things fail because apache commons compress library is not found.
> 
> We have to manually add apache commons compress jar to the daffodil
> classpath.
> 
> Is this expected behavior?
> 
> Mike Beckerle
> Apache Daffodil PMC | daffodil.apache.org
> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> Owl Cyber Defense | www.owlcyberdefense.com
>