Java 9's Modules System - not a help for dependency isolation problem

["Beckerle, Mike" <mb...@owlcyberdefense.com>]

We have many *many* dependencies from Daffodil, and this has the potential to cause conflicts when creating systems that use Daffodil as part of a larger system.

If an application requires libraries A and B (suppose B is daffodil), and those each in turn require library C (suppose C is the ICU libraries), but A and B each depend on different incompatible versions of C, then you have an unresolvable conflict.

This is called the "dependency isolation" problem. One would like to be able to link libraries A and B with their own respective isolated versions of library C, with no interactions.

In the past I had examined the OGSI modular components system (used by Eclipse) and rejected it for excess complexity.
OGSI solves the dependency isolation problem, but it also introduces a lot of additional long-running software system lifecycle mechanism that is quite complex, and not well motivated for a library like Daffodil.

I had hoped that the Java 9 modules stuff would be a rethink on all of this, and that it would be focused on solving only selective linking (leaving out what you don't use), and dependency isolation.

What I have learned is that Java 9 modules does half of what I wanted.

It successfully allows selective linking and has been used to modularize the gigantic Java runtime to enable smaller footprint in memory. That much is good.

But the Java 9 modules system does *not* solve the dependency isolation problem. There can still only be one copy of each library version for all dependencies of all modules.

I would include references, but a web search for Java 9 Modules OGSI gets you many hits and you can find the discussions easily.

Tools like scala-steward don't eliminate the need for dependency isolation, but they do minimize it, especially for open-source software.

People who are building systems combining large libraries like Daffodil with commercial slow-changing software libraries having dependencies on older versions of many other things, those are the people who really run into the need for a dependency isolation solution.

For the time being they will have to continue to rely on running incompatible things in separate processes or containers or using OGSI if they want everything in the same process/JVM.

Mike Beckerle | Principal Engineer

[cid:61c182a7-6d2e-4f39-98e0-cb25ac62333c]

mbeckerle@owlcyberdefense.com<ma...@owlcyberdefense.com>

P +1-781-330-0412

Re: Java 9's Modules System - not a help for dependency isolation problem

[Christofer Dutz <ch...@c-ware.de>]

Sorry for posting again ... with the last email I noticed I had again 
only replied to Mike and not to the list (Still not used to this new 
Email client)

Chris


On 02.07.21 18:25, Christofer Dutz wrote:
> Hi Mike,
> 
> Another option would be to "relocate" them.
> The Maven Shade plugin tries to address this problem by "relocating"
> given packages. This means the classes and resources are renamed and
> moved to a different package. If for example something is in
> org.apache.commons.whatever you can relocate that to
> daffodil.org.apache.commmons.whatever and this resolves the problem.
> 
> But it duplicates code that doesn't need to be duplicated, it makes the
> jar A LOT bigger.
> 
> However you can use it in your application wherever you encounter such a
> problem.
> 
> Here the link to the plugin:
> 
> https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
> 
> Chris
> 
> On 29.06.21 18:42, Beckerle, Mike wrote:
>> We have many *many* dependencies from Daffodil, and this has the
>> potential to cause conflicts when creating systems that use Daffodil as
>> part of a larger system.
>>
>> If an application requires libraries A and B (suppose B is daffodil),
>> and those each in turn require library C (suppose C is the ICU
>> libraries), but A and B each depend on different incompatible versions
>> of C, then you have an unresolvable conflict.
>>
>> This is called the "dependency isolation" problem. One would like to be
>> able to link libraries A and B with their own respective isolated
>> versions of library C, with no interactions.
>>
>> In the past I had examined the OGSI modular components system (used by
>> Eclipse) and rejected it for excess complexity.
>> OGSI solves the dependency isolation problem, but it also introduces a
>> lot of additional long-running software system lifecycle mechanism that
>> is quite complex, and not well motivated for a library like Daffodil.
>>
>> I had hoped that the Java 9 modules stuff would be a rethink on all of
>> this, and that it would be focused on solving only /selective linking/
>> (leaving out what you don't use), and dependency isolation.
>>
>> What I have learned is that Java 9 modules does half of what I wanted.
>>
>> It successfully allows selective linking and has been used to modularize
>> the gigantic Java runtime to enable smaller footprint in memory. That
>> much is good.
>>
>> But the Java 9 modules system does *not* solve the dependency isolation
>> problem. There can still only be one copy of each library version for
>> all dependencies of all modules.
>>
>> I would include references, but a web search for Java 9 Modules OGSI
>> gets you many hits and you can find the discussions easily.
>>
>> Tools like scala-steward don't eliminate the need for dependency
>> isolation, but they do minimize it, especially for open-source software.
>>
>> People who are building systems combining large libraries like Daffodil
>> with commercial slow-changing software libraries having dependencies on
>> older versions of many other things, those are the people who really run
>> into the need for a dependency isolation solution.
>>
>> For the time being they will have to continue to rely on running
>> incompatible things in separate processes or containers or using OGSI if
>> they want everything in the same process/JVM.
>>
>> Mike Beckerle | Principal Engineer
>>
>> mbeckerle@owlcyberdefense.com <ma...@owlcyberdefense.com>
>>
>> P +1-781-330-0412
>>

Re: Java 9's Modules System - not a help for dependency isolation problem

[Christofer Dutz <ch...@c-ware.de>]

Hi Mike,

Another option would be to "relocate" them.
The Maven Shade plugin tries to address this problem by "relocating"
given packages. This means the classes and resources are renamed and
moved to a different package. If for example something is in
org.apache.commons.whatever you can relocate that to
daffodil.org.apache.commmons.whatever and this resolves the problem.

But it duplicates code that doesn't need to be duplicated, it makes the
jar A LOT bigger.

However you can use it in your application wherever you encounter such a
problem.

Here the link to the plugin:

https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html

Chris

On 29.06.21 18:42, Beckerle, Mike wrote:
> We have many *many* dependencies from Daffodil, and this has the 
> potential to cause conflicts when creating systems that use Daffodil as 
> part of a larger system.
> 
> If an application requires libraries A and B (suppose B is daffodil), 
> and those each in turn require library C (suppose C is the ICU 
> libraries), but A and B each depend on different incompatible versions 
> of C, then you have an unresolvable conflict.
> 
> This is called the "dependency isolation" problem. One would like to be 
> able to link libraries A and B with their own respective isolated 
> versions of library C, with no interactions.
> 
> In the past I had examined the OGSI modular components system (used by 
> Eclipse) and rejected it for excess complexity.
> OGSI solves the dependency isolation problem, but it also introduces a 
> lot of additional long-running software system lifecycle mechanism that 
> is quite complex, and not well motivated for a library like Daffodil.
> 
> I had hoped that the Java 9 modules stuff would be a rethink on all of 
> this, and that it would be focused on solving only /selective linking/ 
> (leaving out what you don't use), and dependency isolation.
> 
> What I have learned is that Java 9 modules does half of what I wanted.
> 
> It successfully allows selective linking and has been used to modularize 
> the gigantic Java runtime to enable smaller footprint in memory. That 
> much is good.
> 
> But the Java 9 modules system does *not* solve the dependency isolation 
> problem. There can still only be one copy of each library version for 
> all dependencies of all modules.
> 
> I would include references, but a web search for Java 9 Modules OGSI 
> gets you many hits and you can find the discussions easily.
> 
> Tools like scala-steward don't eliminate the need for dependency 
> isolation, but they do minimize it, especially for open-source software.
> 
> People who are building systems combining large libraries like Daffodil 
> with commercial slow-changing software libraries having dependencies on 
> older versions of many other things, those are the people who really run 
> into the need for a dependency isolation solution.
> 
> For the time being they will have to continue to rely on running 
> incompatible things in separate processes or containers or using OGSI if 
> they want everything in the same process/JVM.
> 
> Mike Beckerle | Principal Engineer
> 
> mbeckerle@owlcyberdefense.com <ma...@owlcyberdefense.com>
> 
> P +1-781-330-0412
> 

Re: Java 9's Modules System - not a help for dependency isolation problem

[Christofer Dutz <ch...@c-ware.de>]

Hi Adam,

as I mentioned in the other post I mistakinly sent only to Mike and not 
to the list. I also agree that Shading is sort of the only way I know of 
which doesn't bring the overhead OSGI and the Java Module system bring 
with them. But it comes at quite a cost.

Usually this is what produces huge jar files. I would guess in 90% of 
the cases there aren't even any problems to solve. Usually it even works 
if different versions of an API are being used as long as the API parts 
being used haven't gone missing.

However a Shaded jar being added as an option in addition to the normal 
jar might be an option. So you can switch to that in case you run into 
problems and the rest can use the small ones.

I would assume that Shading should work for Scala as in the end Jars and 
java bytecode are produced and I think the shade plugin works on this 
bytecode and not on source-level.

But I have to admit, that I haven't tried it.

Chris



On 02.07.21 17:19, Adam Rosien wrote:
> The only other technique I know of is library "shading", where the build
> system aliases the packages of the classes to something unique and
> "relinks" the build to use that custom namespace for the library, allowing
> multiple versions to coexist on the same classpath. Daffodil could then
> shade the most common libs, to avoid upstream mixing, or there could be
> pointers to instructions for integrators of Daffodil so they can more
> easily shade their own dependencies.
> 
> I don't recall the mechanics of shading in Scala, but there's plenty of
> links via Google, like
> https://medium.com/the-code-shelf/are-you-stuck-in-the-dependency-hell-2faa9943798a
> .
> 
> Apologies if this possibility has already been considered.
> 
> .. Adam
> 
> On Tue, Jun 29, 2021 at 9:42 AM Beckerle, Mike <
> mbeckerle@owlcyberdefense.com> wrote:
> 
>> We have many *many* dependencies from Daffodil, and this has the potential
>> to cause conflicts when creating systems that use Daffodil as part of a
>> larger system.
>>
>> If an application requires libraries A and B (suppose B is daffodil), and
>> those each in turn require library C (suppose C is the ICU libraries), but
>> A and B each depend on different incompatible versions of C, then you have
>> an unresolvable conflict.
>>
>> This is called the "dependency isolation" problem. One would like to be
>> able to link libraries A and B with their own respective isolated versions
>> of library C, with no interactions.
>>
>> In the past I had examined the OGSI modular components system (used by
>> Eclipse) and rejected it for excess complexity.
>> OGSI solves the dependency isolation problem, but it also introduces a lot
>> of additional long-running software system lifecycle mechanism that is
>> quite complex, and not well motivated for a library like Daffodil.
>>
>> I had hoped that the Java 9 modules stuff would be a rethink on all of
>> this, and that it would be focused on solving only *selective linking*
>> (leaving out what you don't use), and dependency isolation.
>>
>> What I have learned is that Java 9 modules does half of what I wanted.
>>
>> It successfully allows selective linking and has been used to modularize
>> the gigantic Java runtime to enable smaller footprint in memory. That much
>> is good.
>>
>> But the Java 9 modules system does *not* solve the dependency isolation
>> problem. There can still only be one copy of each library version for all
>> dependencies of all modules.
>>
>> I would include references, but a web search for Java 9 Modules OGSI gets
>> you many hits and you can find the discussions easily.
>>
>> Tools like scala-steward don't eliminate the need for dependency
>> isolation, but they do minimize it, especially for open-source software.
>>
>> People who are building systems combining large libraries like Daffodil
>> with commercial slow-changing software libraries having dependencies on
>> older versions of many other things, those are the people who really run
>> into the need for a dependency isolation solution.
>>
>> For the time being they will have to continue to rely on running
>> incompatible things in separate processes or containers or using OGSI if
>> they want everything in the same process/JVM.
>>
>> Mike Beckerle | Principal Engineer
>>
>> mbeckerle@owlcyberdefense.com <bh...@owlcyberdefense.com>
>> P +1-781-330-0412
>>
>>
> 

Re: Java 9's Modules System - not a help for dependency isolation problem

[Adam Rosien <ad...@rosien.net>]

The only other technique I know of is library "shading", where the build
system aliases the packages of the classes to something unique and
"relinks" the build to use that custom namespace for the library, allowing
multiple versions to coexist on the same classpath. Daffodil could then
shade the most common libs, to avoid upstream mixing, or there could be
pointers to instructions for integrators of Daffodil so they can more
easily shade their own dependencies.

I don't recall the mechanics of shading in Scala, but there's plenty of
links via Google, like
https://medium.com/the-code-shelf/are-you-stuck-in-the-dependency-hell-2faa9943798a
.

Apologies if this possibility has already been considered.

.. Adam

On Tue, Jun 29, 2021 at 9:42 AM Beckerle, Mike <
mbeckerle@owlcyberdefense.com> wrote:

> We have many *many* dependencies from Daffodil, and this has the potential
> to cause conflicts when creating systems that use Daffodil as part of a
> larger system.
>
> If an application requires libraries A and B (suppose B is daffodil), and
> those each in turn require library C (suppose C is the ICU libraries), but
> A and B each depend on different incompatible versions of C, then you have
> an unresolvable conflict.
>
> This is called the "dependency isolation" problem. One would like to be
> able to link libraries A and B with their own respective isolated versions
> of library C, with no interactions.
>
> In the past I had examined the OGSI modular components system (used by
> Eclipse) and rejected it for excess complexity.
> OGSI solves the dependency isolation problem, but it also introduces a lot
> of additional long-running software system lifecycle mechanism that is
> quite complex, and not well motivated for a library like Daffodil.
>
> I had hoped that the Java 9 modules stuff would be a rethink on all of
> this, and that it would be focused on solving only *selective linking*
> (leaving out what you don't use), and dependency isolation.
>
> What I have learned is that Java 9 modules does half of what I wanted.
>
> It successfully allows selective linking and has been used to modularize
> the gigantic Java runtime to enable smaller footprint in memory. That much
> is good.
>
> But the Java 9 modules system does *not* solve the dependency isolation
> problem. There can still only be one copy of each library version for all
> dependencies of all modules.
>
> I would include references, but a web search for Java 9 Modules OGSI gets
> you many hits and you can find the discussions easily.
>
> Tools like scala-steward don't eliminate the need for dependency
> isolation, but they do minimize it, especially for open-source software.
>
> People who are building systems combining large libraries like Daffodil
> with commercial slow-changing software libraries having dependencies on
> older versions of many other things, those are the people who really run
> into the need for a dependency isolation solution.
>
> For the time being they will have to continue to rely on running
> incompatible things in separate processes or containers or using OGSI if
> they want everything in the same process/JVM.
>
> Mike Beckerle | Principal Engineer
>
> mbeckerle@owlcyberdefense.com <bh...@owlcyberdefense.com>
> P +1-781-330-0412
>
>