Re: Java 9's Modules System - not a help for dependency isolation problem

[Adam Rosien <ad...@rosien.net>]

The only other technique I know of is library "shading", where the build
system aliases the packages of the classes to something unique and
"relinks" the build to use that custom namespace for the library, allowing
multiple versions to coexist on the same classpath. Daffodil could then
shade the most common libs, to avoid upstream mixing, or there could be
pointers to instructions for integrators of Daffodil so they can more
easily shade their own dependencies.

I don't recall the mechanics of shading in Scala, but there's plenty of
links via Google, like
https://medium.com/the-code-shelf/are-you-stuck-in-the-dependency-hell-2faa9943798a
.

Apologies if this possibility has already been considered.

.. Adam

On Tue, Jun 29, 2021 at 9:42 AM Beckerle, Mike <
mbeckerle@owlcyberdefense.com> wrote:

> We have many *many* dependencies from Daffodil, and this has the potential
> to cause conflicts when creating systems that use Daffodil as part of a
> larger system.
>
> If an application requires libraries A and B (suppose B is daffodil), and
> those each in turn require library C (suppose C is the ICU libraries), but
> A and B each depend on different incompatible versions of C, then you have
> an unresolvable conflict.
>
> This is called the "dependency isolation" problem. One would like to be
> able to link libraries A and B with their own respective isolated versions
> of library C, with no interactions.
>
> In the past I had examined the OGSI modular components system (used by
> Eclipse) and rejected it for excess complexity.
> OGSI solves the dependency isolation problem, but it also introduces a lot
> of additional long-running software system lifecycle mechanism that is
> quite complex, and not well motivated for a library like Daffodil.
>
> I had hoped that the Java 9 modules stuff would be a rethink on all of
> this, and that it would be focused on solving only *selective linking*
> (leaving out what you don't use), and dependency isolation.
>
> What I have learned is that Java 9 modules does half of what I wanted.
>
> It successfully allows selective linking and has been used to modularize
> the gigantic Java runtime to enable smaller footprint in memory. That much
> is good.
>
> But the Java 9 modules system does *not* solve the dependency isolation
> problem. There can still only be one copy of each library version for all
> dependencies of all modules.
>
> I would include references, but a web search for Java 9 Modules OGSI gets
> you many hits and you can find the discussions easily.
>
> Tools like scala-steward don't eliminate the need for dependency
> isolation, but they do minimize it, especially for open-source software.
>
> People who are building systems combining large libraries like Daffodil
> with commercial slow-changing software libraries having dependencies on
> older versions of many other things, those are the people who really run
> into the need for a dependency isolation solution.
>
> For the time being they will have to continue to rely on running
> incompatible things in separate processes or containers or using OGSI if
> they want everything in the same process/JVM.
>
> Mike Beckerle | Principal Engineer
>
> mbeckerle@owlcyberdefense.com <bh...@owlcyberdefense.com>
> P +1-781-330-0412
>
>

Re: Java 9's Modules System - not a help for dependency isolation problem

[Christofer Dutz <ch...@c-ware.de>]

Hi Adam,

as I mentioned in the other post I mistakinly sent only to Mike and not 
to the list. I also agree that Shading is sort of the only way I know of 
which doesn't bring the overhead OSGI and the Java Module system bring 
with them. But it comes at quite a cost.

Usually this is what produces huge jar files. I would guess in 90% of 
the cases there aren't even any problems to solve. Usually it even works 
if different versions of an API are being used as long as the API parts 
being used haven't gone missing.

However a Shaded jar being added as an option in addition to the normal 
jar might be an option. So you can switch to that in case you run into 
problems and the rest can use the small ones.

I would assume that Shading should work for Scala as in the end Jars and 
java bytecode are produced and I think the shade plugin works on this 
bytecode and not on source-level.

But I have to admit, that I haven't tried it.

Chris



On 02.07.21 17:19, Adam Rosien wrote:
> The only other technique I know of is library "shading", where the build
> system aliases the packages of the classes to something unique and
> "relinks" the build to use that custom namespace for the library, allowing
> multiple versions to coexist on the same classpath. Daffodil could then
> shade the most common libs, to avoid upstream mixing, or there could be
> pointers to instructions for integrators of Daffodil so they can more
> easily shade their own dependencies.
> 
> I don't recall the mechanics of shading in Scala, but there's plenty of
> links via Google, like
> https://medium.com/the-code-shelf/are-you-stuck-in-the-dependency-hell-2faa9943798a
> .
> 
> Apologies if this possibility has already been considered.
> 
> .. Adam
> 
> On Tue, Jun 29, 2021 at 9:42 AM Beckerle, Mike <
> mbeckerle@owlcyberdefense.com> wrote:
> 
>> We have many *many* dependencies from Daffodil, and this has the potential
>> to cause conflicts when creating systems that use Daffodil as part of a
>> larger system.
>>
>> If an application requires libraries A and B (suppose B is daffodil), and
>> those each in turn require library C (suppose C is the ICU libraries), but
>> A and B each depend on different incompatible versions of C, then you have
>> an unresolvable conflict.
>>
>> This is called the "dependency isolation" problem. One would like to be
>> able to link libraries A and B with their own respective isolated versions
>> of library C, with no interactions.
>>
>> In the past I had examined the OGSI modular components system (used by
>> Eclipse) and rejected it for excess complexity.
>> OGSI solves the dependency isolation problem, but it also introduces a lot
>> of additional long-running software system lifecycle mechanism that is
>> quite complex, and not well motivated for a library like Daffodil.
>>
>> I had hoped that the Java 9 modules stuff would be a rethink on all of
>> this, and that it would be focused on solving only *selective linking*
>> (leaving out what you don't use), and dependency isolation.
>>
>> What I have learned is that Java 9 modules does half of what I wanted.
>>
>> It successfully allows selective linking and has been used to modularize
>> the gigantic Java runtime to enable smaller footprint in memory. That much
>> is good.
>>
>> But the Java 9 modules system does *not* solve the dependency isolation
>> problem. There can still only be one copy of each library version for all
>> dependencies of all modules.
>>
>> I would include references, but a web search for Java 9 Modules OGSI gets
>> you many hits and you can find the discussions easily.
>>
>> Tools like scala-steward don't eliminate the need for dependency
>> isolation, but they do minimize it, especially for open-source software.
>>
>> People who are building systems combining large libraries like Daffodil
>> with commercial slow-changing software libraries having dependencies on
>> older versions of many other things, those are the people who really run
>> into the need for a dependency isolation solution.
>>
>> For the time being they will have to continue to rely on running
>> incompatible things in separate processes or containers or using OGSI if
>> they want everything in the same process/JVM.
>>
>> Mike Beckerle | Principal Engineer
>>
>> mbeckerle@owlcyberdefense.com <bh...@owlcyberdefense.com>
>> P +1-781-330-0412
>>
>>
>