You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Plamen Totev <pl...@gmail.com> on 2017/12/20 06:40:24 UTC
Re: main-class + module-version
Hi,
I've started working on implementing support for modular JARs in
Plexus Archiver but as I research the matter I found that they are
more complex than I expected. I've updated the GitHub issue[1] with
the details but I'll post the findings here as well. The modular JAR
files contain not only version and main class but also:
* List of the packages contained - usually populated by the compiler,
but there are some caveats here. The most obvious one is that for jars
created by the Shade Plugin this attribute will almost certainly be
incorrect - the plugin will most likely introduce additional packages
and the list produced by the compiler will be out of date. The other
caveat are the resources - they could be located in packages and
encapsulated the same way as the classes. The jar tools records not
only the class packages but the resource ones too. Packaging resources
together with the classes may result in incorrect module descriptor if
this attribute is not updated.
* The module main class - passed as argument to the jar tool and it
seems that it cannot be set using the compiler.
* The module version - passed as argument to the jar tool. Could be
set using the compiler(again passed as argument) as well.
* Hashes - calculates the hashes of given external modules that depend
on this one. During runtime java checks the recorded hashes against
the resolved module and and if don't match it will fail. The primary
use case is a module A that exports packages to B and C. An easy way
to break the encapsulation of A is to create module named B. To
prevent that you can record the hash of B and C inside A. Then A will
export the packages only to the intended modules. It is used by the
OpenJDK but could be useful for platforms and frameworks as well.
* Module resolution - marks the module to not be resolved by default or
marks it as deprecated or incubating so at run time a warning is
displayed if it is resolved. I don't think you can set it using the
JDK 9 jar tool but it's present in the code. Maybe it will be
implemented in future version of the jar tool.
To me it looks like the packaging of modular JAR file is more complex
than just packing module-info.class file and the jar tool is no longer
just a ZIP program. Something similar is stated in the tool
documentation[2]:
> The jar command is a general-purpose archiving and compression tool,
> based on the ZIP and ZLIB compression formats.
> Initially, the jar command was designed to package Java applets
> or applications; however, beginning with JDK 9, users can use the
> jar command to create modular JARs.
> For transportation and deployment, it’s usually more convenient to
> package modules as modular JARs.
While not at all that complex from technical point of view, I don't
think it is worth implementing, and whats more maintaining, all this
functionality. Probably it would be better if the plugins that produce
JAR files use the jar tool, the same way the compiler plugin does not
compile by itself but uses the Java compiler.
What do you think?
[1] https://github.com/codehaus-plexus/plexus-archiver/issues/69#issuecomment-349095101
[2] https://docs.oracle.com/javase/9/tools/jar.htm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Plamen Totev <pl...@gmail.com>.
Hi,
Actually that might just work. Good idea. I'll take a look at it.
Thanks,
Plamen Totev
On Wed, Dec 20, 2017 at 9:53 PM, Robert Scholte <rf...@apache.org> wrote:
> On Wed, 20 Dec 2017 07:40:24 +0100, Plamen Totev <pl...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I've started working on implementing support for modular JARs in
>> Plexus Archiver but as I research the matter I found that they are
>> more complex than I expected. I've updated the GitHub issue[1] with
>> the details but I'll post the findings here as well. The modular JAR
>> files contain not only version and main class but also:
>>
>> * List of the packages contained - usually populated by the compiler,
>> but there are some caveats here. The most obvious one is that for jars
>> created by the Shade Plugin this attribute will almost certainly be
>> incorrect - the plugin will most likely introduce additional packages
>> and the list produced by the compiler will be out of date. The other
>> caveat are the resources - they could be located in packages and
>> encapsulated the same way as the classes. The jar tools records not
>> only the class packages but the resource ones too. Packaging resources
>> together with the classes may result in incorrect module descriptor if
>> this attribute is not updated.
>>
>> * The module main class - passed as argument to the jar tool and it
>> seems that it cannot be set using the compiler.
>>
>> * The module version - passed as argument to the jar tool. Could be
>> set using the compiler(again passed as argument) as well.
>>
>> * Hashes - calculates the hashes of given external modules that depend
>> on this one. During runtime java checks the recorded hashes against
>> the resolved module and and if don't match it will fail. The primary
>> use case is a module A that exports packages to B and C. An easy way
>> to break the encapsulation of A is to create module named B. To
>> prevent that you can record the hash of B and C inside A. Then A will
>> export the packages only to the intended modules. It is used by the
>> OpenJDK but could be useful for platforms and frameworks as well.
>>
>> * Module resolution - marks the module to not be resolved by default or
>> marks it as deprecated or incubating so at run time a warning is
>> displayed if it is resolved. I don't think you can set it using the
>> JDK 9 jar tool but it's present in the code. Maybe it will be
>> implemented in future version of the jar tool.
>>
>> To me it looks like the packaging of modular JAR file is more complex
>> than just packing module-info.class file and the jar tool is no longer
>> just a ZIP program. Something similar is stated in the tool
>> documentation[2]:
>>
>>> The jar command is a general-purpose archiving and compression tool,
>>> based on the ZIP and ZLIB compression formats.
>>> Initially, the jar command was designed to package Java applets
>>> or applications; however, beginning with JDK 9, users can use the
>>> jar command to create modular JARs.
>>> For transportation and deployment, it’s usually more convenient to
>>> package modules as modular JARs.
>>
>>
>> While not at all that complex from technical point of view, I don't
>> think it is worth implementing, and whats more maintaining, all this
>> functionality. Probably it would be better if the plugins that produce
>> JAR files use the jar tool, the same way the compiler plugin does not
>> compile by itself but uses the Java compiler.
>>
>> What do you think?
>
>
> Based on this message it seems worth implementing a JarToolArchiver, using
> the jar tool via the ToolProvider[1]
> I hope it can still be a org.codehaus.plexus.archiver.Archiver, otherwise
> I'll contact the openjdk team about the details of the specifications. I
> don't think we need them all, good to know the reason for the extra files.
>
> thanks so far,
> Robert
>
>
> [1]
> https://docs.oracle.com/javase/9/docs/api/jdk.jartool-summary.html#module.description
>
>>
>>
>> [1]
>> https://github.com/codehaus-plexus/plexus-archiver/issues/69#issuecomment-349095101
>> [2] https://docs.oracle.com/javase/9/tools/jar.htm
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Plamen Totev <pl...@gmail.com>.
Hi,
I've done some work on implementing `PlexusArchiver` that creates
modular JAR files using the JDK `jar` tool (`ToolProvider`)[1]. It is
still work in progress but I wanted to ask for you opinion - if there
are no objections I'll polish it(fix the code formatting, implement the
TODO items, etc) and would like eventually to merge it in master. Here
are some key points:
* It requires Java 9. This of course is not the minimum required version
for the rest of the archivers, but `java.util.spi.ToolProvider` is
introduced in Java 9. I'll fix the source and target to be 7, but it
will still require Java 9 to compile and currently Plexus Archiver does
not compile on Java 9. This is easy to fix - requires only update of
some of the plugins to yet to be released versions.
* A new class is added `ModularJarArchiver`. This would allow a "normal"
JARs to be created on Java 7 using `JarArchiver` while modular JARs will
require Java 9 (I don't think this is an issue as you need Java 9 to
compile modules anyway).
* `ModularJarArchiver` uses `JarArchiver` to create a "normal" JAR file
and then uses the JDK `jar` tool to update it to modular one. This may
require some additional work during the build compared to creating the
archive using only the `jar` tool, but `JarArchiver` provides some nice
additional features not available in the `jar` tool(such as not
re-compressing the zip files, generating manifest entries, etc).
What do you think? Any objections to implement it this way? I think this
is better approach than my previous attempt[2].
Regards,
Plamen Totev
[1] https://github.com/plamentotev/plexus-archiver/commit/b124aceb3f6a912a9dca44f5d14b9f3f2919bc6d
[2] https://github.com/codehaus-plexus/plexus-archiver/pull/75
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Plamen Totev <pl...@gmail.com>.
Hi Robert,
On 1/15/2018 10:10 PM, Robert Scholte wrote:
> Hi Plamen,
>
> Alan Bateman has provided some valuable information:
Thank you for sharing this. It's really helpful.
On 1/15/2018 10:10 PM, Robert Scholte wrote:
> 1. MR JARs weren't mentioned. The jar tool does validation to ensure
> that the API provided by the classes in the JAR file is the same for all
> versions.
Validation is indeed something I haven't dig in yet. It is interesting
to know what is the impact of Maven not doing validation of modular JAR
files.
On 1/15/2018 10:10 PM, Robert Scholte wrote:
> 2. The Module, ModulePackages, and ModuleMainClass class file attributes
> are specified in the JVMS. ASM supports them so the Archiver could use
> that. The ModulePackages attribute is optional. If this attribute is not
> present then the JAR file will be scanned to get the set of packages in
> the module.
If the JAR is scanned when ModulePackages is not present then for the
time being is ok to not implement it.
On 1/15/2018 10:10 PM, Robert Scholte wrote:
> The ModuleTarget, ModuleResolution, and ModuleHashes class file
> attributes are JDK-specific so you won't find these in the JVMS. The
> ModuleTarget attribute is documented in JEP 261, the others aren't there
> yet but ASM has support in org.objectweb.asm.commons for these
> attributes so you should be okay.
All of those could be set with ASM (I think) but setting them is the
easy part. Calculating the hashes for ModuleHashes will require some
work though.
On 1/15/2018 10:09 PM, Andreas Sewe wrote:
> Also, if two separate tools modify the JAR, the goal of reproducible
> builds is again a bit harder to accomplish. So, please try to use an
> internal, written-in-Java solution if possible.
This is a valid point. I don't think it would be possible to set the
entries timestamps using the JDK jar tool. But a bit harder does not
mean impossible. It could be achieved by "post-processing" the resulting
jar file.
Another point I think we should have in mind is the fact that every time
the class format version is increased we should update the ASM library
and release a new version of the JAR plugin otherwise the build will
fail. Also the users should update the JAR plugin version. That may
prove to be a bit inconvenient.
From my point of view the trade-off is ease of maintainability vs
flexibility and control.
Regards,
Plamen Totev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Robert Scholte <rf...@apache.org>.
Hi Plamen,
Alan Bateman has provided some valuable information:
---
A first step to use the ToolProvider to run the jar tool make sense,
should be easy to do. We've done quite a bit of work on the jar tool
performance in the last few years so the performance might not be too bad.
Updating org.codehaus.plexus.archiver.Archiver to do the same as the jar
tool is probably a lot of work. A few points on this:
1. MR JARs weren't mentioned. The jar tool does validation to ensure that
the API provided by the classes in the JAR file is the same for all
versions.
2. The Module, ModulePackages, and ModuleMainClass class file attributes
are specified in the JVMS. ASM supports them so the Archiver could use
that. The ModulePackages attribute is optional. If this attribute is not
present then the JAR file will be scanned to get the set of packages in
the module.
The ModuleTarget, ModuleResolution, and ModuleHashes class file attributes
are JDK-specific so you won't find these in the JVMS. The ModuleTarget
attribute is documented in JEP 261, the others aren't there yet but ASM
has support in org.objectweb.asm.commons for these attributes so you
should be okay.
3. The Maven Shade Plugin is a good discussion point. There are several
issues to sort if this is extended to work with modules.
---
thanks,
Robert
On Mon, 15 Jan 2018 20:31:54 +0100, Plamen Totev
<pl...@gmail.com> wrote:
> Hi,
>
> On Mon, Jan 15, 2018 at 8:23 PM, Robert Scholte <rf...@apache.org>
> wrote:
>> So maybe we simply have to split it up into smaller pieces.
>> I think we can already make people happy by adding the version to the
>> module-info file, assuming all other added features are actually nice to
>> haves.
>
> I guess you're right. We already have the version and main class[1] -
> I'll update the PR so it compiles and passes the tests.
>
> As for the version - the module version could be set by the compiler
> as well. I was thinking that maybe it would be nice if the compiler
> plugin passes the project version to the Java compiler.
>
> Regards,
> Plamen Totev
>
> [1] https://github.com/codehaus-plexus/plexus-archiver/pull/75
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Plamen Totev <pl...@gmail.com>.
Hi,
On Mon, Jan 15, 2018 at 8:23 PM, Robert Scholte <rf...@apache.org> wrote:
> So maybe we simply have to split it up into smaller pieces.
> I think we can already make people happy by adding the version to the
> module-info file, assuming all other added features are actually nice to
> haves.
I guess you're right. We already have the version and main class[1] -
I'll update the PR so it compiles and passes the tests.
As for the version - the module version could be set by the compiler
as well. I was thinking that maybe it would be nice if the compiler
plugin passes the project version to the Java compiler.
Regards,
Plamen Totev
[1] https://github.com/codehaus-plexus/plexus-archiver/pull/75
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Andreas Sewe <se...@st.informatik.tu-darmstadt.de>.
Robert Scholte wrote:
> Plexus Archiver is an archiver fully written in Java, so there has never
> been the need to add toolchain support.
>
> It would be very nice if this could be solved without the need of the
> external jartool or using it via a ToolProvider[1].
Also, if two separate tools modify the JAR, the goal of reproducible
builds is again a bit harder to accomplish. So, please try to use an
internal, written-in-Java solution if possible.
Just my 2 cents.
Best wishes,
Andreas
Re: main-class + module-version
Posted by Robert Scholte <rf...@apache.org>.
Plexus Archiver is an archiver fully written in Java, so there has never
been the need to add toolchain support.
It would be very nice if this could be solved without the need of the
external jartool or using it via a ToolProvider[1].
So maybe we simply have to split it up into smaller pieces.
I think we can already make people happy by adding the version to the
module-info file, assuming all other added features are actually nice to
haves.
thanks,
Robert
[1]https://docs.oracle.com/javase/9/docs/api/java/util/spi/ToolProvider.html
On Mon, 15 Jan 2018 18:12:40 +0100, Plamen Totev
<pl...@gmail.com> wrote:
> Hi,
>
> On 12/20/2017 9:53 PM, Robert Scholte wrote:
>> Based on this message it seems worth implementing a JarToolArchiver,
>> using the jar tool via the ToolProvider[1]
>> I hope it can still be a org.codehaus.plexus.archiver.Archiver,
>> otherwise I'll contact the openjdk team about the details of the
>> specifications. I don't think we need them all, good to know the reason
>> for the extra files.
>
> So I did some experiments and definitely it is possible to implement it
> as org.codehaus.plexus.archiver.Archiver. I think it would be best to
> reuse JarArchiver to create the non-modular JAR file and then use the
> JDK jar tool to update it to modular JAR file. The downside of this
> approach is that it involves additional work(there is some performance
> penalty). The JDK jar tool updates files the same as way Plexus Archiver
> - creates new file, copies the old entries and adds the new ones. Do you
> think this is really an issue? I think for small JAR files the
> difference would not be noticeable.
>
> Of course we could create the jar file directly using only the JDK jar
> tool. But the Plexus Archiver is quite advanced tool compared to it. If
> we have to implement all of its functionality using only the JDK jar
> tool, it would be easier to update the module descriptors using asm
> (IMHO).
>
> About the ToolProvider - maybe I'm missing something but it is available
> only for Java 9 and does not allow the use of tool chains, does it?
>
> Regards,
> Plamen Totev
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Plamen Totev <pl...@gmail.com>.
Hi,
On 12/20/2017 9:53 PM, Robert Scholte wrote:
>
> Based on this message it seems worth implementing a JarToolArchiver,
> using the jar tool via the ToolProvider[1]
> I hope it can still be a org.codehaus.plexus.archiver.Archiver,
> otherwise I'll contact the openjdk team about the details of the
> specifications. I don't think we need them all, good to know the reason
> for the extra files.
So I did some experiments and definitely it is possible to implement it
as org.codehaus.plexus.archiver.Archiver. I think it would be best to
reuse JarArchiver to create the non-modular JAR file and then use the
JDK jar tool to update it to modular JAR file. The downside of this
approach is that it involves additional work(there is some performance
penalty). The JDK jar tool updates files the same as way Plexus Archiver
- creates new file, copies the old entries and adds the new ones. Do you
think this is really an issue? I think for small JAR files the
difference would not be noticeable.
Of course we could create the jar file directly using only the JDK jar
tool. But the Plexus Archiver is quite advanced tool compared to it. If
we have to implement all of its functionality using only the JDK jar
tool, it would be easier to update the module descriptors using asm (IMHO).
About the ToolProvider - maybe I'm missing something but it is available
only for Java 9 and does not allow the use of tool chains, does it?
Regards,
Plamen Totev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: main-class + module-version
Posted by Robert Scholte <rf...@apache.org>.
On Wed, 20 Dec 2017 07:40:24 +0100, Plamen Totev
<pl...@gmail.com> wrote:
> Hi,
>
> I've started working on implementing support for modular JARs in
> Plexus Archiver but as I research the matter I found that they are
> more complex than I expected. I've updated the GitHub issue[1] with
> the details but I'll post the findings here as well. The modular JAR
> files contain not only version and main class but also:
>
> * List of the packages contained - usually populated by the compiler,
> but there are some caveats here. The most obvious one is that for jars
> created by the Shade Plugin this attribute will almost certainly be
> incorrect - the plugin will most likely introduce additional packages
> and the list produced by the compiler will be out of date. The other
> caveat are the resources - they could be located in packages and
> encapsulated the same way as the classes. The jar tools records not
> only the class packages but the resource ones too. Packaging resources
> together with the classes may result in incorrect module descriptor if
> this attribute is not updated.
>
> * The module main class - passed as argument to the jar tool and it
> seems that it cannot be set using the compiler.
>
> * The module version - passed as argument to the jar tool. Could be
> set using the compiler(again passed as argument) as well.
>
> * Hashes - calculates the hashes of given external modules that depend
> on this one. During runtime java checks the recorded hashes against
> the resolved module and and if don't match it will fail. The primary
> use case is a module A that exports packages to B and C. An easy way
> to break the encapsulation of A is to create module named B. To
> prevent that you can record the hash of B and C inside A. Then A will
> export the packages only to the intended modules. It is used by the
> OpenJDK but could be useful for platforms and frameworks as well.
>
> * Module resolution - marks the module to not be resolved by default or
> marks it as deprecated or incubating so at run time a warning is
> displayed if it is resolved. I don't think you can set it using the
> JDK 9 jar tool but it's present in the code. Maybe it will be
> implemented in future version of the jar tool.
>
> To me it looks like the packaging of modular JAR file is more complex
> than just packing module-info.class file and the jar tool is no longer
> just a ZIP program. Something similar is stated in the tool
> documentation[2]:
>
>> The jar command is a general-purpose archiving and compression tool,
>> based on the ZIP and ZLIB compression formats.
>> Initially, the jar command was designed to package Java applets
>> or applications; however, beginning with JDK 9, users can use the
>> jar command to create modular JARs.
>> For transportation and deployment, it’s usually more convenient to
>> package modules as modular JARs.
>
> While not at all that complex from technical point of view, I don't
> think it is worth implementing, and whats more maintaining, all this
> functionality. Probably it would be better if the plugins that produce
> JAR files use the jar tool, the same way the compiler plugin does not
> compile by itself but uses the Java compiler.
>
> What do you think?
Based on this message it seems worth implementing a JarToolArchiver, using
the jar tool via the ToolProvider[1]
I hope it can still be a org.codehaus.plexus.archiver.Archiver, otherwise
I'll contact the openjdk team about the details of the specifications. I
don't think we need them all, good to know the reason for the extra files.
thanks so far,
Robert
[1]
https://docs.oracle.com/javase/9/docs/api/jdk.jartool-summary.html#module.description
>
>
> [1]
> https://github.com/codehaus-plexus/plexus-archiver/issues/69#issuecomment-349095101
> [2] https://docs.oracle.com/javase/9/tools/jar.htm
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org