You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Rick Hillegas (JIRA)" <ji...@apache.org> on 2018/04/01 15:39:00 UTC
[jira] [Comment Edited] (DERBY-6945) Re-package Derby as a
collection of jigsaw modules
[ https://issues.apache.org/jira/browse/DERBY-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421693#comment-16421693 ]
Rick Hillegas edited comment on DERBY-6945 at 4/1/18 3:38 PM:
--------------------------------------------------------------
Attaching derby-6945-25-ab-moduleDescriptors.diff. This patch, at last, adds the first rev of module descriptors. Tests passed cleanly for me on this patch. The patch adds module descriptors for the production jars, but not for derbyTesting.jar. When I tried to add a module descriptor to derbyTesting.jar, I discovered that the tests share packages with the product, perhaps in order to test package-protected behavior. I will need to untangle this mess.
I am a little disturbed by all of the internals which are exposed to the testing jar. Maybe the experts can suggest another way to test internal apis without bloating product jars with unnecessary code. In particular, I am a little worried that a blackhat could gain access to internal apis by putting malware into a module called org.apache.derby.tests. However, that vulnerability assumes that the blackhat can subvert the production module-path. I suppose that practically any kind of mischief becomes possible for a blackhat who can subvert the module-path.
I am also unsure about how to represent the optional dependency of the engine jar on OSGI. I get the impression that it will take some time before OSGI harmonizes with the Java Package Manager. In the meantime, I have added a dummy module-info to the stub classes which we borrowed from Apache Felix. (Derby's OSGI bundling is a moribund effort at best. I don't think that the engine manifest even complies with current OSGI best practices.)
In crafting the module descriptors, I found the following resources useful:
a) A helpful overview of module jargon: https://labs.consol.de/development/2017/02/13/getting-started-with-java9-modules.html
b) A helpful overview of the directives which appear in module descriptors: https://labs.consol.de/development/2017/02/13/getting-started-with-java9-modules.html
c) Mark Reinhold's overview: http://openjdk.java.net/projects/jigsaw/spec/sotms/
d) Stephen Colbourne's explanation of the automatic module: https://labs.consol.de/development/2017/02/13/getting-started-with-java9-modules.html
I have settled on the following module names, but please feel free to advocate for better names. I thought that all of the names should be nouns. Modules are listed in reverse-dependency order. The engine and the client jars are peers. I will draw a pretty picture after I have untangled derbyTesting.jar.
{noformat}
|---------------------------------|-------------------------|
| MODULE NAME | JAR FILE |
|---------------------------------|-------------------------|
| org.apache.derby.commons | derbyshared.jar |
|---------------------------------|-------------------------|
| org.apache.derby.engine | derby.jar |
|---------------------------------|-------------------------|
| org.apache.derby.client | derbyclient.jar |
|---------------------------------|-------------------------|
| org.apache.derby.tools | derbytools.jar |
|---------------------------------|-------------------------|
| org.apache.derby.optionaltools | derbyoptionaltools.jar |
|---------------------------------|-------------------------|
| org.apache.derby.server | derbynet.jar |
|---------------------------------|-------------------------|
| org.apache.derby.runner | derbyrun.jar |
|---------------------------------|-------------------------|
{noformat}
We will want to consult the experts on the following points:
1) Is there a better way to represent test dependencies?
2) Is there a better way to model the OSGI dependency?
Touches the following files:
{noformat}
--------------------------------------------
A java/client/module-info.java
A java/drda/module-info.java
A java/engine/module-info.java
A java/optional/module-info.java
A java/run/module-info.java
A java/shared/module-info.java
A java/tools/module-info.java
Module descriptors.
--------------------------------------------
M java/client/build.xml
M java/drda/build.xml
M java/engine/build.xml
M java/optional/build.xml
M java/run/build.xml
M java/tools/build.xml
Change the compilation classpaths into compilation module-paths. There was no need to touched java/shared/build.xml because its compilation target doesn't have a classpath to begin with.
--------------------------------------------
M build.xml
A java/stubs/felix/module-info.java
Create a dummy module descriptor for the OSGI stubs. Add the OSGI module to the module-path used by the class size generator.
--------------------------------------------
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derby.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyTesting.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyclient.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbynet.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyoptionaltools.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyrun.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbytools.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derby.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbyclient.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbynet.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbyoptionaltools.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbyrun.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbytools.jar.lastcontents
Account for the new module descriptors which appear in the jar files now.
--------------------------------------------
M java/testing/org/apache/derbyTesting/functionTests/tests/lang/VetJigsawTest.java
Ignore the module descriptors when looking for package overlap across the jar files.
{noformat}
was (Author: rhillegas):
Attaching derby-6945-25-ab-moduleDescriptors.diff. This patch, at last, adds the first rev of module descriptors. Tests passed cleanly for me on this patch. The patch adds module descriptors for the production jars, but not for derbyTesting.jar. When I tried to add a module descriptor to derbyTesting.jar, I discovered that the tests share packages with the product, perhaps in order to test package-protected behavior. I will need to untangle this mess.
I am a little disturbed by all of the internals which are exposed to the testing jar. Maybe the experts can suggest another way to test internal apis without bloating product jars with unnecessary code. In particular, I am a little worried that a blackhat could gain access to internal apis by putting malware into a module called org.apache.derby.tests. However, that vulnerability assumes that the blackhat can subvert the production module-path. I suppose that practically any kind of mischief becomes possible for a blackhat who can subvert the module-path.
I am also unsure about how to represent the optional dependency of the engine jar on OSGI. I get the impression that it will take some time before OSGI harmonizes with the Java Package Manager. In the meantime, I have added a dummy module-info to the stub classes which we borrowed from Apache Felix. (Derby's OSGI bundling is a moribund effort at best. I don't think that the engine manifest even complies with current OSGI best practices.)
In crafting the module descriptors, I found the following resources useful:
a) A helpful overview of module jargon: https://labs.consol.de/development/2017/02/13/getting-started-with-java9-modules.html
b) A helpful overview of the directives which appear in module descriptors: https://labs.consol.de/development/2017/02/13/getting-started-with-java9-modules.html
c) Mark Reinhold's overview: http://openjdk.java.net/projects/jigsaw/spec/sotms/
d) Stephen Colbourne's explanation of the automatic module: https://labs.consol.de/development/2017/02/13/getting-started-with-java9-modules.html
I have settled on the following module names, but please feel free to advocate for better names. I thought that all of the names should be nouns. Modules are listed in reverse-dependency order. The engine and the client jars are peers. I will draw a pretty picture after I have untangled derbyTesting.jar.
{noformat}
|---------------------------------|-------------------------|
| MODULE NAME | JAR FILE |
|---------------------------------|-------------------------|
| org.apache.derby.commons | derbyshared.jar |
|---------------------------------|-------------------------|
| org.apache.derby.engine | derby.jar |
|---------------------------------|-------------------------|
| org.apache.derby.client | derbyclient.jar |
|---------------------------------|-------------------------|
| org.apache.derby.tools | derbytools.jar |
|---------------------------------|-------------------------|
| org.apache.derby.optionaltools | derbyoptionaltools.jar |
|---------------------------------|-------------------------|
| org.apache.derby.server | derbynet.jar |
|---------------------------------|-------------------------|
| org.apache.derby.runner | derbyrun.jar |
|---------------------------------|-------------------------|
{noformat}
We will want to consult the experts on the following points:
1) Is there a better way to represent test dependencies?
2) Is there a better way to model the OSGI dependency?
Touches the following files:
{noformat}
--------------------------------------------
A java/client/module-info.java
A java/drda/module-info.java
A java/engine/module-info.java
A java/optional/module-info.java
A java/run/module-info.java
A java/shared/module-info.java
A java/tools/module-info.java
Module descriptors.
--------------------------------------------
M java/client/build.xml
M java/drda/build.xml
M java/engine/build.xml
M java/optional/build.xml
M java/run/build.xml
M java/tools/build.xml
Change the compilation classpaths into compilation module-paths. There was no need to touched java/shared/build.xml because its compilation target doesn't have a classpath to begin with.
--------------------------------------------
M build.xml
A java/stubs/felix/module-info.java
Create a dummy module descriptor for the OSGI stubs. Add the OSGI module to the module-path used by the class size generator.
--------------------------------------------
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derby.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyTesting.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyclient.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbynet.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyoptionaltools.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbyrun.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/insane.derbytools.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derby.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbyclient.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbynet.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbyoptionaltools.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbyrun.jar.lastcontents
M java/build/org/apache/derbyBuild/lastgoodjarcontents/sane.derbytools.jar.lastcontents
Account for the new module descriptors which appear in the jar files now.
--------------------------------------------
M java/testing/org/apache/derbyTesting/functionTests/tests/lang/VetJigsawTest.java
Ignore the module descriptors when looking for package overlap across the jar files.
{noformat}
> Re-package Derby as a collection of jigsaw modules
> --------------------------------------------------
>
> Key: DERBY-6945
> URL: https://issues.apache.org/jira/browse/DERBY-6945
> Project: Derby
> Issue Type: Improvement
> Affects Versions: 10.13.1.2
> Reporter: Rick Hillegas
> Priority: Major
> Attachments: derby-6945-01-aa-remove_derbyPreBuild_dep.diff, derby-6945-02-ab-newDerbySharedJar.diff, derby-6945-02-ac-newDerbySharedJar.diff, derby-6945-03-aa-partitionTest.diff, derby-6945-04-aa-moveRunClass.diff, derby-6945-05-aa-removeRedundant_Attribute_SQLState.diff, derby-6945-06-aa-removeOtherSharedDuplicates.diff, derby-6945-07-aa-net_client_overlap.diff, derby-6945-08-aa-move_shared_iapi_under_shared.diff, derby-6945-08-ab-move_shared_iapi_under_shared.diff, derby-6945-08-ad-move_shared_iapi_under_shared.diff, derby-6945-09-ab-moveInternalDriver.diff, derby-6945-10-aa-moveDriver42.diff, derby-6945-11-ab-moveAutoloadedDriver.diff, derby-6945-12-ab-moveClientDataSourceInterface.diff, derby-6945-13-aa-create-ClientAutoloadedDriver.diff, derby-6945-14-aa-moveMoreEmbeddedClassesOutOfPublicAPI.diff, derby-6945-15-aa-moveMoreClientClassesOutOfPublicAPI.diff, derby-6945-16-aa-consolidatePublicAPI.diff, derby-6945-17-aa-resealPackagesIn-derbyshared.diff, derby-6945-18-aa-resealLocalizationPackage.diff, derby-6945-19-ad-movePublicAPIintoToolsJar.diff, derby-6945-20-aa-removeClasslister.diff, derby-6945-21-aa-simplifyBuildScripts.diff, derby-6945-21-ab-simplifyBuildScripts.diff, derby-6945-22-ad-isolateClassesByArtifact.diff, derby-6945-22-ae-isolateClassesByArtifact.diff, derby-6945-22-af-isolateClassesByArtifact.diff, derby-6945-23-aa-upgradeToAnt_1.10.2.diff, derby-6945-24-aa-fixNativeAuthenticationServiceTest.diff, derby-6945-25-ab-moduleDescriptors.diff, derby-6945-XX-moveDataSourceFactories, jdeps.out.tar, weirdAnt.out
>
>
> Once we commit to building with Java 9 (see DERBY-6856), we should consider re-packaging Derby as a set of jigsaw modules. This would result in a different set of release artifacts. This might be a good opportunity to address the Tomcat artifactory issues raised by issue DERBY-6944.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)