You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Romain Manni-Bucau <rm...@gmail.com> on 2018/02/07 15:23:37 UTC
dependencies.txt in META-INF?
Hi guys,
I have a use case where I would resolve beam classpath programmatically. I
wonder if it would be possible to add in META-INF (or BEAM-INF, in the jar
is the main request ;)) a dependencies.txt (or other file) listing all the
mandatory dependencies. I'm mainly interested by the java sdk core module
but can be beneficial to others as well.
With maven it is just a matter of defining:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>${dependency-plugin.version}</version>
<executions>
<execution>
<id>create-META-INF/dependencies.txt</id>
<phase>prepare-package</phase>
<goals>
<goal>list</goal>
</goals>
<configuration>
<outputFile>${project.build.outputDirectory}/META-INF/dependencies.txt</outputFile>
</configuration>
</execution>
</executions>
</plugin>
with gradle it is a loop around a resolvedconfiguration which dumps the
artifacts in a maven format (group:name:type:version)
My interest of it being in beam is to be able to upgrade beam without
having to re-release these metadata.
Is it something the project could be interested in?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Re: dependencies.txt in META-INF?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Was too much abused by libs and not supported everywhere :(
Le 8 févr. 2018 22:39, "Lukasz Cwik" <lc...@google.com> a écrit :
> It is unfortunate that setting Class-Path is so broken.
>
> On Wed, Feb 7, 2018 at 10:55 PM, Romain Manni-Bucau <rmannibucau@gmail.com
> > wrote:
>
>> Not really:
>> 1. I need to have the gav
>> 2. Please never set Class-Path of the manifest. It leads to broken
>> runtime in most environments :(.
>>
>>
>> Le 8 févr. 2018 05:19, "Lukasz Cwik" <lc...@google.com> a écrit :
>>
>>> Looking at the Gradle shadow plugin, it seems like it is doing what you
>>> ask. Does this fit your usecase?
>>>
>>> From: http://imperceptiblethoughts.com/shadow/#configuring_the_run
>>> time_classpath
>>>
>>> Additionally, Shadow automatically configures the manifest of the
>>> shadowJar task to contain a Class-Path entry in the JAR manifest. The
>>> value of the Class-Path entry is the name of all dependencies resolved
>>> in the shadow configuration for the project.
>>>
>>> dependencies {
>>> shadow 'junit:junit:3.8.2'
>>> }
>>>
>>> Inspecting the META-INF/MANIFEST.MF entry in the JAR file will reveal
>>> the following attribute:
>>>
>>> Class-Path: junit-3.8.2.jar
>>>
>>>
>>>
>>> On Wed, Feb 7, 2018 at 9:27 AM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> 2018-02-07 18:21 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>>>
>>>>> What kinds of features would this enable within the Apache Beam SDK or
>>>>> allow for users to write (looking for some reason as to why this is not
>>>>> just a one off change to support a use case)?
>>>>>
>>>>
>>>> It allows to build a classpath and to rely on beam without requiring
>>>> maven to resolve the poms and it is way faster than resolving pom model. I
>>>> full ack it is a bit limit like case but it doesn't cost much to beam too
>>>> so thought I could ask before
>>>> doing something 100% custom.
>>>>
>>>>
>>>>> Would it list all the transitive dependencies?
>>>>>
>>>>
>>>> all runtime ones (= not test and provided ones - even if i can live
>>>> with it listing them all, I just don't see why it would)
>>>>
>>>>
>>>>>
>>>>> How would you test that it works?
>>>>>
>>>>
>>>> It is a maven plugin so not sure it requires a test in beam itself but
>>>> on my side I have some test for this kind of thing already running a server
>>>> from this kind of file typically.
>>>>
>>>>
>>>>>
>>>>> On Wed, Feb 7, 2018 at 7:23 AM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I have a use case where I would resolve beam classpath
>>>>>> programmatically. I wonder if it would be possible to add in META-INF (or
>>>>>> BEAM-INF, in the jar is the main request ;)) a dependencies.txt (or other
>>>>>> file) listing all the mandatory dependencies. I'm mainly interested by the
>>>>>> java sdk core module but can be beneficial to others as well.
>>>>>>
>>>>>> With maven it is just a matter of defining:
>>>>>>
>>>>>> <plugin>
>>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>>> <artifactId>maven-dependency-plugin</artifactId>
>>>>>> <version>${dependency-plugin.version}</version>
>>>>>> <executions>
>>>>>> <execution>
>>>>>> <id>create-META-INF/dependencies.txt</id>
>>>>>> <phase>prepare-package</phase>
>>>>>> <goals>
>>>>>> <goal>list</goal>
>>>>>> </goals>
>>>>>> <configuration>
>>>>>> <outputFile>${project.build.outputDirectory}/META-INF/dependencies.txt</outputFile>
>>>>>> </configuration>
>>>>>> </execution>
>>>>>> </executions>
>>>>>> </plugin>
>>>>>>
>>>>>> with gradle it is a loop around a resolvedconfiguration which dumps
>>>>>> the artifacts in a maven format (group:name:type:version)
>>>>>>
>>>>>> My interest of it being in beam is to be able to upgrade beam without
>>>>>> having to re-release these metadata.
>>>>>>
>>>>>> Is it something the project could be interested in?
>>>>>>
>>>>>> Romain Manni-Bucau
>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>
Re: dependencies.txt in META-INF?
Posted by Lukasz Cwik <lc...@google.com>.
It is unfortunate that setting Class-Path is so broken.
On Wed, Feb 7, 2018 at 10:55 PM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Not really:
> 1. I need to have the gav
> 2. Please never set Class-Path of the manifest. It leads to broken runtime
> in most environments :(.
>
>
> Le 8 févr. 2018 05:19, "Lukasz Cwik" <lc...@google.com> a écrit :
>
>> Looking at the Gradle shadow plugin, it seems like it is doing what you
>> ask. Does this fit your usecase?
>>
>> From: http://imperceptiblethoughts.com/shadow/#configuring_the_run
>> time_classpath
>>
>> Additionally, Shadow automatically configures the manifest of the
>> shadowJar task to contain a Class-Path entry in the JAR manifest. The
>> value of the Class-Path entry is the name of all dependencies resolved
>> in the shadow configuration for the project.
>>
>> dependencies {
>> shadow 'junit:junit:3.8.2'
>> }
>>
>> Inspecting the META-INF/MANIFEST.MF entry in the JAR file will reveal
>> the following attribute:
>>
>> Class-Path: junit-3.8.2.jar
>>
>>
>>
>> On Wed, Feb 7, 2018 at 9:27 AM, Romain Manni-Bucau <rmannibucau@gmail.com
>> > wrote:
>>
>>>
>>>
>>> 2018-02-07 18:21 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>>
>>>> What kinds of features would this enable within the Apache Beam SDK or
>>>> allow for users to write (looking for some reason as to why this is not
>>>> just a one off change to support a use case)?
>>>>
>>>
>>> It allows to build a classpath and to rely on beam without requiring
>>> maven to resolve the poms and it is way faster than resolving pom model. I
>>> full ack it is a bit limit like case but it doesn't cost much to beam too
>>> so thought I could ask before
>>> doing something 100% custom.
>>>
>>>
>>>> Would it list all the transitive dependencies?
>>>>
>>>
>>> all runtime ones (= not test and provided ones - even if i can live with
>>> it listing them all, I just don't see why it would)
>>>
>>>
>>>>
>>>> How would you test that it works?
>>>>
>>>
>>> It is a maven plugin so not sure it requires a test in beam itself but
>>> on my side I have some test for this kind of thing already running a server
>>> from this kind of file typically.
>>>
>>>
>>>>
>>>> On Wed, Feb 7, 2018 at 7:23 AM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I have a use case where I would resolve beam classpath
>>>>> programmatically. I wonder if it would be possible to add in META-INF (or
>>>>> BEAM-INF, in the jar is the main request ;)) a dependencies.txt (or other
>>>>> file) listing all the mandatory dependencies. I'm mainly interested by the
>>>>> java sdk core module but can be beneficial to others as well.
>>>>>
>>>>> With maven it is just a matter of defining:
>>>>>
>>>>> <plugin>
>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>> <artifactId>maven-dependency-plugin</artifactId>
>>>>> <version>${dependency-plugin.version}</version>
>>>>> <executions>
>>>>> <execution>
>>>>> <id>create-META-INF/dependencies.txt</id>
>>>>> <phase>prepare-package</phase>
>>>>> <goals>
>>>>> <goal>list</goal>
>>>>> </goals>
>>>>> <configuration>
>>>>> <outputFile>${project.build.outputDirectory}/META-INF/dependencies.txt</outputFile>
>>>>> </configuration>
>>>>> </execution>
>>>>> </executions>
>>>>> </plugin>
>>>>>
>>>>> with gradle it is a loop around a resolvedconfiguration which dumps
>>>>> the artifacts in a maven format (group:name:type:version)
>>>>>
>>>>> My interest of it being in beam is to be able to upgrade beam without
>>>>> having to re-release these metadata.
>>>>>
>>>>> Is it something the project could be interested in?
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>
>>>>
>>>
>>
Re: dependencies.txt in META-INF?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Not really:
1. I need to have the gav
2. Please never set Class-Path of the manifest. It leads to broken runtime
in most environments :(.
Le 8 févr. 2018 05:19, "Lukasz Cwik" <lc...@google.com> a écrit :
> Looking at the Gradle shadow plugin, it seems like it is doing what you
> ask. Does this fit your usecase?
>
> From: http://imperceptiblethoughts.com/shadow/#configuring_the_
> runtime_classpath
>
> Additionally, Shadow automatically configures the manifest of the
> shadowJar task to contain a Class-Path entry in the JAR manifest. The
> value of the Class-Path entry is the name of all dependencies resolved in
> the shadow configuration for the project.
>
> dependencies {
> shadow 'junit:junit:3.8.2'
> }
>
> Inspecting the META-INF/MANIFEST.MF entry in the JAR file will reveal the
> following attribute:
>
> Class-Path: junit-3.8.2.jar
>
>
>
> On Wed, Feb 7, 2018 at 9:27 AM, Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>>
>>
>> 2018-02-07 18:21 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>
>>> What kinds of features would this enable within the Apache Beam SDK or
>>> allow for users to write (looking for some reason as to why this is not
>>> just a one off change to support a use case)?
>>>
>>
>> It allows to build a classpath and to rely on beam without requiring
>> maven to resolve the poms and it is way faster than resolving pom model. I
>> full ack it is a bit limit like case but it doesn't cost much to beam too
>> so thought I could ask before
>> doing something 100% custom.
>>
>>
>>> Would it list all the transitive dependencies?
>>>
>>
>> all runtime ones (= not test and provided ones - even if i can live with
>> it listing them all, I just don't see why it would)
>>
>>
>>>
>>> How would you test that it works?
>>>
>>
>> It is a maven plugin so not sure it requires a test in beam itself but on
>> my side I have some test for this kind of thing already running a server
>> from this kind of file typically.
>>
>>
>>>
>>> On Wed, Feb 7, 2018 at 7:23 AM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I have a use case where I would resolve beam classpath
>>>> programmatically. I wonder if it would be possible to add in META-INF (or
>>>> BEAM-INF, in the jar is the main request ;)) a dependencies.txt (or other
>>>> file) listing all the mandatory dependencies. I'm mainly interested by the
>>>> java sdk core module but can be beneficial to others as well.
>>>>
>>>> With maven it is just a matter of defining:
>>>>
>>>> <plugin>
>>>> <groupId>org.apache.maven.plugins</groupId>
>>>> <artifactId>maven-dependency-plugin</artifactId>
>>>> <version>${dependency-plugin.version}</version>
>>>> <executions>
>>>> <execution>
>>>> <id>create-META-INF/dependencies.txt</id>
>>>> <phase>prepare-package</phase>
>>>> <goals>
>>>> <goal>list</goal>
>>>> </goals>
>>>> <configuration>
>>>> <outputFile>${project.build.outputDirectory}/META-INF/dependencies.txt</outputFile>
>>>> </configuration>
>>>> </execution>
>>>> </executions>
>>>> </plugin>
>>>>
>>>> with gradle it is a loop around a resolvedconfiguration which dumps the
>>>> artifacts in a maven format (group:name:type:version)
>>>>
>>>> My interest of it being in beam is to be able to upgrade beam without
>>>> having to re-release these metadata.
>>>>
>>>> Is it something the project could be interested in?
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>
>>>
>>
>
Re: dependencies.txt in META-INF?
Posted by Lukasz Cwik <lc...@google.com>.
Looking at the Gradle shadow plugin, it seems like it is doing what you
ask. Does this fit your usecase?
From:
http://imperceptiblethoughts.com/shadow/#configuring_the_runtime_classpath
Additionally, Shadow automatically configures the manifest of the
shadowJar task
to contain a Class-Path entry in the JAR manifest. The value of the
Class-Path entry is the name of all dependencies resolved in the
shadow configuration
for the project.
dependencies {
shadow 'junit:junit:3.8.2'
}
Inspecting the META-INF/MANIFEST.MF entry in the JAR file will reveal the
following attribute:
Class-Path: junit-3.8.2.jar
On Wed, Feb 7, 2018 at 9:27 AM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
>
>
> 2018-02-07 18:21 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>
>> What kinds of features would this enable within the Apache Beam SDK or
>> allow for users to write (looking for some reason as to why this is not
>> just a one off change to support a use case)?
>>
>
> It allows to build a classpath and to rely on beam without requiring maven
> to resolve the poms and it is way faster than resolving pom model. I full
> ack it is a bit limit like case but it doesn't cost much to beam too so
> thought I could ask before
> doing something 100% custom.
>
>
>> Would it list all the transitive dependencies?
>>
>
> all runtime ones (= not test and provided ones - even if i can live with
> it listing them all, I just don't see why it would)
>
>
>>
>> How would you test that it works?
>>
>
> It is a maven plugin so not sure it requires a test in beam itself but on
> my side I have some test for this kind of thing already running a server
> from this kind of file typically.
>
>
>>
>> On Wed, Feb 7, 2018 at 7:23 AM, Romain Manni-Bucau <rmannibucau@gmail.com
>> > wrote:
>>
>>> Hi guys,
>>>
>>> I have a use case where I would resolve beam classpath programmatically.
>>> I wonder if it would be possible to add in META-INF (or BEAM-INF, in the
>>> jar is the main request ;)) a dependencies.txt (or other file) listing all
>>> the mandatory dependencies. I'm mainly interested by the java sdk core
>>> module but can be beneficial to others as well.
>>>
>>> With maven it is just a matter of defining:
>>>
>>> <plugin>
>>> <groupId>org.apache.maven.plugins</groupId>
>>> <artifactId>maven-dependency-plugin</artifactId>
>>> <version>${dependency-plugin.version}</version>
>>> <executions>
>>> <execution>
>>> <id>create-META-INF/dependencies.txt</id>
>>> <phase>prepare-package</phase>
>>> <goals>
>>> <goal>list</goal>
>>> </goals>
>>> <configuration>
>>> <outputFile>${project.build.outputDirectory}/META-INF/dependencies.txt</outputFile>
>>> </configuration>
>>> </execution>
>>> </executions>
>>> </plugin>
>>>
>>> with gradle it is a loop around a resolvedconfiguration which dumps the
>>> artifacts in a maven format (group:name:type:version)
>>>
>>> My interest of it being in beam is to be able to upgrade beam without
>>> having to re-release these metadata.
>>>
>>> Is it something the project could be interested in?
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>
>>
>
Re: dependencies.txt in META-INF?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
2018-02-07 18:21 GMT+01:00 Lukasz Cwik <lc...@google.com>:
> What kinds of features would this enable within the Apache Beam SDK or
> allow for users to write (looking for some reason as to why this is not
> just a one off change to support a use case)?
>
It allows to build a classpath and to rely on beam without requiring maven
to resolve the poms and it is way faster than resolving pom model. I full
ack it is a bit limit like case but it doesn't cost much to beam too so
thought I could ask before
doing something 100% custom.
> Would it list all the transitive dependencies?
>
all runtime ones (= not test and provided ones - even if i can live with it
listing them all, I just don't see why it would)
>
> How would you test that it works?
>
It is a maven plugin so not sure it requires a test in beam itself but on
my side I have some test for this kind of thing already running a server
from this kind of file typically.
>
> On Wed, Feb 7, 2018 at 7:23 AM, Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> I have a use case where I would resolve beam classpath programmatically.
>> I wonder if it would be possible to add in META-INF (or BEAM-INF, in the
>> jar is the main request ;)) a dependencies.txt (or other file) listing all
>> the mandatory dependencies. I'm mainly interested by the java sdk core
>> module but can be beneficial to others as well.
>>
>> With maven it is just a matter of defining:
>>
>> <plugin>
>> <groupId>org.apache.maven.plugins</groupId>
>> <artifactId>maven-dependency-plugin</artifactId>
>> <version>${dependency-plugin.version}</version>
>> <executions>
>> <execution>
>> <id>create-META-INF/dependencies.txt</id>
>> <phase>prepare-package</phase>
>> <goals>
>> <goal>list</goal>
>> </goals>
>> <configuration>
>> <outputFile>${project.build.outputDirectory}/META-INF/dependencies.txt</outputFile>
>> </configuration>
>> </execution>
>> </executions>
>> </plugin>
>>
>> with gradle it is a loop around a resolvedconfiguration which dumps the
>> artifacts in a maven format (group:name:type:version)
>>
>> My interest of it being in beam is to be able to upgrade beam without
>> having to re-release these metadata.
>>
>> Is it something the project could be interested in?
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>
>
Re: dependencies.txt in META-INF?
Posted by Lukasz Cwik <lc...@google.com>.
What kinds of features would this enable within the Apache Beam SDK or
allow for users to write (looking for some reason as to why this is not
just a one off change to support a use case)?
Would it list all the transitive dependencies?
How would you test that it works?
On Wed, Feb 7, 2018 at 7:23 AM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Hi guys,
>
> I have a use case where I would resolve beam classpath programmatically. I
> wonder if it would be possible to add in META-INF (or BEAM-INF, in the jar
> is the main request ;)) a dependencies.txt (or other file) listing all the
> mandatory dependencies. I'm mainly interested by the java sdk core module
> but can be beneficial to others as well.
>
> With maven it is just a matter of defining:
>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-dependency-plugin</artifactId>
> <version>${dependency-plugin.version}</version>
> <executions>
> <execution>
> <id>create-META-INF/dependencies.txt</id>
> <phase>prepare-package</phase>
> <goals>
> <goal>list</goal>
> </goals>
> <configuration>
> <outputFile>${project.build.outputDirectory}/META-INF/dependencies.txt</outputFile>
> </configuration>
> </execution>
> </executions>
> </plugin>
>
> with gradle it is a loop around a resolvedconfiguration which dumps the
> artifacts in a maven format (group:name:type:version)
>
> My interest of it being in beam is to be able to upgrade beam without
> having to re-release these metadata.
>
> Is it something the project could be interested in?
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>