You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Srikanth Sampath <ss...@gmail.com> on 2016/08/26 13:59:19 UTC
Assembly build on spark 2.0.0
Hi,
mvn assembly is creating a .tgz distribution. How can I create a plain jar
archive? I would like to create a spark-assembly-<version>.jar
-Srikanth
Re: Assembly build on spark 2.0.0
Posted by Srikanth Sampath <ss...@gmail.com>.
Found the answer. This is the reason
https://issues.apache.org/jira/browse/SPARK-11157
-Srikanth
On Sat, Aug 27, 2016 at 8:54 AM, Srikanth Sampath <ssampath.apache@gmail.com
> wrote:
> Hi,
> Thanks Radek. However mvn package does not build the uber jar. I am
> looking for an uber jar and not a distribution. I have seen references to
> the uber jar here
> <http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_ig_running_spark_on_yarn.html>
>
> What I see in the spark 2.0 codeline (assembly/pom.xml) builds a
> distribution. I
>
> <profile>
>
> <id>bigtop-dist</id>
>
> <!-- This profile uses the assembly plugin to create a special
> "dist" package for BigTop
>
> that contains Spark but not the Hadoop JARs it depends on. -->
>
> <build>
>
> <plugins>
>
> <plugin>
>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-assembly-plugin</artifactId>
>
> <executions>
>
> <execution>
>
> <id>dist</id>
>
> <phase>package</phase>
>
> <goals>
>
> <goal>single</goal>
>
> </goals>
>
> <configuration>
>
> <descriptors>
>
> <descriptor>src/main/assembly/assembly.xml
> </descriptor>
>
> </descriptors>
>
> </configuration>
>
> </execution>
>
> </executions>
>
> </plugin>
>
> ... </plugins>
>
> </build>
>
> </profile>
>
>
> In src/main/assembly/assembly.xml we see
>
> <assembly>
>
> <id>dist</id>
>
> <formats>
>
> <format>tar.gz</format>
>
> <format>dir</format>
>
> </formats>
>
> <includeBaseDirectory>false</includeBaseDirectory>
>
> .....
>
>
>
> On Sat, Aug 27, 2016 at 1:02 AM, Radoslaw Gruchalski <radek@gruchalski.com
> > wrote:
>
>> mvn package might be the command you’re looking for.
>>
>> –
>> Best regards,
>> Radek Gruchalski
>> radek@gruchalski.com
>>
>>
>> On August 26, 2016 at 3:59:24 PM, Srikanth Sampath (
>> ssampath.apache@gmail.com) wrote:
>>
>> Hi,
>> mvn assembly is creating a .tgz distribution. How can I create a plain
>> jar archive? I would like to create a spark-assembly-<version>.jar
>> -Srikanth
>>
>>
>
Re: Assembly build on spark 2.0.0
Posted by Radoslaw Gruchalski <ra...@gruchalski.com>.
Ah, an uberjar. Normally one would build the uberjar with a Maven Shade plugin. Haven't looked into Spark code much recently, it wouldn't make much sense having a separate maven command to build an uberjar while building a distribution because, from memory, if you open the tgz file, the uberjar sits in the lib directory.
--
Best regards,
Rad
_____________________________
From: Srikanth Sampath <ss...@gmail.com>
Sent: Saturday, August 27, 2016 5:24 am
Subject: Re: Assembly build on spark 2.0.0
To: Radoslaw Gruchalski <ra...@gruchalski.com>
Cc: <de...@spark.apache.org>
Hi,Thanks Radek. However mvn package does not build the uber jar. I am looking for an uber jar and not a distribution. I have seen references to the uber jar here What I see in the spark 2.0 codeline (assembly/pom.xml) builds a distribution. I
<profile>
<id>bigtop-dist</id>
<!-- This profile uses the assembly plugin to create a special "dist" package for BigTop
that contains Spark but not the Hadoop JARs it depends on. -->
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>dist</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
<descriptor>src/main/assembly/assembly.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>
... </plugins>
</build>
</profile>
In src/main/assembly/assembly.xml we see
<assembly>
<id>dist</id>
<formats>
<format>tar.gz</format>
<format>dir</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
.....
On Sat, Aug 27, 2016 at 1:02 AM, Radoslaw Gruchalski <ra...@gruchalski.com> wrote:
mvn package might be the command you’re looking for.
–
Best regards,
RadekGruchalski
radek@gruchalski.com
On August 26, 2016 at 3:59:24 PM, Srikanth Sampath (ssampath.apache@gmail.com) wrote: Hi,mvn assembly is creating a .tgz distribution. How can Icreate a plain jar archive? I would like to create aspark-assembly-<version>.jar-Srikanth
Re: Assembly build on spark 2.0.0
Posted by Srikanth Sampath <ss...@gmail.com>.
Hi,
Thanks Radek. However mvn package does not build the uber jar. I am
looking for an uber jar and not a distribution. I have seen references to
the uber jar here
<http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_ig_running_spark_on_yarn.html>
What I see in the spark 2.0 codeline (assembly/pom.xml) builds a
distribution. I
<profile>
<id>bigtop-dist</id>
<!-- This profile uses the assembly plugin to create a special "dist"
package for BigTop
that contains Spark but not the Hadoop JARs it depends on. -->
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>dist</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
<descriptor>src/main/assembly/assembly.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>
... </plugins>
</build>
</profile>
In src/main/assembly/assembly.xml we see
<assembly>
<id>dist</id>
<formats>
<format>tar.gz</format>
<format>dir</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
.....
On Sat, Aug 27, 2016 at 1:02 AM, Radoslaw Gruchalski <ra...@gruchalski.com>
wrote:
> mvn package might be the command you’re looking for.
>
> –
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com
>
>
> On August 26, 2016 at 3:59:24 PM, Srikanth Sampath (
> ssampath.apache@gmail.com) wrote:
>
> Hi,
> mvn assembly is creating a .tgz distribution. How can I create a plain
> jar archive? I would like to create a spark-assembly-<version>.jar
> -Srikanth
>
>
Re: Assembly build on spark 2.0.0
Posted by Radoslaw Gruchalski <ra...@gruchalski.com>.
mvn package might be the command you’re looking for.
–
Best regards,
Radek Gruchalski
radek@gruchalski.com
On August 26, 2016 at 3:59:24 PM, Srikanth Sampath (
ssampath.apache@gmail.com) wrote:
Hi,
mvn assembly is creating a .tgz distribution. How can I create a plain jar
archive? I would like to create a spark-assembly-<version>.jar
-Srikanth