You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "melin (Jira)" <ji...@apache.org> on 2024/09/04 04:45:00 UTC
[jira] [Updated] (SPARK-49508) Optimized hadoop-aws dependency, aws-java-sdk-bundle jar is too large
[ https://issues.apache.org/jira/browse/SPARK-49508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
melin updated SPARK-49508:
--------------------------
Description:
aws-java-sdk-bundle jar is too large，The size of the spark image will double。hadoop aws only requires the use of aws-java-sdk-s3 and aws-java-sdk-dynamodb
{code:java}
// code placeholder
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>${awssdk.v1.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>${awssdk.v1.version}</version>
</dependency> {code}
was:
aws-java-sdk-bundle jar is too large，hadoop aws only requires the use of AWs-Java-SDK-S3 and AWs-java-SDK-Dynamodb
{code:java}
// code placeholder
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>${awssdk.v1.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>${awssdk.v1.version}</version>
</dependency> {code}
> Optimized hadoop-aws dependency, aws-java-sdk-bundle jar is too large
> ---------------------------------------------------------------------
>
> Key: SPARK-49508
> URL: https://issues.apache.org/jira/browse/SPARK-49508
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 4.0.0, 3.5.2
> Reporter: melin
> Priority: Major
>
> aws-java-sdk-bundle jar is too large，The size of the spark image will double。hadoop aws only requires the use of aws-java-sdk-s3 and aws-java-sdk-dynamodb
>
> {code:java}
> // code placeholder
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-aws</artifactId>
> <version>${hadoop.version}</version>
> <exclusions>
> <exclusion>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-bundle</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-s3</artifactId>
> <version>${awssdk.v1.version}</version>
> </dependency>
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-dynamodb</artifactId>
> <version>${awssdk.v1.version}</version>
> </dependency> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org