You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/09/01 19:33:39 UTC
[GitHub] [hadoop-thirdparty] pjfanning opened a new pull request, #21: [HADOOP-18342] shaded avro jar
pjfanning opened a new pull request, #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21
https://issues.apache.org/jira/browse/HADOOP-18342
This is based on latest avro jar (1.11.1). The shaded avro jar also contains a shaded copy of jackson (based on version avro needs). This is jackson 2.12.7 which matches what hadoop uses but I think it might be safer not to rely on avro and hadoop needing the same version of jackson and jackson upgrades can prove problematic for downstream projects.
I have omitted these jars from the shaded jar:
* slf4j-api
* commons-compress
These 2 are more stable and might be tidier not to completely bloat the avro shaded jar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1235398501
it's that because the parquet bit of the shaded jar has a version in it, upgrading is harder than just bumping up the build number, i need to change the module name etc too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] pjfanning commented on a diff in pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
pjfanning commented on code in PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#discussion_r1000937480
##########
hadoop-shaded-avro/pom.xml:
##########
@@ -0,0 +1,100 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hadoop-thirdparty</artifactId>
+ <groupId>org.apache.hadoop.thirdparty</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>hadoop-shaded-avro</artifactId>
+ <name>Apache Hadoop shaded Avro</name>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ <version>${avro.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <resources>
+ <resource>
+ <directory>${project.basedir}/..</directory>
+ <targetPath>META-INF</targetPath>
+ <includes>
+ <include>licenses-binary/*</include>
+ <include>NOTICE.txt</include>
+ <include>NOTICE-binary</include>
+ </includes>
+ <excludes>
+ <exclude>META-INF/maven/org.apache.avro/*</exclude>
+ </excludes>
+ </resource>
+ <resource>
+ <directory>${project.basedir}/src/main/resources</directory>
+ </resource>
+ </resources>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
+ <createDependencyReducedPom>true</createDependencyReducedPom>
+ </configuration>
+ <executions>
+ <execution>
+ <id>shade-avro</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <artifactSet>
+ <includes>
+ <include>org.apache.avro:avro</include>
+ </includes>
+ </artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>org/apache/avro</pattern>
+ <shadedPattern>${shaded.prefix}/avro</shadedPattern>
+ </relocation>
+ </relocations>
+ <transformers>
+ <transformer
+ implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
+ <resource>META-INF/LICENSE.txt</resource>
+ <file>${basedir}/../LICENSE-binary</file>
+ </transformer>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] pjfanning commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
pjfanning commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1235390690
Thanks @steveloughran - I'll look into changing the shaded avro jar and pom.xml as you suggest.
I'm not sure if I understand your point about parquet. Do you think parquet needs to be shaded as an extra hadoop-thirdparty jar? Or does hadoop use parquet-avro jar and if so, would we need to shade parquet-avro too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1288080069
this is in; we need to get the protobuf update in and it'll be good for a release. though we may also want to update guava, which maybe we can do by adding a guava artifact with a version...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1286929501
let's do that. yes, upgrading is harder, but it's a lot more obvious what is in use.
(the other strategy would be to include it in the version itself, but that'd just confuse people)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] pjfanning commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
pjfanning commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1236929076
@steveloughran I think it's easier to keep using the standard avro-maven-plugin and to postprocess the generated source to replace the org.apache.avro package names in the generated source. I have this approach working in a local hadoop build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] pjfanning commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
pjfanning commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1285924920
> do see #19 and discussion about whether artifacts need names. it may be good to use a version number here...
I can rename the module and jar to hadoop-shaded-avro_1_11 if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] pjfanning commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
pjfanning commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1261122891
@steveloughran could you review this? https://github.com/apache/hadoop/pull/4854 uses a temp jar I published myself but can be adjusted when this hadoop-thirdparty PR is merged and released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1235423266
no problem, great that you are working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1285907443
do see #19 and discussion about whether artifacts need names. it may be good to use a version number here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on a diff in pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#discussion_r1000914786
##########
hadoop-shaded-avro/pom.xml:
##########
@@ -0,0 +1,100 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hadoop-thirdparty</artifactId>
+ <groupId>org.apache.hadoop.thirdparty</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>hadoop-shaded-avro</artifactId>
+ <name>Apache Hadoop shaded Avro</name>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ <version>${avro.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <resources>
+ <resource>
+ <directory>${project.basedir}/..</directory>
+ <targetPath>META-INF</targetPath>
+ <includes>
+ <include>licenses-binary/*</include>
+ <include>NOTICE.txt</include>
+ <include>NOTICE-binary</include>
+ </includes>
+ <excludes>
+ <exclude>META-INF/maven/org.apache.avro/*</exclude>
+ </excludes>
+ </resource>
+ <resource>
+ <directory>${project.basedir}/src/main/resources</directory>
+ </resource>
+ </resources>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
+ <createDependencyReducedPom>true</createDependencyReducedPom>
+ </configuration>
+ <executions>
+ <execution>
+ <id>shade-avro</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <artifactSet>
+ <includes>
+ <include>org.apache.avro:avro</include>
+ </includes>
+ </artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>org/apache/avro</pattern>
+ <shadedPattern>${shaded.prefix}/avro</shadedPattern>
+ </relocation>
+ </relocations>
+ <transformers>
+ <transformer
+ implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
+ <resource>META-INF/LICENSE.txt</resource>
+ <file>${basedir}/../LICENSE-binary</file>
+ </transformer>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
Review Comment:
nit, add a newline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] pjfanning commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
pjfanning commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1236399887
@steveloughran I have made some progress with the hadoop-avro jar. I still need to test it in a hadoop build though. I have noticed that hadoop also uses avro-maven-plugin to compile generated classes. Do you think this will need to be shaded too? I'm guessing it should be ok to just use v1.11.1 of the actual avro-maven-plugin (to match the avro version shaded in the hadoop-avro jar).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1235381597
I'd prefer retaining jackson unshaded just to avoid the need to do another release of this *every time there's a jackson CVE emergency*
leaving out slf4j is good and flexible; though the dependencies of the shaded jar should declare that and compress as either provided or required.
now, the parquet lib includes the version. I'm not sure that is the right thing to do (as it complicates that update), but it would allow someone brave to have the different versions coexisting.
thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran commented on pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21#issuecomment-1285900443
lgtm; needs a newline at the end of the new pom.
+1 pending that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop-thirdparty] steveloughran merged pull request #21: [HADOOP-18342] shaded avro jar
Posted by GitBox <gi...@apache.org>.
steveloughran merged PR #21:
URL: https://github.com/apache/hadoop-thirdparty/pull/21
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org