You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/01 09:31:06 UTC
[GitHub] [hadoop-ozone] elek opened a new pull request #744: Hdds 3312
elek opened a new pull request #744: Hdds 3312
URL: https://github.com/apache/hadoop-ozone/pull/744
## What changes were proposed in this pull request?
`hadoop-ozone/common` and `hadoop-hdds/common` projects are common between client and server. Therefore we should remove any server side utilities / dependencies from them as they would be added to the client classpath which makes harder the Ozone client / ozonefs adoption.
1. hadoop-hdds should depend on a minimal set of hadoop dependencies (can be managed a separated technical project / pom.xml)
2. code shared between server side projects (and not with the client) should be moved to the framework
3. OM related code should be moved to the ozone-manager instead of ozone/common
4. For technical reason some test utilities are imported to the Ozone code base (from Hadoop)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3312
## How was this patch tested?
Full integration test. This is a project organization patch without real change of the logic. If the clients can work without class-not-found exception. It should be fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #744: HDDS-3312.
Remove server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on issue #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744#issuecomment-608651601
Thanks for merging #701 first. In return I have merged `master` into this branch and resolved conflicts: https://github.com/adoroszlai/hadoop-ozone/commit/0a76d0cf0b02016962b4b32bb58f29fe4090f20f
Checkstyle still fails due to the new module `hadoop-hdds-hadoop-dependency-client`, fix is in progress in #768.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744#discussion_r402977363
##########
File path: hadoop-ozone/ozonefs-lib-current/pom.xml
##########
@@ -179,36 +179,36 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-ozone-filesystem</artifactId>
<scope>compile</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs-client</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.code.findbugs</groupId>
- <artifactId>jsr305</artifactId>
- </exclusion>
- </exclusions>
+<!-- <exclusions>-->
+<!-- <exclusion>-->
+<!-- <groupId>org.apache.hadoop</groupId>-->
+<!-- <artifactId>hadoop-common</artifactId>-->
+<!-- </exclusion>-->
Review comment:
These look strange.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #744: HDDS-3312. Remove
server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
elek commented on issue #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744#issuecomment-610278952
Merged to the master. Thanks the review (and the help with the rebase) @adoroszlai
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #744: HDDS-3312. Remove
server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
elek closed pull request #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #744: HDDS-3312.
Remove server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on issue #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744#issuecomment-608983034
Checkstyle problem is fixed on master.
Hadoop-related acceptance tests are failing:
```
hadoop31-hadoopo3fs :: Test ozone fs with hadoopfs
==============================================================================
Test hadoop dfs | FAIL |
1 != 0
```
due to
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/ozone/shaded/org/apache/http/client/utils/URIBuilder
at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:141)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
at org.apache.hadoop.fs.shell.CommandWithDestination.getRemoteDestination(CommandWithDestination.java:195)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processOptions(CopyCommands.java:259)
at org.apache.hadoop.fs.shell.Command.run(Command.java:175)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.ozone.shaded.org.apache.http.client.utils.URIBuilder
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
```
https://github.com/apache/hadoop-ozone/pull/744/checks?check_run_id=558258420
https://github.com/adoroszlai/hadoop-ozone/runs/559314792
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744#discussion_r402977774
##########
File path: hadoop-hdds/hadoop-dependency-client/README.txt
##########
@@ -0,0 +1,60 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This helper project provides a reduced view to the Hadoop dependencies. Hadoop common by default depends on a lot of other projects
+which are not required for the parts used in Ozone.
+
+The exclude rules should be updated on each version bump. If Hadoop introduce new dependencies we need to exclude them (unless they are required).
+
+
+For the reference here are the minimal jar files wich are required by Ozone:
+
+
+```
Review comment:
I think this file should be renamed to `README.md`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #744: HDDS-3312. Remove
server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
elek commented on issue #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744#issuecomment-608465309
> It will have a few conflicts with #701 so if you need to update this anyway, I'd appreciate if you could review my PR. :)
@adoroszlai: #701 is already conflicting. Do a rebase and I will merge it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #744:
HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #744: HDDS-3312. Remove server-side dependencies from hdds/ozone-common.
URL: https://github.com/apache/hadoop-ozone/pull/744#discussion_r403021558
##########
File path: hadoop-ozone/ozonefs-lib-current/pom.xml
##########
@@ -179,36 +179,36 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-ozone-filesystem</artifactId>
<scope>compile</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs-client</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.code.findbugs</groupId>
- <artifactId>jsr305</artifactId>
- </exclusion>
- </exclusions>
+<!-- <exclusions>-->
+<!-- <exclusion>-->
+<!-- <groupId>org.apache.hadoop</groupId>-->
+<!-- <artifactId>hadoop-common</artifactId>-->
+<!-- </exclusion>-->
Review comment:
Yes, I just noticed yesterday night. It's restored now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org