You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2019/12/17 23:13:37 UTC
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #342: HDDS-2709.
Maven property skipShade should not skip ozonefs compilation
bharatviswa504 commented on issue #342: HDDS-2709. Maven property skipShade should not skip ozonefs compilation
URL: https://github.com/apache/hadoop-ozone/pull/342#issuecomment-566791654
> > As I understood we use either legacy or current jar files for ozone file system.
>
> Hi @elek , this makes sense, but I'm not sure if that's always the case . At a time I was debugging HDDS-2665 from a `docker-compose` cluster:
>
> ```
> cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> docker-compose up -d
> docker-compose exec om /bin/bash
> # Run some ozone shell commands
> ```
>
> So I tried to speed up the compilation with `-DskipShade` but found out this would skip out compiling `hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar`. (I made some change under `hadoop-ozone/ozonefs`, for example `BasicOzoneFileSystem` class.)
>
> Then after I move `ozonefs` module out of `skipShade` scope (i.e. the change in this PR), run the same mvn command with `-DskipShade` **does refresh** the `hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar`. My full mvn command:
>
> ```
> mvn clean install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true -DskipRecon -DskipShade
> ```
>
> This might have something to do with `clean`?
>
> It seems `ozone fs` is still using the unshaded jar `hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar`? I ran the command below in the `docker-compose` cluster after compiling without `-skipShade`, so it would have the shaded jars `hadoop-ozone-filesystem-lib-current-0.5.0-SNAPSHOT.jar` and `hadoop-ozone-filesystem-lib-legacy-0.5.0-SNAPSHOT.jar`:
>
> ```
> bash-4.2$ HADOOP_ROOT_LOGGER=TRACE,console ozone fs -ls o3fs://bucket1.vol1/
> ...
> 2019-12-13 23:04:39,327 DEBUG fs.FileSystem: o3fs:// = class org.apache.hadoop.fs.ozone.OzoneFileSystem from /opt/hadoop/share/ozone/lib/hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar
> 2019-12-13 23:04:39,327 DEBUG fs.FileSystem: Looking for FS supporting o3fs
> 2019-12-13 23:04:39,327 DEBUG fs.FileSystem: looking for configuration option fs.o3fs.impl
> 2019-12-13 23:04:39,366 DEBUG fs.FileSystem: Looking in service filesystems for implementation class
> 2019-12-13 23:04:39,366 DEBUG fs.FileSystem: FS for o3fs is class org.apache.hadoop.fs.ozone.OzoneFileSystem
> 2019-12-13 23:04:39,382 TRACE ozone.BasicOzoneFileSystem: Ozone URI for ozfs initialization is o3fs://bucket1.vol1
> ```
>
> and I can confirm both shaded jars are in the path:
>
> ```
> bash-4.2$ ls -lah /opt/hadoop/share/ozone/lib/hadoop-ozone-filesystem-*
> -rw-r--r-- 1 hadoop users 90K Dec 5 19:44 /opt/hadoop/share/ozone/lib/hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar
> -rw-r--r-- 1 hadoop users 58M Dec 5 19:44 /opt/hadoop/share/ozone/lib/hadoop-ozone-filesystem-lib-current-0.5.0-SNAPSHOT.jar
> -rw-r--r-- 1 hadoop users 100M Dec 5 19:44 /opt/hadoop/share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-0.5.0-SNAPSHOT.jar
> ```
Thanks for the clear info @smengcl. I figured out reason for picking hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar. This is picking hadoop-ozone-filesystem jar because when ozone fs command runs the classpath used is hadoop-ozone-tools, which has hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar. So, I see a benefit of compiling ozonefs by default and skipShade to skip legacy and current fat jar skipping.
$HDDS_LIB_JARS_DIR/hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org