You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ChenSammi (via GitHub)" <gi...@apache.org> on 2026/05/19 05:05:46 UTC
[PR] HDDS-15318. Download libhadoop library during maven build [ozone]
ChenSammi opened a new pull request, #10305:
URL: https://github.com/apache/ozone/pull/10305
## What changes were proposed in this pull request?
download libhadoop library during maven build, for local unit test, and dist package.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15318
## How was this patch tested?
Test locally.
```
mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pnative-mac -Pdist
sammi@X3HF9K66K7 target % find . -name libhadoop*
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.dylib
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.1.0.0.dylib
sammi@X3HF9K66K7 target % tar -tvf ozone-2.2.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x 0 sammi staff 0 May 18 20:30 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.dylib -> libhadoop.1.0.0.dylib
-rw-r--r-- 0 sammi staff 157368 May 18 20:30 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.1.0.0.dylib
```
```
mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pnative -Pdist
sammi@X3HF9K66K7 target % find . -name libhadoop*
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
sammi@X3HF9K66K7 target % tar -tvf ozone-2.2.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x 0 sammi staff 0 May 19 12:02 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x 0 sammi staff 890256 May 19 12:02 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
```
```
mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pdist
sammi@X3HF9K66K7 target % find . -name libhadoop*
sammi@X3HF9K66K7 target % tar -tvf ozone-2.2.0-SNAPSHOT.tar.gz| grep libhadoop
```
The downloaded file in cached in maven local repo, so only download once if it's not changed.
```
sammi@X3HF9K66K7 target % ls -al ~/.m2/repository/.cache/download-maven-plugin
total 1006024
drwxr-xr-x 5 sammi staff 160 May 18 19:47 .
drwxr-xr-x 3 sammi staff 96 May 18 10:56 ..
-rw-r--r-- 1 sammi staff 514917037 May 18 11:50 hadoop-3.4.3.tar.gz_0deb1bfef085d775400e262f41867f8e
-rw-r--r-- 1 sammi staff 478 May 18 19:47 index.ser
-rw-r--r-- 1 sammi staff 157368 May 18 19:47 libhadoop.1.0.0.dylib_1aa24cad2f558d1e1012dfd83037efc1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10305:
URL: https://github.com/apache/ozone/pull/10305#issuecomment-4495311996
> @ChenSammi thanks for updating the patch
>
> Since download happens only when `native`/`native-mac` profile is enabled, this is not tested by CI. I still think we should download _all_ libs in `dist` profile (i.e. I suggest removing new profiles). Linking platform-specific lib should happen at runtime (remove existing link if exists and does not match platform), in `ozone` or before unit/integration tests (`process-test-resources` phase?) if you want to use it in JUnit tests.
Can I keep the current two profiles in this patch since it works correctly now?
I will try to do the suggested refactor once I move on to the unit test patch to see if it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264690274
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
Review Comment:
Version (newer one) already defined.
```suggestion
```
It's this version on feature branch. Next time, when merge master, it will be updated with master branch version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265117426
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
What do you mean "it should be platform-agnostic", can you elaborate it a bit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265378735
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean? Or what you mean is we have two dist ozone-2.1.0-SNAPSHOT directory, for example ozone-2.1.0-SNAPSHOT-x86-64, and ozone-2.1.0-SNAPSHOT-aarch64?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10305:
URL: https://github.com/apache/ozone/pull/10305#issuecomment-4490514292
@ChenSammi thanks for updating the patch
Since download happens only when `native`/`native-mac` profile is enabled, this is not tested by CI. I still think we should download _all_ libs in `dist` profile (i.e. I suggest removing new profiles). Linking platform-specific lib should happen at runtime (remove existing link if exists and does not match platform), in `ozone` or before unit/integration tests (`process-test-resources` phase?) if you want to use it in JUnit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264112601
##########
hadoop-ozone/dist/dev-support/bin/dist-layout-stitching:
##########
@@ -143,6 +143,24 @@ for file in $(find "${ROOT}" -path '*/share/ozone/lib/*jar' | sort); do
cp -p "$file" share/ozone/lib/
done
+# ---------------------------------------------------------
+# Copy Hadoop Native Libraries (libhadoop.so) - Conditionally
+# ---------------------------------------------------------
+NATIVE_LIBS_DIR="${ROOT}/target/native-lib"
+
+# Check if the .so files actually exist before attempting to copy
Review Comment:
these can be .dylib files too, right?
```suggestion
# Check if the libhadoop files actually exist before attempting to copy
```
##########
hadoop-ozone/dist/dev-support/bin/dist-layout-stitching:
##########
@@ -143,6 +143,24 @@ for file in $(find "${ROOT}" -path '*/share/ozone/lib/*jar' | sort); do
cp -p "$file" share/ozone/lib/
done
+# ---------------------------------------------------------
+# Copy Hadoop Native Libraries (libhadoop.so) - Conditionally
+# ---------------------------------------------------------
+NATIVE_LIBS_DIR="${ROOT}/target/native-lib"
+
+# Check if the .so files actually exist before attempting to copy
+if ls "${NATIVE_LIBS_DIR}"/libhadoop.* 1> /dev/null 2>&1; then
+ echo "Found Hadoop native libraries. Copying to distribution..."
+
+ # Create the native directory in the final staging area
+ run mkdir -p ./lib/native
+
+ # Copy the files and symlinks safely
+ run cp -rP "${NATIVE_LIBS_DIR}/"libhadoop.* ./lib/native
Review Comment:
will the ozone-env.sh script (or docker-config for our Docker deployments) need to set NATIVE_LIBS_DIR to lib/native for ozone to pick up libhadoop from this path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3267544885
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
```
mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pdist,native,native-mac
sammi@X3HF9K66K7 target % find . -name libhadoop*
./ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.dylib
./ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.1.0.0.dylib
./ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so
./ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
sammi@X3HF9K66K7 target % tar -tvf ozone-2.1.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x 0 sammi staff 0 May 19 22:02 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.dylib -> libhadoop.1.0.0.dylib
-rw-r--r-- 0 sammi staff 157368 May 19 22:02 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.1.0.0.dylib
lrwxr-xr-x 0 sammi staff 0 May 19 22:02 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x 0 sammi staff 890256 May 19 22:02 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
sammi@X3HF9K66K7 target % ls -al ./ozone-2.1.0-SNAPSHOT/lib/native/
total 2056
drwxr-xr-x 6 sammi staff 192 May 19 23:17 .
drwxr-xr-x 3 sammi staff 96 May 19 23:17 ..
lrwxr-xr-x 1 sammi staff 28 May 19 23:17 libhadoop.dylib -> libhadoop_osx_aarch_64.dylib
lrwxr-xr-x 1 sammi staff 25 May 19 23:17 libhadoop.so -> libhadoop_linux_x86_64.so
-rw-r--r-- 1 sammi staff 890256 May 19 23:17 libhadoop_linux_x86_64.so
-rw-r--r-- 1 sammi staff 157368 May 19 23:17 libhadoop_osx_aarch_64.dylib
sammi@X3HF9K66K7 target % tar -tvf ozone-2.1.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x 0 sammi staff 0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.dylib -> libhadoop_osx_aarch_64.dylib
-rw-r--r-- 0 sammi staff 157368 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_osx_aarch_64.dylib
-rw-r--r-- 0 sammi staff 890256 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_linux_x86_64.so
lrwxr-xr-x 0 sammi staff 0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so -> libhadoop_linux_x86_64.so
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264834454
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <id>fetch-hadoop-native-tar</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <url>https://archive.apache.org/dist/hadoop/common/hadoop-${hadoop.version}/hadoop-${hadoop.version}.tar.gz</url>
Review Comment:
It only download once for every hadoop version locally. Maven will cache it. See above "How was this patch tested" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10305:
URL: https://github.com/apache/ozone/pull/10305#issuecomment-4496182629
Thanks @jojochuang @adoroszlai for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #10305:
URL: https://github.com/apache/ozone/pull/10305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265086137
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
Review Comment:
It's this version on feature branch. Next time, when merge master, it will be updated with master branch version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264690274
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
Review Comment:
Version (newer one) already defined.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265117426
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
What "it should be platform-agnostic" specific mean, can you elaborate it a bit more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3267544885
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
```
mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pdist,native,native-mac
sammi@X3HF9K66K7 target % ls -al ./ozone-2.1.0-SNAPSHOT/lib/native/
total 2056
drwxr-xr-x 6 sammi staff 192 May 19 23:17 .
drwxr-xr-x 3 sammi staff 96 May 19 23:17 ..
lrwxr-xr-x 1 sammi staff 28 May 19 23:17 libhadoop.dylib -> libhadoop_osx_aarch_64.dylib
lrwxr-xr-x 1 sammi staff 25 May 19 23:17 libhadoop.so -> libhadoop_linux_x86_64.so
-rw-r--r-- 1 sammi staff 890256 May 19 23:17 libhadoop_linux_x86_64.so
-rw-r--r-- 1 sammi staff 157368 May 19 23:17 libhadoop_osx_aarch_64.dylib
sammi@X3HF9K66K7 target % tar -tvf ozone-2.1.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x 0 sammi staff 0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.dylib -> libhadoop_osx_aarch_64.dylib
-rw-r--r-- 0 sammi staff 157368 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_osx_aarch_64.dylib
-rw-r--r-- 0 sammi staff 890256 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_linux_x86_64.so
lrwxr-xr-x 0 sammi staff 0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so -> libhadoop_linux_x86_64.so
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10305:
URL: https://github.com/apache/ozone/pull/10305#issuecomment-4495261751
> BTW, I think the feature branch should be updated from `master`. `compile (8)` check is timing out because it uses outdated `macos-13` runner, which is no longer supported by GitHub ([HDDS-13732](https://issues.apache.org/jira/browse/HDDS-13732)).
I plan to merge master after this one is committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264687933
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <id>fetch-hadoop-native-tar</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <url>https://archive.apache.org/dist/hadoop/common/hadoop-${hadoop.version}/hadoop-${hadoop.version}.tar.gz</url>
+ <unpack>true</unpack>
+ <outputDirectory>${project.build.directory}/hadoop-bin</outputDirectory>
+ <skipCache>false</skipCache>
+ <overwrite>false</overwrite>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>3.1.0</version>
Review Comment:
```suggestion
```
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
Review Comment:
Version (newer one) already defined.
```suggestion
```
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
Release manager may be using Mac or Linux, but needs to create package with lib for all supported platforms. I think these download/extract steps should be moved to existing `dist` profile (preferably in `hadoop-ozone/dist/pom.xml`).
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <id>fetch-hadoop-native-tar</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <url>https://archive.apache.org/dist/hadoop/common/hadoop-${hadoop.version}/hadoop-${hadoop.version}.tar.gz</url>
Review Comment:
We should not download 500MB tarball for each build. Need to find a way to get these libs with less overhead.
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <id>fetch-hadoop-native-tar</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <url>https://archive.apache.org/dist/hadoop/common/hadoop-${hadoop.version}/hadoop-${hadoop.version}.tar.gz</url>
+ <unpack>true</unpack>
+ <outputDirectory>${project.build.directory}/hadoop-bin</outputDirectory>
+ <skipCache>false</skipCache>
+ <overwrite>false</overwrite>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>3.1.0</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ <phase>prepare-package</phase>
+ <configuration>
+ <target>
+ <mkdir dir="${project.build.directory}/native-lib" />
+ <exec executable="bash" failonerror="true">
+ <arg value="-c" />
+ <arg value="cp -P ${project.build.directory}/hadoop-bin/hadoop-*/lib/native/libhadoop.so* ${project.build.directory}/native-lib/" />
+ </exec>
+ </target>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ <profile>
+ <id>native-mac</id>
+ <activation>
+ <os>
+ <family>mac</family>
+ <arch>aarch64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
Review Comment:
```suggestion
```
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <id>fetch-hadoop-native-tar</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <url>https://archive.apache.org/dist/hadoop/common/hadoop-${hadoop.version}/hadoop-${hadoop.version}.tar.gz</url>
+ <unpack>true</unpack>
+ <outputDirectory>${project.build.directory}/hadoop-bin</outputDirectory>
+ <skipCache>false</skipCache>
+ <overwrite>false</overwrite>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>3.1.0</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ <phase>prepare-package</phase>
+ <configuration>
+ <target>
+ <mkdir dir="${project.build.directory}/native-lib" />
+ <exec executable="bash" failonerror="true">
+ <arg value="-c" />
+ <arg value="cp -P ${project.build.directory}/hadoop-bin/hadoop-*/lib/native/libhadoop.so* ${project.build.directory}/native-lib/" />
+ </exec>
+ </target>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ <profile>
+ <id>native-mac</id>
+ <activation>
+ <os>
+ <family>mac</family>
+ <arch>aarch64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <id>fetch-mac-binary</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <!-- TODO: change ChenSammi to apache after https://github.com/apache/ozone-docker-runner/pull/66 is merged-->
+ <url>https://raw.githubusercontent.com/ChenSammi/ozone-docker-runner/master/hadoop-native-lib/Mac_OS_X-aarch64/${hadoop.version}/libhadoop.1.0.0.dylib</url>
+ <unpack>false</unpack>
+ <outputDirectory>${project.build.directory}/native-lib</outputDirectory>
+ <skipCache>false</skipCache>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>3.1.0</version>
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264834454
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <inherited>false</inherited>
+ <executions>
+ <execution>
+ <id>fetch-hadoop-native-tar</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>generate-resources</phase>
+ <configuration>
+ <url>https://archive.apache.org/dist/hadoop/common/hadoop-${hadoop.version}/hadoop-${hadoop.version}.tar.gz</url>
Review Comment:
It only download once for every hadoop version locally. Maven will cache it. See above "How was this patch tested" section. And if -Pnative or -Pnative-mac is not enabled on maven build, it will not trigger the download.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264897167
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
OK, if it's needed for tests, then keep it in root `pom.xml`, still, it should be platform-agnostic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264897167
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
OK, if it's needed for tests, then keep it in root `pom.xml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10305:
URL: https://github.com/apache/ozone/pull/10305#issuecomment-4493804067
> @ChenSammi thanks for updating the patch
>
> Since download happens only when `native`/`native-mac` profile is enabled, this is not tested by CI. I still think we should download _all_ libs in `dist` profile (i.e. I suggest removing new profiles). Linking platform-specific lib should happen at runtime (remove existing link if exists and does not match platform), in `ozone` or before unit/integration tests (`process-test-resources` phase?) if you want to use it in JUnit tests.
Does dist profile run at last after all packages compile and unit test finished in CI? If so, unit tests and integration tests will fail for native library not ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264860564
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
I did put it into dist pom.xml in first version. But there are at least two more packages (unit test and integration test) requires native library to run(not covered in this patch). So to avoid duplicate the profiles in multiple pom.xml, gemini suggests to put it in root pom.xml, and with inherited false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265378735
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean? Or what you mean is we have two dist ozone-2.1.0-SNAPSHOT directory, for example ozone-2.1.0-SNAPSHOT-x86-64, and ozone-2.1.0-SNAPSHOT-aarch64?
I think -Pnative is the main one for official release, -Pnative-mac is mainly for dev with mac environment. I don't expect we will release macos package officially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265442174
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
> If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean?
Yes.
> I think -Pnative is the main one for official release, -Pnative-mac is mainly for dev with mac environment. I don't expect we will release macos package officially.
OK. (We do include some Mac/Windows native libs for Netty.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3264176777
##########
hadoop-ozone/dist/dev-support/bin/dist-layout-stitching:
##########
@@ -143,6 +143,24 @@ for file in $(find "${ROOT}" -path '*/share/ozone/lib/*jar' | sort); do
cp -p "$file" share/ozone/lib/
done
+# ---------------------------------------------------------
+# Copy Hadoop Native Libraries (libhadoop.so) - Conditionally
+# ---------------------------------------------------------
+NATIVE_LIBS_DIR="${ROOT}/target/native-lib"
+
+# Check if the .so files actually exist before attempting to copy
+if ls "${NATIVE_LIBS_DIR}"/libhadoop.* 1> /dev/null 2>&1; then
+ echo "Found Hadoop native libraries. Copying to distribution..."
+
+ # Create the native directory in the final staging area
+ run mkdir -p ./lib/native
+
+ # Copy the files and symlinks safely
+ run cp -rP "${NATIVE_LIBS_DIR}/"libhadoop.* ./lib/native
Review Comment:
./lib/native doesn't always exists. But yes, I think ozone-env.sh or ozone.sh can auto add lib/native to java library path if it exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265262366
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
> What "it should be platform-agnostic" specific mean, can you elaborate it a bit more?
Build "needs to create package with lib for all supported platforms" regardless of which OS/arch it runs on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10305:
URL: https://github.com/apache/ozone/pull/10305#discussion_r3265378735
##########
pom.xml:
##########
@@ -2633,5 +2633,126 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>native</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>x86_64</arch>
+ </os>
+ <property>
+ <name>fetch-native-hadoop</name>
+ </property>
+ </activation>
Review Comment:
If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean? Or what you mean is we have two dist ozone-2.1.0-SNAPSHOT directory, for example ozone-2.1.0-SNAPSHOT-x86-64, and ozone-2.1.0-SNAPSHOT-aarch64?
I think -Pnative is the main one for official release, -Pnative-mac is mainly for dev with mac environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10305:
URL: https://github.com/apache/ozone/pull/10305#issuecomment-4494950281
> Does dist profile run at last after all packages compile and unit test finished in CI?
No, it can execute plugins at any phase, and has some definition in root POM, just like the new profiles:
https://github.com/apache/ozone/blob/92374bcc5efc6fc59ced4689e66e50a163476b0c/pom.xml#L2553
It is activated by `-Pdist`, which CI uses and release manager guide mentions, so we don't need change in those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15318. Download libhadoop library during maven build [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10305:
URL: https://github.com/apache/ozone/pull/10305#issuecomment-4494960337
BTW, I think the feature branch should be updated from `master`. `compile (8)` check is timing out because it uses outdated `macos-13` runner, which is no longer supported by GitHub (HDDS-13732).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org