You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "MacChen02 (via GitHub)" <gi...@apache.org> on 2023/02/13 11:25:38 UTC
[GitHub] [bigtop] MacChen02 opened a new pull request, #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
MacChen02 opened a new pull request, #1088:
URL: https://github.com/apache/bigtop/pull/1088
https://issues.apache.org/jira/browse/BIGTOP-3896
The so of alluxio is not compiled by default, and uses the x86 version, so we need to compile libjnifuse*.so and copy it to the corresponding directory of alluxio when building adaptation
1. Modify the _bigtop-release-3.2.0/bigtop-packages/src/common/alluxio/do-component-build_ file to adapt Arm64 architecture code for fedora OS.
2. Modify the _bigtop-release-3.2.0/bigtop-packages/src/common/alluxio/ install_alluxio.sh_ file to copy the libjnifuse*.so file.
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (BIGTOP-3896)
- [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] MacChen02 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1429210790
@yoda-mon
The error log show the version of GLIBC needs 2.2.5，which version do you have?
The version in my env is:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] yoda-mon commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "yoda-mon (via GitHub)" <gi...@apache.org>.
yoda-mon commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1473009139
@MacChen02 The link on my comment above, 3 weeks ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1471368474
> @guyuqi It need to install fuse3 and fuse3-devel, command is: **yum install -y fuse3 fuse3-devel**
Please add fuse3 installation into Bigtio toolchain.
Thanks @MacChen02
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1467795210
Failed to build Alluxio on Fedora-36 Arm64 against in docker image: `bigtop/slaves:3.2.0-fedora-36-aarch64`:
```
[INFO] [exec] In file included from ../../src/main/native/libjnifuse/jnifuse_helper.cc:22:
[INFO] [exec] ../../src/main/native/libjnifuse/jnifuse_impls.h:42:62: error: 'fuse_readdir_flags' has not been declared
[INFO] [exec] 42 | off_t offset, struct fuse_file_info *fi, fuse_readdir_flags flags);
[INFO] [exec] | ^~~~~~~~~~~~~~~~~~
[INFO] [exec] ../../src/main/native/libjnifuse/jnifuse_helper.cc: In function 'jint Java_alluxio_jnifuse_LibFuse_fuse_1main_1real(JNIEnv*, jobject, jobject, jint, jobjectArray)':
[INFO] [exec] ../../src/main/native/libjnifuse/jnifuse_helper.cc:46:23: error: invalid conversion from 'void* (*)(fuse_conn_info*, fuse_config*)' to 'void* (*)(fuse_conn_info*)' [-fpermissive]
[INFO] [exec] 46 | jnifuse_oper.init = init_wrapper;
[INFO] [exec] | ^~~~~~~~~~~~
[INFO] [exec] | |
[INFO] [exec] | void* (*)(fuse_conn_info*, fuse_config*)
[INFO] [exec] ../../src/main/native/libjnifuse/jnifuse_helper.cc:47:24: error: invalid conversion from 'int (*)(const char*, mode_t, fuse_file_info*)' {aka 'int (*)(const char*, unsigned int, fuse_file_info*)'} to 'int (*)(const char*, mode_t)' {aka 'int (*)(const char*, unsigned int)'} [-fpermissive]
.....
...
[exec] make: *** [/ws/bigtop/build/alluxio/rpm/BUILD/alluxio-2.8.0/integration/jnifuse/native/src/main/native/libjnifuse/Makefile:57: jnifuse_helper.fuse3.o] Error 1
```
Could you please verify it on our Bigtop toolchian environment again?
@MacChen02
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] MacChen02 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1429252721
@yoda-mon
update the file _bigtop-packages/src/common/alluxio/install_alluxio.sh_ , try it.
I test it in docker env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36 [bigtop]
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 closed pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
URL: https://github.com/apache/bigtop/pull/1088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] yoda-mon commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "yoda-mon (via GitHub)" <gi...@apache.org>.
yoda-mon commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1429121258
@MacChen02
Thank you for the PR. I tried to buiild and test your PR, but it seemed failed. Could you describe how you tested your patch ?
```sh
# On Rocky Linux 8, ARM64 hostmachine
./gradlew -POS=fedora-36 -Pdocker-run-option="--privileged" alluxio-clean alluxio-pkg-ind bigtop-utils-pkg-ind repo-ind
cd provisioner/docker
./docker-hadoop.sh -d --create 1 -F docker-compose-cgroupv2.yml -C config_fedora-36.yaml --enable-local-repo --repo file:///bigtop-home/output --disable-gpg-check --stack alluxio --smoke-tests alluxio
...
Error: Could not update: Execution of '/usr/bin/dnf -d 0 -e 1 -y install alluxio' returned 1: Error:
Problem: conflicting requests
- nothing provides libc.so.6(GLIBC_2.2.5)(64bit) needed by alluxio-2.8.0-2.fc36.aarch64
- nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by alluxio-2.8.0-2.fc36.aarch64
Error: /Stage[main]/Alluxio::Common/Package[alluxio]/ensure: change from 'purged' to 'latest' failed: Could not update: Execution of '/usr/bin/dnf -d 0 -e 1 -y install alluxio' returned 1: Error:
Problem: conflicting requests
- nothing provides libc.so.6(GLIBC_2.2.5)(64bit) needed by alluxio-2.8.0-2.fc36.aarch64
- nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by alluxio-2.8.0-2.fc36.aarch64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] Hann0321 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "Hann0321 (via GitHub)" <gi...@apache.org>.
Hann0321 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1486472566
@guyuqi
The code seems to require a start with trunk, for example, ./build.sh trunk-fedora-36-aarch64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1486504073
> trunk-fedora-36-aarch64
@Hann0321
It's just the tag name for docker images and could be named whatever you like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] yoda-mon commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "yoda-mon (via GitHub)" <gi...@apache.org>.
yoda-mon commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1471283018
@MacChen02 I showed an example how to update the toolchain code, I hope this will be helpful for you to update this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] MacChen02 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1483648616
> @MacChen02 The link on my comment above, 3 weeks ago.
thx，it's work well in local env, and update the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] yoda-mon commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "yoda-mon (via GitHub)" <gi...@apache.org>.
yoda-mon commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1429238645
@MacChen02
```sh
# After running the test procedure above
./docker-hadoop.sh --exec 1 bash
dnf install binutils
strings /usr/lib64/libc.so.6 |grep GLIBC
GLIBC_2.17
GLIBC_2.18
GLIBC_2.22
GLIBC_2.23
GLIBC_2.24
GLIBC_2.25
GLIBC_2.26
GLIBC_2.27
GLIBC_2.28
GLIBC_2.29
GLIBC_2.30
GLIBC_2.31
GLIBC_2.32
GLIBC_2.33
GLIBC_2.34
GLIBC_2.35
GLIBC_PRIVATE
```
```sh
# check the patch is applied
cat bigtop-home/bigtop-packages/src/common/alluxio/do-component-build |grep sed
# Licensed to the Apache Software Foundation (ASF) under one or more
sed -i "s|<nodeVersion>v10.11.0</nodeVersion>|<nodeVersion>v12.22.1</nodeVersion>|" webui/pom.xml
sed -i "s|<npmVersion>6.4.1</npmVersion>|<npmVersion>6.14.7</npmVersion>|" webui/pom.xml
sed -i "s|<nodeVersion>v10.11.0</nodeVersion>|<nodeVersion>v12.22.1</nodeVersion>|" webui/pom.xml
sed -i "s|<npmVersion>6.4.1</npmVersion>|<npmVersion>6.14.7</npmVersion>|" webui/pom.xml
sed -i "s|<activeByDefault>false</activeByDefault>|<activeByDefault>true</activeByDefault>|g" integration/jnifuse/native/pom.xml
```
If you tested on baremetals or VMs, it might be the cause of difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] yoda-mon commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "yoda-mon (via GitHub)" <gi...@apache.org>.
yoda-mon commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1429344083
@MacChen02 Same error occured and failed. I guess `do-component-build` does not recognize the OS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] yoda-mon commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "yoda-mon (via GitHub)" <gi...@apache.org>.
yoda-mon commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1440523817
@MacChen02
After cheking around this, I found you have to include `libfuse3` related libraries to tool chain.
cf. https://github.com/yoda-mon/bigtop/pull/2/files#diff-9337fb34935804ab7b56cd89ae94e14491ad4c64804daa9b1843eeec336772ed
After adding these the error above seemed solved. I guess you tested your patch on your customized environment.
Could you check again and update the PR message with how you test your patch ? I recommend you to follow the PR template for checking the reproducibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36 [bigtop]
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1876185104
New PR solved this problem, close the one.
https://github.com/apache/bigtop/pull/1229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] MacChen02 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1429505090
> @MacChen02 Same error occured and failed. I guess `do-component-build` does not recognize the OS.
@yoda-mon
The file _bigtop-packages/src/common/alluxio/do-component-build_ has the Fedora OS type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1484580581
@MacChen02
Failed to puppet install (`bigtop_toolchain::installer`) when buildng `bigtop/slaves` docker images on Fedora-36:
```
Error: Evaluation Error: Cannot reassign variable '$pkgs'
(file: /etc/puppet/code/modules/bigtop_toolchain/manifests/packages.pp, line: 76, column: 17) on node ...
```
Please verify it correctly on your local host.
```
$ pwd
bigtop/docker/bigtop-slaves
$ ./build.sh alluxio-fedora-36-aarch64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] MacChen02 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1429257346
@yoda-mon We solved this problem in the process of adapting the openEuler OS, the JIRA is:
https://issues.apache.org/jira/browse/BIGTOP-3881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] MacChen02 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1471116943
@guyuqi
It need to install fuse3 and fuse3-devel, command is:
**yum install -y fuse3 fuse3-devel**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] MacChen02 commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "MacChen02 (via GitHub)" <gi...@apache.org>.
MacChen02 commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1472997063
> @MacChen02 I showed an example how to update the toolchain code, I hope this will be helpful for you to update this PR.
Thanks，where can I see this example ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] yoda-mon commented on pull request #1088: BIGTOP-3896. Failed to deploy Alluxio on Arm64/ppc64le Fedora-36
Posted by "yoda-mon (via GitHub)" <gi...@apache.org>.
yoda-mon commented on PR #1088:
URL: https://github.com/apache/bigtop/pull/1088#issuecomment-1486069455
@MacChen02
Here shows an example for local testing.
```sh
cd docker/bigtop-puppet/
./build.sh trunk-fedora-36
cd ../bigtop-slaves/
./build.sh trunk-fedora-36
cd ../..
./gradlew -POS=fedora-36 -Pdocker-run-option="--privileged" alluxio-clean alluxio-pkg-ind bigtop-utils-pkg-ind repo-ind
cd provisioner/docker
./docker-hadoop.sh -d --create 1 -F docker-compose-cgroupv2.yml -C config_fedora-36.yaml --enable-local-repo --repo file:///bigtop-home/output --disable-gpg-check --stack alluxio --smoke-tests alluxio
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org