You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2022/09/30 22:45:54 UTC
[GitHub] [bigtop] cheelio opened a new pull request, #1029: BIGTOP-3835: Add Apache Knox package
cheelio opened a new pull request, #1029:
URL: https://github.com/apache/bigtop/pull/1029
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
Apache Knox provides a gateway for interacting with the REST APIs and UIs of Apache Hadoop deployments.
It's a crucial component in secure environments and therefore I thinks it should be part of the Bigtop stack.
I can understand that it's maybe too late to add this to the into Bigtop 3.2 BOM. If so, maybe we can add this to Bigtop 3.3?
### How was this patch tested?
Build with:
```
#CentOS
docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-centos-7 bash -l -c './gradlew knox-clean knox-pkg yum'
#Ubuntu
docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-ubuntu-16.04 bash -l -c './gradlew knox-clean knox-pkg repo'
```
I created basic smoke tests:
```
cd provisioner/docker
# CentOS:
./docker-hadoop.sh --destroy
./docker-hadoop.sh --enable-local-repo --create 1 -C config_centos-7.yaml --stack knox --smoke-tests knox
# Ubuntu:
./docker-hadoop.sh --destroy
./docker-hadoop.sh --enable-local-repo --create 1 -C config_ubuntu-20.04.yaml --stack knox --smoke-tests knox
```
Further, the application can be tested by launching a basic LDAP server:
```
cd provisioner/docker
./docker-hadoop.sh --destroy
./docker-hadoop.sh --enable-local-repo --create 1 -C config_centos-7.yaml --stack knox
./docker-hadoop.sh --exec 1 sudo -u knox KNOX_LDAP_PID_DIR=/var/run/knox KNOX_LDAP_CONF_DIR=/etc/knox KNOX_LDAP_LOG_DIR=/var/log/knox KNOX_GATEWAY_DATA_DIR=/var/lib/knox/data /usr/lib/knox/bin/ldap.sh start
```
Now, you can login by using the username/password combination admin:admin-password.
By default, the application runs on port 8443
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3835: Add Apache Knox package [bigtop]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 closed pull request #1029: BIGTOP-3835: Add Apache Knox package
URL: https://github.com/apache/bigtop/pull/1029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] cheelio commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by GitBox <gi...@apache.org>.
cheelio commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1272387215
I just tested Knox basic service integration:
```
#start container with knox and livy service:
(cd provisioner/docker && ./docker-hadoop.sh --destroy && ./docker-hadoop.sh --enable-local-repo --create 1 -C config_centos-7.yaml --stack knox,livy)
#start integrated (test) LDAP server on container:
(cd provisioner/docker; ./docker-hadoop.sh --exec 1 /usr/lib/knox/bin/ldap.sh start)
#Get IP of container:
export DOCKER_IP=$(cd provisioner/docker; ./docker-hadoop.sh --exec 1 hostname -I)
#Test Knox API Request:
curl -ku admin:admin-password -XGET https://${DOCKER_IP}:8443/gateway/admin/api/v1/topologies/sandbox
#Test Livy with knox:
curl -ku admin:admin-password -XGET https://${DOCKER_IP}:8443/gateway/sandbox/livy/sessions
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1647109852
> @guyuqi is there a plan that we add knox to bigtop 3.3?
Hi @JiaLiangC,
Masatake Iwasaki is working on 3.2.1 patch release to upgrade Hadoop to 3.3.5 to fix some critical issues.
For Bigtop-v3.3, I have no objections to adding Knox to into Bigtop from my side.
But we‘d better listen to the voices of other people in the community.
Could you kindly please send the `DISCUSSION` mail on our Bigtop mailing list?
OR please also add the comments on the relevant [Jira](https://issues.apache.org/jira/browse/BIGTOP-3909).
Let's give others the opportunity to chime in with their opinions.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1650859545
Could you guys kindly rebase the PR to resolve the conficts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] cheelio commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by GitBox <gi...@apache.org>.
cheelio commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1264393853
I just realized that I have to make a few more modifications on this PR regarding to the paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] JiaLiangC commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by "JiaLiangC (via GitHub)" <gi...@apache.org>.
JiaLiangC commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1681677192
@guyuqi
Since cheelio is no longer active, is there anything I can do to help you move forward with this pull request? Do you need assistance with testing or any other tasks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3835: Add Apache Knox package [bigtop]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-2041375196
Thanks @cheelio , the work is now moving to #1247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1275919195
@cheelio
Thank you for working on it. It's awesome.
For we've defined new comming 3.2 release in https://issues.apache.org/jira/browse/BIGTOP-3695,
this PR would be merged in the future 3.3 release after we test it. Thanks again for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1652815870
@JiaLiangC @cheelio
I just rebased the PR to resolve the conficts and the PR has provided the smoke tests.
Let me test it on some Distros and x86/Arm64 platform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] JiaLiangC commented on pull request #1029: BIGTOP-3835: Add Apache Knox package
Posted by "JiaLiangC (via GitHub)" <gi...@apache.org>.
JiaLiangC commented on PR #1029:
URL: https://github.com/apache/bigtop/pull/1029#issuecomment-1645336780
@guyuqi is there a plan that we add knox to bigtop 3.3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org