You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2020/04/28 01:38:44 UTC
[GitHub] [bigtop] sekikn opened a new pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
sekikn opened a new pull request #634:
URL: https://github.com/apache/bigtop/pull/634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn merged pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
sekikn merged pull request #634:
URL: https://github.com/apache/bigtop/pull/634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
sekikn commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r425764478
##########
File path: bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome
##########
@@ -36,6 +36,7 @@ OPENJAVA8_HOME_CANDIDATES=(
'/usr/lib/jvm/java-1.8.0-openjdk-ppc64el'
'/usr/lib/jvm/java-1.8.0-openjdk'
'/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0'
+ '/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64'
Review comment:
Thanks @guyuqi, I missed that. Will fix it together when rebasing this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #634:
URL: https://github.com/apache/bigtop/pull/634#issuecomment-620336224
I confirmed that this PR worked with Debian 10 and Ubuntu 18.04, as follows.
(In the following steps, I temporarily rewrote the repository URL in the config files to avoid failure caused by non-existent one)
```
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
$ curl -sL https://github.com/apache/bigtop/pull/634.patch | git apply -
$ git status
On branch master
Your branch is up to date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: bigtop-deploy/puppet/manifests/bigtop_repo.pp
modified: bigtop-deploy/puppet/manifests/jdk.pp
modified: bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome
modified: bigtop_toolchain/bin/puppetize.sh
Untracked files:
(use "git add <file>..." to include in what will be committed)
provisioner/docker/config_debian-10.yaml
provisioner/docker/config_ubuntu-18.04.yaml
no changes added to commit (use "git add" and/or "git commit -a")
$ vi provisioner/docker/config_debian-10.yaml
$ vi provisioner/docker/config_ubuntu-18.04.yaml
$ cat provisioner/docker/config_debian-10.yaml
(snip)
docker:
memory_limit: "4g"
image: "bigtop/puppet:trunk-debian-10"
repo: "http://repos.bigtop.apache.org/releases/1.4.0/debian/9/$(ARCH)"
distro: debian
components: [hdfs, yarn, mapreduce]
enable_local_repo: true
smoke_test_components: [hdfs, yarn, mapreduce]
$ cat provisioner/docker/config_ubuntu-18.04.yaml
(snip)
docker:
memory_limit: "4g"
image: "bigtop/puppet:trunk-ubuntu-18.04"
repo: "http://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/$(ARCH)"
distro: debian
components: [hdfs, yarn, mapreduce]
enable_local_repo: true
smoke_test_components: [hdfs, yarn, mapreduce]
$ ./build.sh trunk-debian-10
(snip)
Successfully built 86e144cd84e5
Successfully tagged bigtop/puppet:trunk-debian-10
++ rm -f Dockerfile puppetize.sh
$ ./build.sh trunk-ubuntu-18.04
(snip)
Successfully built aee3ab6549bc
Successfully tagged bigtop/puppet:trunk-ubuntu-18.04
++ rm -f Dockerfile puppetize.sh
$ cd -
/home/sekikn/repos/bigtop
$ cd provisioner/docker
$ ./docker-hadoop.sh -d; ./docker-hadoop.sh -c 1 -C config_debian-10.yaml
(snip)
$ docker exec -it 20200428105855r15042_bigtop_1 cat /etc/issue
Debian GNU/Linux 10 \n \l
$ docker exec -it 20200428105855r15042_bigtop_1 java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)
$ docker exec -it 20200428105855r15042_bigtop_1 jps
4960 Jps
3456 ResourceManager
4433 DataNode
3921 JobHistoryServer
4754 NodeManager
3016 NameNode
4154 WebAppProxyServer
$ ./docker-hadoop.sh -d; ./docker-hadoop.sh -c 1 -C config_ubuntu-18.04.yaml
(snip)
$ docker exec -it 20200428110523r10746_bigtop_1 cat /etc/issue
Ubuntu 18.04.3 LTS \n \l
$ docker exec -it 20200428110523r10746_bigtop_1 java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~18.04-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
$ docker exec -it 20200428110523r10746_bigtop_1 jps
6260 NodeManager
5080 ResourceManager
4668 NameNode
5517 JobHistoryServer
6446 Jps
5726 WebAppProxyServer
5983 DataNode
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] evans-ye commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r422330723
##########
File path: bigtop-deploy/puppet/manifests/bigtop_repo.pp
##########
@@ -43,9 +43,10 @@
# I couldn't enforce the sequence -> anymore because of this
# https://twitter.com/c0sin/status/875869783979196416
apt::conf { "disable_keys":
- content => "APT::Get::AllowUnauthenticated 1;",
+ content => "APT::Get::AllowUnauthenticated 1;\nAcquire::AllowInsecureRepositories \"true\";",
Review comment:
@elukey I did some trace and it was originally added in:
https://issues.apache.org/jira/browse/BIGTOP-1676
I'm the author. I guess that's for testing purpose. Anyhow it seems not secure as what you're concerning. We should make it default to secure mode and disable if needed in test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
elukey commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r416355636
##########
File path: bigtop-deploy/puppet/manifests/bigtop_repo.pp
##########
@@ -43,9 +43,10 @@
# I couldn't enforce the sequence -> anymore because of this
# https://twitter.com/c0sin/status/875869783979196416
apt::conf { "disable_keys":
- content => "APT::Get::AllowUnauthenticated 1;",
+ content => "APT::Get::AllowUnauthenticated 1;\nAcquire::AllowInsecureRepositories \"true\";",
Review comment:
Is there a reason why we have these two? Asking as a genuine question, I am new to BigTop but I wouldn't have expected something like this..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] evans-ye commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r422534764
##########
File path: bigtop-deploy/puppet/manifests/bigtop_repo.pp
##########
@@ -43,9 +43,10 @@
# I couldn't enforce the sequence -> anymore because of this
# https://twitter.com/c0sin/status/875869783979196416
apt::conf { "disable_keys":
- content => "APT::Get::AllowUnauthenticated 1;",
+ content => "APT::Get::AllowUnauthenticated 1;\nAcquire::AllowInsecureRepositories \"true\";",
Review comment:
Alright. I'm working on the fix. BIGTOP-3355 is to track this. How about we leave it as-is here and fix it in BIGTOP-3355 @sekikn @elukey.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] evans-ye commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r422534764
##########
File path: bigtop-deploy/puppet/manifests/bigtop_repo.pp
##########
@@ -43,9 +43,10 @@
# I couldn't enforce the sequence -> anymore because of this
# https://twitter.com/c0sin/status/875869783979196416
apt::conf { "disable_keys":
- content => "APT::Get::AllowUnauthenticated 1;",
+ content => "APT::Get::AllowUnauthenticated 1;\nAcquire::AllowInsecureRepositories \"true\";",
Review comment:
Alright. I'm working on the fix. [BIGTOP-3355](https://issues.apache.org/jira/browse/BIGTOP-3355) is to track this. How about we leave it as-is here and fix it in BIGTOP-3355 @sekikn @elukey.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #634:
URL: https://github.com/apache/bigtop/pull/634#issuecomment-653969884
Updated the PR. Rebased on master and addressed Yuqi's comment.
After building bigtop-utils and zookeeper, and re-creating the bigtop-puppet and bigtop-slaves images locally, I confirmed this fix worked as follows:
```
~/repos/bigtop/provisioner/docker$ ./docker-hadoop.sh -d -C ~/config_ubuntu-18.04.yaml -c 1 -s
(snip)
Notice: /Stage[main]/Hadoop_zookeeper::Server/Service[zookeeper-server]/ensure: ensure changed 'stopped' to 'running'
Notice: Applied catalog in 49.91 seconds
(snip)
===== START TO RUN SMOKE TESTS: zookeeper =====
(snip)
TestZookeeper > testZkServerStatus STANDARD_ERROR
20/07/06 00:20:06 INFO lang.Object: Running zkServer.sh status
20/07/06 00:20:06 INFO lang.Object: zkServer.sh status checks out.
Finished generating test XML results (0.01 secs) into: /bigtop-home/bigtop-tests/smoke-tests/zookeeper/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.024 secs) into: /bigtop-home/bigtop-tests/smoke-tests/zookeeper/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:zookeeper:test (Thread[Task worker for ':' Thread 7,5,main]) completed. Took 1.211 secs.
BUILD SUCCESSFUL in 1m 35s
```
```
~/repos/bigtop/provisioner/docker$ ./docker-hadoop.sh -d -C ~/config_debian-10.yaml -c 1 -s
(snip)
Notice: /Stage[main]/Hadoop_zookeeper::Server/Service[zookeeper-server]/ensure: ensure changed 'stopped' to 'running'
Notice: Applied catalog in 75.49 seconds
(snip)
===== START TO RUN SMOKE TESTS: zookeeper =====
(snip)
TestZookeeper > testZkServerStatus STANDARD_ERROR
20/07/06 00:53:54 INFO lang.Object: Running zkServer.sh status
20/07/06 00:53:54 INFO lang.Object: zkServer.sh status checks out.
Finished generating test XML results (0.008 secs) into: /bigtop-home/bigtop-tests/smoke-tests/zookeeper/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.018 secs) into: /bigtop-home/bigtop-tests/smoke-tests/zookeeper/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:zookeeper:test (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 1.145 secs.
BUILD SUCCESSFUL in 1m 10s
```
where these config files were:
```
$ cat ~/config_ubuntu-18.04.yaml
docker:
memory_limit: "4g"
image: "bigtop/puppet:trunk-ubuntu-18.04"
repo: "http://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/$(ARCH)"
distro: debian
components: [zookeeper]
enable_local_repo: true
smoke_test_components: [zookeeper]
```
```
$ cat ~/config_debian-10.yaml
docker:
memory_limit: "4g"
image: "bigtop/puppet:trunk-debian-10"
repo: "http://repos.bigtop.apache.org/releases/1.4.0/debian/9/$(ARCH)"
distro: debian
components: [zookeeper]
enable_local_repo: true
smoke_test_components: [zookeeper]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
sekikn commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r425761122
##########
File path: bigtop-deploy/puppet/manifests/bigtop_repo.pp
##########
@@ -43,9 +43,10 @@
# I couldn't enforce the sequence -> anymore because of this
# https://twitter.com/c0sin/status/875869783979196416
apt::conf { "disable_keys":
- content => "APT::Get::AllowUnauthenticated 1;",
+ content => "APT::Get::AllowUnauthenticated 1;\nAcquire::AllowInsecureRepositories \"true\";",
Review comment:
Thanks for the comment @elukey and the fix @evans-ye, I'll rebase this PR after #639 is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
elukey commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r422540991
##########
File path: bigtop-deploy/puppet/manifests/bigtop_repo.pp
##########
@@ -43,9 +43,10 @@
# I couldn't enforce the sequence -> anymore because of this
# https://twitter.com/c0sin/status/875869783979196416
apt::conf { "disable_keys":
- content => "APT::Get::AllowUnauthenticated 1;",
+ content => "APT::Get::AllowUnauthenticated 1;\nAcquire::AllowInsecureRepositories \"true\";",
Review comment:
+1 thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #634:
URL: https://github.com/apache/bigtop/pull/634#issuecomment-654915036
I'm going to merge this PR following Bigtop's CTR model. Then I'll update container images and examine if packaging, deployment and smoke tests succeed on Jenkins.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #634: BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner.
Posted by GitBox <gi...@apache.org>.
guyuqi commented on a change in pull request #634:
URL: https://github.com/apache/bigtop/pull/634#discussion_r422450445
##########
File path: bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome
##########
@@ -36,6 +36,7 @@ OPENJAVA8_HOME_CANDIDATES=(
'/usr/lib/jvm/java-1.8.0-openjdk-ppc64el'
'/usr/lib/jvm/java-1.8.0-openjdk'
'/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0'
+ '/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64'
Review comment:
`bigtop-detect-javahome` leverages `ls -rvd $JAVA_HOME-CADIDATES` to detect `JAVA_HOME`.
So pls do not add Arch-tag here (just: `/usr/lib/jvm/adoptopenjdk-8-hotspot` ), otherwise `JAVA_HOME ` couldn't be detected on some platforms (arm64/ppcle).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org