You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/03/26 21:44:04 UTC
[GitHub] [bookkeeper] pkumar-singh opened a new pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
pkumar-singh opened a new pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663
### Motivation
Migrate bookkeeper to gradle.
- Build BKCTL and generate artifacts
- Build docker image from the jar created from existing source and run integration tests.
### Changes
- A Dockerfile which will build docker image from existing source.
- A Shell file which will generate all the artifacts for building Docker image
- All build.gradle files to build and run integration tests.
Master Issue: #2640
### How to build
./gradlew :tests:integration:cluster:test
` Task :tests:integration:cluster:buildImage
#1 [internal] load build definition from Dockerfile
#1 sha256:13c696b34811c50ef1582dd1c27e59e8cf726660222ada83feb0b294a4197b14
#1 transferring dockerfile: 37B 0.0s done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:028ece09d81436333eece58b183b9dc0a52cbc28bf2d07cfad5f3080e03d38b7
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/centos:7
#3 sha256:30875b35a89c8e8a29cd7cf120689bb68cdab8d769419707e07138dfe977d237
#3 DONE 9.7s
#4 [ 1/16] FROM docker.io/library/centos:7@sha256:0f4ec88e21daf75124b8a9e5ca03c37a5e937e0e108a255d890492430789b60e
#4 sha256:a82657dc9009ab4ecab0a966d899b2972800cfe1ee9d42fc736a0cac1581ac52
#4 DONE 0.0s
#11 [internal] load build context
#11 sha256:2ba06714f2ca9169cd544ca120d73724654078b8159628baf9e4af328b2d617a
#11 transferring context: 23B
#11 transferring context: 122.54MB 4.5s done
#11 DONE 4.5s
#13 [ 9/16] RUN mv /opt/server/lib/*.jar /opt/bookkeeper/lib/
#13 sha256:0edb1ad87b6c9035382a0500b77b4577388841c8eef32516507007f857f058a8
#13 CACHED
#18 [14/16] COPY ./conf/* /opt/bookkeeper/conf/
#18 sha256:53a4f93597253dff0934c46de43bac4f857ced79886d97523b3f5c190624aeb6
#18 CACHED
#19 [15/16] RUN chmod +x -R /opt/bookkeeper/scripts
#19 sha256:6be237cc3422cc33cb527d6ac9dac6132f82d474e1f0a28a4f50784454319dee
#19 CACHED
#7 [ 4/16] RUN mkdir /opt/bookkeeper/lib
#7 sha256:3fe6bccdc94f36263533c94142448099fa427415412b40e6c2d7003f0f9172f9
#7 CACHED
#17 [13/16] COPY ./temp_conf/* /opt/bookkeeper/conf/
#17 sha256:da439db0c1a32b3a7a60d1ffabc8fe4c4d23f1c4a7bd6095fc5d4a4530691f9d
#17 CACHED
#6 [ 3/16] WORKDIR /opt/bookkeeper
#6 sha256:75cb16fec34eda0a19b049218df4fd332274fb1af59e20dea96ce8bda3cac37f
#6 CACHED
#16 [12/16] COPY ./bin/* /opt/bookkeeper/bin/
#16 sha256:fe3bceec3581b5e8183cf2804a8e8c0ef06c5eb11b498260bd6fa5a3df7ec139
#16 CACHED
#8 [ 5/16] RUN mkdir /opt/bookkeeper/bin
#8 sha256:1efcf781a6c50daf837d1ed53d859145b6d07b9050cfa5ed2cab6a370d64c0fb
#8 CACHED
#15 [11/16] COPY ./temp_bin/* /opt/bookkeeper/bin/
#15 sha256:7e7fb09841111dd8b7444c6007be8c945bf0b84d698d81cf2e7cfda7df01bff0
#15 CACHED
#5 [ 2/16] RUN set -x && adduser "bookkeeper" && yum install -y java-1.8.0-openjdk-headless wget bash sudo && wget -q https://bootstrap.pypa.io/pip/2.7/get-pip.py && python get-pip.py && pip install zk-shell && rm -rf get-pip.py && mkdir -pv /opt && cd /opt && yum clean all
#5 sha256:055562ef0c2a679a10597b7a9c15d731dc1235a63e74c44e0ec246d059b2cdbd
#5 CACHED
#10 [ 7/16] RUN mkdir /opt/bookkeeper/scripts
#10 sha256:8ed4308f6f3b2710fc497036cc4d75df7e0c8d335b9f4ca3962f2fbac5ba55f5
#10 CACHED
#14 [10/16] COPY ./scripts/* /opt/bookkeeper/scripts/
#14 sha256:f78d90c1c59a8d4bf347fc9109ea28ad5a811ab605fc688a1e60336daf6e1564
#14 CACHED
#12 [ 8/16] ADD ./dist/server.tar /opt/
#12 sha256:1f8a62db7f0bcabf8bbc7b9293c362a4c5b91a90801fab595e264a106529ac45
#12 CACHED
#9 [ 6/16] RUN mkdir /opt/bookkeeper/conf
#9 sha256:271a4ee43b22ab37ab9ba5e511fc9a477950119c63da39d5d399bf72f4c7ab08
#9 CACHED
#20 [16/16] RUN chmod +x -R /opt/bookkeeper/bin
#20 sha256:65e54de91545e3a96a707a741727068de55978138d7ead28000b789a7dc4818f
#20 CACHED
#21 exporting to image
#21 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#21 exporting layers done
#21 writing image sha256:3c816ab89d94c687e47c91df2ccd85a20f481679bb76540848d137e60adbcbb0 done
#21 naming to docker.io/apachebookkeeper/bookkeeper-current:latest done
#21 DONE 0.0s
> Task :tests:integration:cluster:test
org.apache.bookkeeper.tests.integration.cluster.SimpleClusterTest
Test test000_ClusterIsEmpty PASSED
Test test001_StartBookie PASSED (5.9s)
Test test002_StopBookie PASSED (10.5s)
org.apache.bookkeeper.tests.integration.stream.LocationClientTest
Test testLocateStorageContainers PASSED
org.apache.bookkeeper.tests.integration.stream.BkCtlTest
Test simpleTest PASSED (2.9s)
Test putGetKey PASSED (7.9s)
Test createTable PASSED (2.5s)
Test createNamespace PASSED (2.4s)
Test listBookies PASSED (1.3s)
Test incGetKey PASSED (6.9s)
Test showLastMark PASSED (1s)
org.apache.bookkeeper.tests.integration.stream.StorageAdminClientTest
Test testStreamAPIClientSideRouting PASSED
Test testNamespaceAPIClientSideRouting PASSED
Test testStreamAPIServerSideRouting PASSED
Test testNamespaceAPIServerSideRouting PASSED
org.apache.bookkeeper.tests.integration.stream.TableClientTest
Test testTableAPIClientSideRouting PASSED (3.9s)
Test testTableAPIServerSideRouting PASSED (4s)
SUCCESS: Executed 17 tests in 2m 47s
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 3m 6s
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat merged pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
merlimat merged pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] pkumar-singh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602615940
##########
File path: tests/docker-images/statestore-image/bin/bkctl
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
Review comment:
Yeah. It's stripped down version of that.
IN the bin it builds the classpath like the following.
`CLI_MODULE_NAME="(org.apache.bookkeeper-)?bookkeeper-tools"
CLI_MODULE_HOME=${BK_HOME}/${CLI_MODULE_PATH}
# find the module jar
CLI_JAR=$(find_module_jar ${CLI_MODULE_PATH} ${CLI_MODULE_NAME})
# set up the classpath
CLI_CLASSPATH=$(set_module_classpath ${CLI_MODULE_PATH})` While in the docker image we know where the jars are put so we do not have to search for module jar etc.
Therefore we can directly use the classpath.
`CLI_CLASSPATH=${BK_HOME}/lib/*`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] pkumar-singh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602637774
##########
File path: tests/docker-images/statestore-image/bin/bkctl
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
Review comment:
I understand that. It can be done If we name the jars accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602613602
##########
File path: tests/docker-images/statestore-image/bin/bkctl
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
Review comment:
Are these different from the scripts that are already in top level `./bin` directory?
##########
File path: dependencies.gradle
##########
@@ -25,6 +25,7 @@ depVersions = [
arquillianCubeDocker: "1.18.2",
arquillianJunit: "1.6.0.Final",
bcFips: "1.0.1",
+ codehale: "3.1.0",
Review comment:
the old name was "codahale" with "A", but it was anyway renamed to dropwizard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] pkumar-singh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602617197
##########
File path: dependencies.gradle
##########
@@ -25,6 +25,7 @@ depVersions = [
arquillianCubeDocker: "1.18.2",
arquillianJunit: "1.6.0.Final",
bcFips: "1.0.1",
+ codehale: "3.1.0",
Review comment:
Its just a variable I will rename it to dropwizard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] sursingh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
sursingh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602603728
##########
File path: settings.gradle
##########
@@ -24,39 +24,49 @@ pluginManagement {
rootProject.name = 'bookkeeper'
-include 'bookkeeper-benchmark',
+include('circe-checksum',
+ 'circe-checksum:src:main:circe',
+ 'cpu-affinity',
+ 'cpu-affinity:src:main:affinity',
'bookkeeper-common',
- 'bookkeeper-common-allocator',
'bookkeeper-http:http-server',
'bookkeeper-http:vertx-http-server',
- 'bookkeeper-proto',
'bookkeeper-server',
+ 'bookkeeper-common-allocator',
+ 'bookkeeper-proto',
'bookkeeper-stats',
+ 'tests:integration-tests-base',
+ 'tests:integration-tests-utils',
+ 'tests:integration-tests-topologies',
+ 'tests:integration:cluster',
+ 'tests:integration:smoke',
+ 'tests:integration:standalone',
Review comment:
Can we keep this list sorted. That will make it easier to resolve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] pkumar-singh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602609086
##########
File path: settings.gradle
##########
@@ -24,39 +24,49 @@ pluginManagement {
rootProject.name = 'bookkeeper'
-include 'bookkeeper-benchmark',
+include('circe-checksum',
+ 'circe-checksum:src:main:circe',
+ 'cpu-affinity',
+ 'cpu-affinity:src:main:affinity',
'bookkeeper-common',
- 'bookkeeper-common-allocator',
'bookkeeper-http:http-server',
'bookkeeper-http:vertx-http-server',
- 'bookkeeper-proto',
'bookkeeper-server',
+ 'bookkeeper-common-allocator',
+ 'bookkeeper-proto',
'bookkeeper-stats',
+ 'tests:integration-tests-base',
+ 'tests:integration-tests-utils',
+ 'tests:integration-tests-topologies',
+ 'tests:integration:cluster',
+ 'tests:integration:smoke',
+ 'tests:integration:standalone',
Review comment:
Yeah.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602633160
##########
File path: tests/docker-images/statestore-image/bin/bkctl
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
Review comment:
Conf files I think are ok, but the bin scripts it would be nice to have tests to use the same one (so that we can actually test them). Do you see a way to support the 2 different cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] pkumar-singh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602615940
##########
File path: tests/docker-images/statestore-image/bin/bkctl
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
Review comment:
Yeah. It's stripped down version of that.
IN the bin it builds the classpath like the following.
`CLI_MODULE_NAME="(org.apache.bookkeeper-)?bookkeeper-tools"
CLI_MODULE_HOME=${BK_HOME}/${CLI_MODULE_PATH}
find the module jar
CLI_JAR=$(find_module_jar ${CLI_MODULE_PATH} ${CLI_MODULE_NAME})
set up the classpath
CLI_CLASSPATH=$(set_module_classpath ${CLI_MODULE_PATH})`
While in the docker image we know where the jars are put so we do not have to search for module jar or any jar for that matter.
Therefore we can directly use the classpath.
`CLI_CLASSPATH=${BK_HOME}/lib/*`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] pkumar-singh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602616535
##########
File path: tests/docker-images/statestore-image/bin/bkctl
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
Review comment:
Same things for conf files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] pkumar-singh commented on a change in pull request #2663: ISSUE-2640: BP-43: Run integration tests with current source code.
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on a change in pull request #2663:
URL: https://github.com/apache/bookkeeper/pull/2663#discussion_r602615940
##########
File path: tests/docker-images/statestore-image/bin/bkctl
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
Review comment:
Yeah. It's stripped down version of that.
IN the bin it builds the classpath like the following.
`CLI_MODULE_NAME="(org.apache.bookkeeper-)?bookkeeper-tools"
CLI_MODULE_HOME=${BK_HOME}/${CLI_MODULE_PATH}
find the module jar
CLI_JAR=$(find_module_jar ${CLI_MODULE_PATH} ${CLI_MODULE_NAME})
set up the classpath
CLI_CLASSPATH=$(set_module_classpath ${CLI_MODULE_PATH})` While in the docker image we know where the jars are put so we do not have to search for module jar etc.
Therefore we can directly use the classpath.
`CLI_CLASSPATH=${BK_HOME}/lib/*`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org