You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by el...@apache.org on 2020/06/09 11:05:27 UTC
[hadoop-ozone] branch HDDS-3757 created (now 8666fa6)
This is an automated email from the ASF dual-hosted git repository.
elek pushed a change to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
at 8666fa6 Merge remote-tracking branch 'origin/master' into HDDS-3757
This branch includes the following new commits:
new 5044208 acceptance-coverage
new 620ab91 fix .aws permissions
new e88dd11 what can we loose?
new cf68644 run only acceptance tests
new 5aff66d debug acceptance test
new 8f4cf1a try with tmate v1
new 35973d8 tmate fix
new b77854d fix permission fixing
new 690fc90 add new line
new 4268959 fix path
new 8d550c6 Revert "run only acceptance tests"
new 8d9a44d remove set +x
new f14f70c remove tmate
new 8666fa6 Merge remote-tracking branch 'origin/master' into HDDS-3757
The 14 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 02/14: fix .aws permissions
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 620ab91a0f8520d3a1093bf723c29e21e71707a2
Author: Elek Márton <el...@apache.org>
AuthorDate: Sun Jun 7 20:52:53 2020 +0200
fix .aws permissions
---
.github/workflows/post-commit.yml | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 2e5d180..16c139b 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -104,7 +104,10 @@ jobs:
git checkout ${GITHUB_SHA}
git reset --hard
- run: sudo pip install robotframework
- - run: cd /mnt/ozone && hadoop-ozone/dev-support/checks/acceptance.sh
+ - run: |
+ cd /mnt/ozone
+ mkdir .aws && sudo chown 1000 .aws
+ hadoop-ozone/dev-support/checks/acceptance.sh
env:
KEEP_IMAGE: false
- uses: actions/upload-artifact@master
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 09/14: add new line
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 690fc90f942c3c2700a9d9adf323b8c8ece8503c
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 14:39:55 2020 +0200
add new line
---
.github/workflows/post-commit.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 1f4ddef..4672117 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -41,7 +41,7 @@ jobs:
git checkout ${GITHUB_SHA}
git reset --hard
- name: run a full build
- run:
+ run: |
cd /mnt/ozone
hadoop-ozone/dev-support/checks/build.sh -Pjacoco
- run: sudo pip install robotframework
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 10/14: fix path
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 42689596c03515bcdfb0c93b48dbe524cfe3d652
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 15:01:32 2020 +0200
fix path
---
.github/workflows/post-commit.yml | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 4672117..f75aa49 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -46,9 +46,8 @@ jobs:
hadoop-ozone/dev-support/checks/build.sh -Pjacoco
- run: sudo pip install robotframework
- run: |
- cd /mnt/ozone/hadoop-ozone/dist/target/ozone-*
- sudo mkdir .aws && sudo chmod 777 .aws && sudo chown 1000 .aws
- hadoop-ozone/dev-support/checks/acceptance.sh
+ cd /mnt/ozone/hadoop-ozone/dist/target/ozone-* && sudo mkdir .aws && sudo chmod 777 .aws && sudo chown 1000 .aws
+ cd /mnt/ozone && hadoop-ozone/dev-support/checks/acceptance.sh
env:
KEEP_IMAGE: false
- uses: actions/upload-artifact@master
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 01/14: acceptance-coverage
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 50442081cf95df69dfe7c2dde0fa5ec268b1086b
Author: Elek Márton <el...@apache.org>
AuthorDate: Fri Jun 5 14:02:32 2020 +0200
acceptance-coverage
---
.github/workflows/post-commit.yml | 43 ++++++++++++----------
hadoop-ozone/dev-support/checks/acceptance.sh | 18 ++++++++-
hadoop-ozone/dev-support/checks/build.sh | 2 +-
hadoop-ozone/dist/pom.xml | 42 +++++++++++++++++++++
.../src/main/compose/ozone-csi/docker-compose.yaml | 6 +++
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 6 +++
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 4 ++
.../compose/ozone-om-ha-s3/docker-compose.yaml | 8 ++++
.../compose/ozone-topology/docker-compose.yaml | 14 +++++++
.../src/main/compose/ozone/docker-compose.yaml | 5 +++
.../compose/ozonesecure-mr/docker-compose.yaml | 4 ++
.../compose/ozonesecure-om-ha/docker-compose.yaml | 13 +++++--
.../main/compose/ozonesecure/docker-compose.yaml | 5 +++
hadoop-ozone/dist/src/main/compose/test-all.sh | 2 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 2 +
15 files changed, 147 insertions(+), 27 deletions(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index fc2dda1..2e5d180 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -93,25 +93,30 @@ jobs:
name: acceptance
runs-on: ubuntu-18.04
steps:
- - uses: actions/checkout@master
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/build.sh
- # remove image created for 'buildenv'
- - run: docker image rm $(docker images -a -q | head -1) || true
- # remove its big parent build image
- - run: docker image rm apache/ozone-build || true
- - run: sudo pip install robotframework
- - run: sudo mv ./hadoop-ozone/dist/target/ozone-* /mnt/ozone && sudo chown runner -R /mnt/ozone
- - run: cd /mnt/ozone && mkdir .aws && sudo chown 1000 .aws
- - run: cd /mnt/ozone/compose && ./test-all.sh
- env:
- KEEP_IMAGE: false
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: acceptance
- path: /mnt/ozone/compose/result
+ - uses: actions/cache@v2
+ with:
+ path: ~/.m2/repository
+ key: maven-repo-${{ hashFiles('**/pom.xml') }}
+ - run: |
+ sudo chmod 777 /mnt
+ git clone https://github.com/${GITHUB_REPOSITORY}.git /mnt/ozone
+ cd /mnt/ozone
+ git checkout ${GITHUB_SHA}
+ git reset --hard
+ - run: sudo pip install robotframework
+ - run: cd /mnt/ozone && hadoop-ozone/dev-support/checks/acceptance.sh
+ env:
+ KEEP_IMAGE: false
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: acceptance
+ path: /mnt/ozone/target/acceptance
+ - run: |
+ #Never cache local artifacts
+ rm -rf ~/.m2/repository/org/apache/hadoop/hdds
+ rm -rf ~/.m2/repository/org/apache/hadoop/ozone
+ if: always()
it-freon:
name: it-freon
runs-on: ubuntu-18.04
diff --git a/hadoop-ozone/dev-support/checks/acceptance.sh b/hadoop-ozone/dev-support/checks/acceptance.sh
index c3c1aa0..ea9f36d 100755
--- a/hadoop-ozone/dev-support/checks/acceptance.sh
+++ b/hadoop-ozone/dev-support/checks/acceptance.sh
@@ -17,19 +17,33 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR/../../.." || exit 1
REPORT_DIR=${OUTPUT_DIR:-"$DIR/../../../target/acceptance"}
-mkdir -p "$REPORT_DIR"
OZONE_VERSION=$(grep "<ozone.version>" "pom.xml" | sed 's/<[^>]*>//g'| sed 's/^[ \t]*//')
DIST_DIR="$DIR/../../dist/target/ozone-$OZONE_VERSION"
if [ ! -d "$DIST_DIR" ]; then
echo "Distribution dir is missing. Doing a full build"
- "$DIR/build.sh"
+ "$DIR/build.sh" -Pjacoco
fi
+mkdir -p "$REPORT_DIR"
+
+export HADOOP_OPTS='-javaagent:/opt/hadoop/share/jacoco/jacoco-agent.jar=destfile=/tmp/jacoco.exec,includes=org.apache.hadoop.ozone.*:org.apache.hadoop.hdds'
+
cd "$DIST_DIR/compose" || exit 1
./test-all.sh
RES=$?
cp result/* "$REPORT_DIR/"
cp "$REPORT_DIR/log.html" "$REPORT_DIR/summary.html"
+set -x
+
+EXEC_FILES=$(find "$REPORT_DIR" -name "*.jacoco.exec" | wc -l)
+if [[ $EXEC_FILES -gt 1 ]]; then
+ java -jar "$DIST_DIR/share/jacoco/jacoco-cli.jar" merge $(find "$REPORT_DIR" -name "*.jacoco.exec") --destdir "$REPORT_DIR/jacoco-combined.exec"
+fi
+
+if [[ $EXEC_FILES == 1 ]]; then
+ cp "$REPORT_DIR"/*.jacoco.exec "$REPORT_DIR/jacoco-combined.exec"
+fi
+
exit $RES
diff --git a/hadoop-ozone/dev-support/checks/build.sh b/hadoop-ozone/dev-support/checks/build.sh
index f461da1..2cdc4fe 100755
--- a/hadoop-ozone/dev-support/checks/build.sh
+++ b/hadoop-ozone/dev-support/checks/build.sh
@@ -17,5 +17,5 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR/../../.." || exit 1
export MAVEN_OPTS="-Xmx4096m"
-mvn -B -Dmaven.javadoc.skip=true -DskipTests clean install
+mvn -B -Dmaven.javadoc.skip=true -DskipTests clean install "$@"
exit $?
diff --git a/hadoop-ozone/dist/pom.xml b/hadoop-ozone/dist/pom.xml
index 0929b24..0487b75 100644
--- a/hadoop-ozone/dist/pom.xml
+++ b/hadoop-ozone/dist/pom.xml
@@ -379,7 +379,49 @@
<artifactId>hadoop-ozone-recon</artifactId>
</dependency>
</dependencies>
+ </profile>
+ <profile>
+ <id>jacoco</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>3.1.1</version>
+ <executions>
+ <execution>
+ <id>copy-jacoco-files</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>
+ target/ozone-${ozone.version}/share/jacoco
+ </outputDirectory>
+ <artifactItems>
+ <artifactItem>
+ <groupId>org.jacoco</groupId>
+ <artifactId>org.jacoco.agent</artifactId>
+ <version>0.8.5</version>
+ <classifier>runtime</classifier>
+ <destFileName>jacoco-agent.jar</destFileName>
+ </artifactItem>
+ <artifactItem>
+ <groupId>org.jacoco</groupId>
+ <artifactId>org.jacoco.cli</artifactId>
+ <version>0.8.5</version>
+ <classifier>nodeps</classifier>
+ <destFileName>jacoco-cli.jar</destFileName>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
</profile>
<profile>
diff --git a/hadoop-ozone/dist/src/main/compose/ozone-csi/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozone-csi/docker-compose.yaml
index e175dfa..1fd7ebd 100644
--- a/hadoop-ozone/dist/src/main/compose/ozone-csi/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozone-csi/docker-compose.yaml
@@ -23,6 +23,8 @@ services:
- ../..:/opt/hadoop
env_file:
- docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
ports:
- 9864
- 9882
@@ -35,6 +37,7 @@ services:
- docker-config
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
ports:
- 9874:9874
- 9862:9862
@@ -49,6 +52,7 @@ services:
- 9876:9876
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["ozone","scm"]
csi:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
@@ -56,4 +60,6 @@ services:
- ../..:/opt/hadoop
env_file:
- docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["ozone","csi"]
diff --git a/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/docker-compose.yaml
index 351e1e9..dc7261f 100644
--- a/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/docker-compose.yaml
@@ -26,6 +26,8 @@ services:
env_file:
- docker-config
- ../common-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
om:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
hostname: om
@@ -37,6 +39,7 @@ services:
environment:
WAITFOR: scm:9876
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- docker-config
- ../common-config
@@ -51,6 +54,8 @@ services:
env_file:
- ./docker-config
- ../common-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","s3g"]
scm:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
@@ -65,6 +70,7 @@ services:
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","scm"]
rm:
image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
diff --git a/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop32/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop32/docker-compose.yaml
index 35303cb..89c4f7a 100644
--- a/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop32/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop32/docker-compose.yaml
@@ -37,6 +37,7 @@ services:
environment:
WAITFOR: scm:9876
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- docker-config
- ../common-config
@@ -51,6 +52,8 @@ services:
env_file:
- ./docker-config
- ../common-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","s3g"]
scm:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
@@ -65,6 +68,7 @@ services:
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","scm"]
rm:
image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
diff --git a/hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-compose.yaml
index ff7f260..2359482 100644
--- a/hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-compose.yaml
@@ -26,6 +26,8 @@ services:
command: ["/opt/hadoop/bin/ozone","datanode"]
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
om1:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
privileged: true #required by the profiler
@@ -36,6 +38,7 @@ services:
- 9890:9872
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -49,6 +52,7 @@ services:
- 9892:9872
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -62,6 +66,7 @@ services:
- 9894:9872
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -77,6 +82,7 @@ services:
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","scm"]
s3g:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
@@ -86,5 +92,7 @@ services:
- 9878:9878
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["ozone","s3g"]
diff --git a/hadoop-ozone/dist/src/main/compose/ozone-topology/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozone-topology/docker-compose.yaml
index 7086c3f..e9d94ea 100644
--- a/hadoop-ozone/dist/src/main/compose/ozone-topology/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozone-topology/docker-compose.yaml
@@ -27,6 +27,8 @@ services:
command: ["/opt/hadoop/bin/ozone","datanode"]
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
net:
ipv4_address: 10.5.0.4
@@ -41,6 +43,8 @@ services:
command: ["/opt/hadoop/bin/ozone","datanode"]
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
net:
ipv4_address: 10.5.0.5
@@ -55,6 +59,8 @@ services:
command: ["/opt/hadoop/bin/ozone","datanode"]
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
net:
ipv4_address: 10.5.0.6
@@ -69,6 +75,8 @@ services:
command: ["/opt/hadoop/bin/ozone","datanode"]
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
net:
ipv4_address: 10.5.0.7
@@ -83,6 +91,8 @@ services:
command: ["/opt/hadoop/bin/ozone","datanode"]
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
net:
ipv4_address: 10.5.0.8
@@ -97,6 +107,8 @@ services:
command: ["/opt/hadoop/bin/ozone","datanode"]
env_file:
- ./docker-config
+ environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
net:
ipv4_address: 10.5.0.9
@@ -110,6 +122,7 @@ services:
- 9862:9862
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -128,6 +141,7 @@ services:
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-6}"
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","scm"]
networks:
net:
diff --git a/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
index 869aa05..2597612 100644
--- a/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
@@ -37,11 +37,13 @@ services:
- 9882
environment:
<<: *replication
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["ozone","datanode"]
om:
<<: *common-config
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ HADOOP_OPTS: ${HADOOP_OPTS}
<<: *replication
ports:
- 9874:9874
@@ -54,11 +56,13 @@ services:
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: ${OZONE_SAFEMODE_MIN_DATANODES:-1}
+ HADOOP_OPTS: ${HADOOP_OPTS}
<<: *replication
command: ["ozone","scm"]
s3g:
<<: *common-config
environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
<<: *replication
ports:
- 9878:9878
@@ -68,5 +72,6 @@ services:
ports:
- 9888:9888
environment:
+ HADOOP_OPTS: ${HADOOP_OPTS}
<<: *replication
command: ["ozone","recon"]
diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml
index e6d5a77..d9cbb34 100644
--- a/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml
@@ -48,6 +48,7 @@ services:
- docker-config
environment:
KERBEROS_KEYTABS: dn HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
om:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
hostname: om
@@ -61,6 +62,7 @@ services:
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP testuser
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -77,6 +79,7 @@ services:
- ./docker-config
environment:
KERBEROS_KEYTABS: s3g HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","s3g"]
scm:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
@@ -93,6 +96,7 @@ services:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
KERBEROS_KEYTABS: scm HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","scm"]
rm:
image: apache/hadoop:${HADOOP_VERSION}
diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/docker-compose.yaml
index ba0bd19..5c7f156 100644
--- a/hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/docker-compose.yaml
@@ -48,6 +48,7 @@ services:
- docker-config
environment:
KERBEROS_KEYTABS: dn HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
ozone_net:
ipv4_address: 172.25.0.102
@@ -62,6 +63,7 @@ services:
- docker-config
environment:
KERBEROS_KEYTABS: dn HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
ozone_net:
ipv4_address: 172.25.0.103
@@ -76,6 +78,7 @@ services:
- docker-config
environment:
KERBEROS_KEYTABS: dn HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
ozone_net:
ipv4_address: 172.25.0.104
@@ -91,7 +94,7 @@ services:
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP
- #HADOOP_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:18001"
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -110,7 +113,7 @@ services:
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP
- #HADOOP_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:18002"
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -129,7 +132,7 @@ services:
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP
- #HADOOP_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:18003"
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -148,6 +151,7 @@ services:
command: ["/opt/hadoop/bin/ozone","s3g"]
environment:
KERBEROS_KEYTABS: s3g HTTP testuser
+ HADOOP_OPTS: ${HADOOP_OPTS}
networks:
ozone_net:
ipv4_address: 172.25.0.114
@@ -163,7 +167,7 @@ services:
- ./docker-config
environment:
KERBEROS_KEYTABS: recon HTTP
- #HADOOP_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:18000"
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","recon"]
extra_hosts:
- "om1: 172.25.0.111"
@@ -185,6 +189,7 @@ services:
KERBEROS_KEYTABS: scm HTTP testuser testuser2
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-3}"
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","scm"]
extra_hosts:
- "om1: 172.25.0.111"
diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
index 18d5848..6c67d1e 100644
--- a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
+++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
@@ -43,6 +43,7 @@ services:
- docker-config
environment:
KERBEROS_KEYTABS: dn HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
om:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
hostname: om
@@ -54,6 +55,7 @@ services:
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
env_file:
- docker-config
command: ["/opt/hadoop/bin/ozone","om"]
@@ -70,6 +72,7 @@ services:
command: ["/opt/hadoop/bin/ozone","s3g"]
environment:
KERBEROS_KEYTABS: s3g HTTP testuser
+ HADOOP_OPTS: ${HADOOP_OPTS}
recon:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
hostname: recon
@@ -81,6 +84,7 @@ services:
- ./docker-config
environment:
KERBEROS_KEYTABS: recon HTTP
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","recon"]
scm:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
@@ -95,4 +99,5 @@ services:
KERBEROS_KEYTABS: scm HTTP testuser testuser2
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
+ HADOOP_OPTS: ${HADOOP_OPTS}
command: ["/opt/hadoop/bin/ozone","scm"]
diff --git a/hadoop-ozone/dist/src/main/compose/test-all.sh b/hadoop-ozone/dist/src/main/compose/test-all.sh
index 809123b..a84abb2 100755
--- a/hadoop-ozone/dist/src/main/compose/test-all.sh
+++ b/hadoop-ozone/dist/src/main/compose/test-all.sh
@@ -41,7 +41,7 @@ for test in $(find "$SCRIPT_DIR" -name test.sh | sort); do
echo "ERROR: Test execution of $(dirname "$test") is FAILED!!!!"
fi
RESULT_DIR="$(dirname "$test")/result"
- cp "$RESULT_DIR"/robot-*.xml "$RESULT_DIR"/docker-*.log "$RESULT_DIR"/*.out* "$ALL_RESULT_DIR"/
+ cp "$RESULT_DIR"/robot-*.xml "$RESULT_DIR"/docker-*.log "$RESULT_DIR"/*.out* "$RESULT_DIR"/*.exec "$ALL_RESULT_DIR"/
done
rebot -N "smoketests" -d "$SCRIPT_DIR/result" "$SCRIPT_DIR/result/robot-*.xml"
diff --git a/hadoop-ozone/dist/src/main/compose/testlib.sh b/hadoop-ozone/dist/src/main/compose/testlib.sh
old mode 100755
new mode 100644
index 064b19d..5948ac9
--- a/hadoop-ozone/dist/src/main/compose/testlib.sh
+++ b/hadoop-ozone/dist/src/main/compose/testlib.sh
@@ -108,7 +108,9 @@ execute_robot_test(){
local -i rc=$?
FULL_CONTAINER_NAME=$(docker-compose -f "$COMPOSE_FILE" ps | grep "_${CONTAINER}_" | head -n 1 | awk '{print $1}')
+
docker cp "$FULL_CONTAINER_NAME:$OUTPUT_PATH" "$RESULT_DIR/"
+ docker cp "$FULL_CONTAINER_NAME:/tmp/jacoco.exec" "$RESULT_DIR/$OUTPUT_NAME.jacoco.exec" || true
copy_daemon_logs
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 14/14: Merge remote-tracking branch 'origin/master'
into HDDS-3757
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 8666fa6cec20f092dd7e109eba9f598d94cddad5
Merge: f14f70c 10b6470
Author: Elek Márton <el...@apache.org>
AuthorDate: Tue Jun 9 13:04:13 2020 +0200
Merge remote-tracking branch 'origin/master' into HDDS-3757
.github/coverage-report.sh | 2 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 7 +-
.../hadoop/hdds/scm/pipeline/PipelineID.java | 6 +-
.../hadoop/hdds/conf/SimpleConfiguration.java | 2 +-
.../hdds/conf/ConfigurationReflectionUtil.java | 7 +
.../apache/hadoop/ozone/HddsDatanodeService.java | 18 +-
.../common/statemachine/DatanodeConfiguration.java | 37 +-
.../common/statemachine/DatanodeStateMachine.java | 4 +
.../common/statemachine/StateContext.java | 9 +-
.../server/ratis/ContainerStateMachine.java | 26 +-
.../container/keyvalue/KeyValueContainer.java | 3 +
.../ozone/container/keyvalue/KeyValueHandler.java | 2 +
.../keyvalue/impl/FilePerChunkStrategy.java | 28 +-
.../statemachine/TestDatanodeConfiguration.java | 85 ++
.../common/statemachine/TestStateContext.java | 69 +-
.../container/keyvalue/TestKeyValueHandler.java | 57 ++
.../keyvalue/impl/TestFilePerChunkStrategy.java | 38 +
hadoop-hdds/docs/content/gdpr/_index.md | 2 +-
.../hdds/server/http/RatisDropwizardExports.java | 36 +
.../server/http/RatisNameRewriteSampleBuilder.java | 2 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 36 +-
.../apache/hadoop/hdds/utils/db/TableConfig.java | 5 +
.../apache/hadoop/hdds/utils/db/TestRDBStore.java | 44 +
.../src/main/proto/DatanodeClientProtocol.proto | 1 +
.../interface-client/src/main/proto/proto.lock | 4 +
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 1 +
.../interface-server/src/main/proto/proto.lock | 4 +
.../container/AbstractContainerReportHandler.java | 50 +-
.../IncrementalContainerReportHandler.java | 14 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 15 +-
.../TestIncrementalContainerReportHandler.java | 45 +
.../hadoop/ozone/om/helpers/ServiceInfo.java | 19 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 12 +-
.../ozone/om/TestOzoneManagerHAMetadataOnly.java | 5 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 23 +-
.../protocolPB/OzoneManagerRequestHandler.java | 13 +-
pom.xml | 2 +-
tools/fault-injection-service/CMakeLists.txt | 114 +++
.../FileSystem/cpp_unit/TestFailureInjector.cc | 175 ++++
.../FileSystem/cpp_unit/TestFilePathFailures.cc | 138 ++++
.../FileSystem/failure_injector.cc | 201 +++++
.../FileSystem/failure_injector.h | 145 ++++
.../FileSystem/failure_injector_fs.cc | 912 +++++++++++++++++++++
.../FileSystem/failure_injector_fs.h | 176 ++++
tools/fault-injection-service/README.md | 56 ++
.../Service/cpp/failure_injector_svc_client.cc | 209 +++++
.../Service/cpp/failure_injector_svc_client.h | 51 ++
.../Service/cpp/failure_injector_svc_server.cc | 93 +++
.../Service/cpp/failure_injector_svc_server.h | 63 ++
.../Service/cpp/run_grpc_service.cc | 58 ++
.../Service/cpp/run_grpc_service.h | 36 +
.../Service/protos/failure_injection_service.proto | 68 ++
52 files changed, 3100 insertions(+), 128 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 05/14: debug acceptance test
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 5aff66d83063e6aa21eec58ec9739185ae55f2d1
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 14:03:15 2020 +0200
debug acceptance test
---
.github/workflows/post-commit.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 228e333..5db84b9 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -24,6 +24,8 @@ jobs:
with:
path: ~/.m2/repository
key: maven-repo-${{ hashFiles('**/pom.xml') }}
+ - name: Setup tmate session
+ uses: mxschmitt/action-tmate@v2
- run: |
sudo chmod 777 /mnt
git clone https://github.com/${GITHUB_REPOSITORY}.git /mnt/ozone
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 03/14: what can we loose?
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit e88dd117bdc438618b2a283169b1f84aa3e3fa73
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 01:11:56 2020 +0200
what can we loose?
---
.github/workflows/post-commit.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 16c139b..55e7c05 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -106,7 +106,7 @@ jobs:
- run: sudo pip install robotframework
- run: |
cd /mnt/ozone
- mkdir .aws && sudo chown 1000 .aws
+ sudo mkdir .aws && sudo chmod 777 .aws && sudo chown 1000 .aws
hadoop-ozone/dev-support/checks/acceptance.sh
env:
KEEP_IMAGE: false
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 11/14: Revert "run only acceptance tests"
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 8d550c63e691560ca9b473b081dc15a985456ffb
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 17:29:33 2020 +0200
Revert "run only acceptance tests"
This reverts commit cf68644571248154f25afd4b90d3e5b0bea0d78a.
---
.github/workflows/post-commit.yml | 212 ++++++++++++++++++++++++++++++++++++++
1 file changed, 212 insertions(+)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index f75aa49..b447e22 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -16,6 +16,79 @@ name: build-branch
on:
- push
jobs:
+ build:
+ name: compile
+ runs-on: ubuntu-18.04
+ steps:
+ - uses: actions/checkout@master
+ - uses: ./.github/buildenv
+ with:
+ args: ./hadoop-ozone/dev-support/checks/build.sh
+ rat:
+ name: rat
+ runs-on: ubuntu-18.04
+ steps:
+ - uses: actions/checkout@master
+ - uses: ./.github/buildenv
+ with:
+ args: ./hadoop-ozone/dev-support/checks/rat.sh
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: rat
+ path: target/rat
+ author:
+ name: author
+ runs-on: ubuntu-18.04
+ steps:
+ - uses: actions/checkout@master
+ - uses: ./.github/buildenv
+ with:
+ args: ./hadoop-ozone/dev-support/checks/author.sh
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: author
+ path: target/author
+ unit:
+ name: unit
+ runs-on: ubuntu-18.04
+ steps:
+ - uses: actions/checkout@master
+ - uses: ./.github/buildenv
+ with:
+ args: ./hadoop-ozone/dev-support/checks/unit.sh
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: unit
+ path: target/unit
+ checkstyle:
+ name: checkstyle
+ runs-on: ubuntu-18.04
+ steps:
+ - uses: actions/checkout@master
+ - uses: ./.github/buildenv
+ with:
+ args: ./hadoop-ozone/dev-support/checks/checkstyle.sh
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: checkstyle
+ path: target/checkstyle
+ findbugs:
+ name: findbugs
+ runs-on: ubuntu-18.04
+ steps:
+ - uses: actions/checkout@master
+ - uses: ./.github/buildenv
+ with:
+ args: ./hadoop-ozone/dev-support/checks/findbugs.sh
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: findbugs
+ path: target/findbugs
acceptance:
name: acceptance
runs-on: ubuntu-18.04
@@ -60,3 +133,142 @@ jobs:
rm -rf ~/.m2/repository/org/apache/hadoop/hdds
rm -rf ~/.m2/repository/org/apache/hadoop/ozone
if: always()
+ it-freon:
+ name: it-freon
+ runs-on: ubuntu-18.04
+ needs:
+ - build
+ steps:
+ - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
+ - uses: actions/checkout@master
+ with:
+ path: mnt/ozone
+ - uses: ./mnt/ozone/.github/buildenv
+ with:
+ args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pfreon
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: it-freon
+ path: mnt/ozone/target/integration
+ it-filesystem:
+ name: it-filesystem
+ runs-on: ubuntu-18.04
+ needs:
+ - build
+ steps:
+ - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
+ - uses: actions/checkout@master
+ with:
+ path: mnt/ozone
+ - uses: ./mnt/ozone/.github/buildenv
+ with:
+ args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pfilesystem
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: it-filesystem
+ path: mnt/ozone/target/integration
+ it-filesystem-contract:
+ name: it-filesystem-contract
+ runs-on: ubuntu-18.04
+ needs:
+ - build
+ steps:
+ - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
+ - uses: actions/checkout@master
+ with:
+ path: mnt/ozone
+ - uses: ./mnt/ozone/.github/buildenv
+ with:
+ args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pfilesystem-contract
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: it-filesystem-contract
+ path: mnt/ozone/target/integration
+ it-client:
+ name: it-client
+ runs-on: ubuntu-18.04
+ needs:
+ - build
+ steps:
+ - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
+ - uses: actions/checkout@master
+ with:
+ path: mnt/ozone
+ - uses: ./mnt/ozone/.github/buildenv
+ with:
+ args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pclient
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: it-client
+ path: mnt/ozone/target/integration
+ it-hdds-om:
+ name: it-hdds-om
+ runs-on: ubuntu-18.04
+ needs:
+ - build
+ steps:
+ - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
+ - uses: actions/checkout@master
+ with:
+ path: mnt/ozone
+ - uses: ./mnt/ozone/.github/buildenv
+ with:
+ args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Phdds-om
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: it-hdds-om
+ path: mnt/ozone/target/integration
+ it-ozone:
+ name: it-ozone
+ runs-on: ubuntu-18.04
+ needs:
+ - build
+ steps:
+ - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
+ - uses: actions/checkout@master
+ with:
+ path: mnt/ozone
+ - uses: ./mnt/ozone/.github/buildenv
+ with:
+ args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pozone
+ - uses: actions/upload-artifact@master
+ if: always()
+ with:
+ name: it-ozone
+ path: mnt/ozone/target/integration
+ coverage:
+ name: coverage
+ runs-on: ubuntu-18.04
+ needs:
+ - it-ozone
+ - it-hdds-om
+ - it-client
+ - it-filesystem-contract
+ - it-filesystem
+ - it-freon
+ - unit
+ steps:
+ - uses: actions/checkout@v2
+ - uses: ./.github/buildenv
+ with:
+ args: ./hadoop-ozone/dev-support/checks/build.sh
+ - uses: actions/download-artifact@v2
+ with:
+ path: target/artifacts
+ - run: ./.github/coverage-report.sh
+ - uses: ./.github/buildenv
+ if: github.repository == 'apache/hadoop-ozone'
+ with:
+ args: ./hadoop-ozone/dev-support/checks/sonar.sh
+ env:
+ SONAR_TOKEN: ${{ secrets.SONARCLOUD_TOKEN }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ - uses: actions/upload-artifact@master
+ with:
+ name: coverage
+ path: target/coverage
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 07/14: tmate fix
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 35973d809876039df3086b8e9540b960c49e464d
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 14:26:54 2020 +0200
tmate fix
---
.github/workflows/post-commit.yml | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index c32028f..4184bd1 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -25,7 +25,14 @@ jobs:
path: ~/.m2/repository
key: maven-repo-${{ hashFiles('**/pom.xml') }}
- name: Setup tmate session
- uses: mxschmitt/action-tmate@v1
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y tmate openssh-client
+ echo -e 'y\n'|ssh-keygen -q -t rsa -N "" -f ~/.ssh/id_rsa
+ tmate -S /tmp/tmate.sock new-session -d
+ tmate -S /tmp/tmate.sock wait tmate-ready
+ tmate -S /tmp/tmate.sock display -p '#{tmate_ssh}'
+ tmate -S /tmp/tmate.sock display -p '#{tmate_web}'
- run: |
sudo chmod 777 /mnt
git clone https://github.com/${GITHUB_REPOSITORY}.git /mnt/ozone
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 08/14: fix permission fixing
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit b77854df1785c9ea9dcd616f5b13a84543ce69fb
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 14:33:04 2020 +0200
fix permission fixing
---
.github/workflows/post-commit.yml | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 4184bd1..1f4ddef 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -33,15 +33,20 @@ jobs:
tmate -S /tmp/tmate.sock wait tmate-ready
tmate -S /tmp/tmate.sock display -p '#{tmate_ssh}'
tmate -S /tmp/tmate.sock display -p '#{tmate_web}'
- - run: |
+ - name: checkout to /mnt/ozone
+ run: |
sudo chmod 777 /mnt
git clone https://github.com/${GITHUB_REPOSITORY}.git /mnt/ozone
cd /mnt/ozone
git checkout ${GITHUB_SHA}
git reset --hard
+ - name: run a full build
+ run:
+ cd /mnt/ozone
+ hadoop-ozone/dev-support/checks/build.sh -Pjacoco
- run: sudo pip install robotframework
- run: |
- cd /mnt/ozone
+ cd /mnt/ozone/hadoop-ozone/dist/target/ozone-*
sudo mkdir .aws && sudo chmod 777 .aws && sudo chown 1000 .aws
hadoop-ozone/dev-support/checks/acceptance.sh
env:
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 12/14: remove set +x
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 8d9a44df077cac483a05b822b3c429fb9851d052
Author: Elek Márton <el...@apache.org>
AuthorDate: Tue Jun 9 12:48:32 2020 +0200
remove set +x
---
hadoop-ozone/dev-support/checks/acceptance.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/hadoop-ozone/dev-support/checks/acceptance.sh b/hadoop-ozone/dev-support/checks/acceptance.sh
index ea9f36d..a511bf8 100755
--- a/hadoop-ozone/dev-support/checks/acceptance.sh
+++ b/hadoop-ozone/dev-support/checks/acceptance.sh
@@ -35,7 +35,6 @@ cd "$DIST_DIR/compose" || exit 1
RES=$?
cp result/* "$REPORT_DIR/"
cp "$REPORT_DIR/log.html" "$REPORT_DIR/summary.html"
-set -x
EXEC_FILES=$(find "$REPORT_DIR" -name "*.jacoco.exec" | wc -l)
if [[ $EXEC_FILES -gt 1 ]]; then
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 13/14: remove tmate
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit f14f70c3f83ee2f76190deb6ce68cd4791ffb5e7
Author: Elek Márton <el...@apache.org>
AuthorDate: Tue Jun 9 12:49:33 2020 +0200
remove tmate
---
.github/workflows/post-commit.yml | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index b447e22..84e4c2e 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -97,15 +97,6 @@ jobs:
with:
path: ~/.m2/repository
key: maven-repo-${{ hashFiles('**/pom.xml') }}
- - name: Setup tmate session
- run: |
- sudo apt-get update
- sudo apt-get install -y tmate openssh-client
- echo -e 'y\n'|ssh-keygen -q -t rsa -N "" -f ~/.ssh/id_rsa
- tmate -S /tmp/tmate.sock new-session -d
- tmate -S /tmp/tmate.sock wait tmate-ready
- tmate -S /tmp/tmate.sock display -p '#{tmate_ssh}'
- tmate -S /tmp/tmate.sock display -p '#{tmate_web}'
- name: checkout to /mnt/ozone
run: |
sudo chmod 777 /mnt
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 06/14: try with tmate v1
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 8f4cf1ac1a740fe076079d549f9b499a4627a05c
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 14:12:27 2020 +0200
try with tmate v1
---
.github/workflows/post-commit.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 5db84b9..c32028f 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -25,7 +25,7 @@ jobs:
path: ~/.m2/repository
key: maven-repo-${{ hashFiles('**/pom.xml') }}
- name: Setup tmate session
- uses: mxschmitt/action-tmate@v2
+ uses: mxschmitt/action-tmate@v1
- run: |
sudo chmod 777 /mnt
git clone https://github.com/${GITHUB_REPOSITORY}.git /mnt/ozone
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 04/14: run only acceptance tests
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3757
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit cf68644571248154f25afd4b90d3e5b0bea0d78a
Author: Elek Márton <el...@apache.org>
AuthorDate: Mon Jun 8 14:02:20 2020 +0200
run only acceptance tests
---
.github/workflows/post-commit.yml | 212 --------------------------------------
1 file changed, 212 deletions(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index 55e7c05..228e333 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -16,79 +16,6 @@ name: build-branch
on:
- push
jobs:
- build:
- name: compile
- runs-on: ubuntu-18.04
- steps:
- - uses: actions/checkout@master
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/build.sh
- rat:
- name: rat
- runs-on: ubuntu-18.04
- steps:
- - uses: actions/checkout@master
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/rat.sh
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: rat
- path: target/rat
- author:
- name: author
- runs-on: ubuntu-18.04
- steps:
- - uses: actions/checkout@master
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/author.sh
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: author
- path: target/author
- unit:
- name: unit
- runs-on: ubuntu-18.04
- steps:
- - uses: actions/checkout@master
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/unit.sh
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: unit
- path: target/unit
- checkstyle:
- name: checkstyle
- runs-on: ubuntu-18.04
- steps:
- - uses: actions/checkout@master
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/checkstyle.sh
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: checkstyle
- path: target/checkstyle
- findbugs:
- name: findbugs
- runs-on: ubuntu-18.04
- steps:
- - uses: actions/checkout@master
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/findbugs.sh
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: findbugs
- path: target/findbugs
acceptance:
name: acceptance
runs-on: ubuntu-18.04
@@ -120,142 +47,3 @@ jobs:
rm -rf ~/.m2/repository/org/apache/hadoop/hdds
rm -rf ~/.m2/repository/org/apache/hadoop/ozone
if: always()
- it-freon:
- name: it-freon
- runs-on: ubuntu-18.04
- needs:
- - build
- steps:
- - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
- - uses: actions/checkout@master
- with:
- path: mnt/ozone
- - uses: ./mnt/ozone/.github/buildenv
- with:
- args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pfreon
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: it-freon
- path: mnt/ozone/target/integration
- it-filesystem:
- name: it-filesystem
- runs-on: ubuntu-18.04
- needs:
- - build
- steps:
- - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
- - uses: actions/checkout@master
- with:
- path: mnt/ozone
- - uses: ./mnt/ozone/.github/buildenv
- with:
- args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pfilesystem
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: it-filesystem
- path: mnt/ozone/target/integration
- it-filesystem-contract:
- name: it-filesystem-contract
- runs-on: ubuntu-18.04
- needs:
- - build
- steps:
- - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
- - uses: actions/checkout@master
- with:
- path: mnt/ozone
- - uses: ./mnt/ozone/.github/buildenv
- with:
- args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pfilesystem-contract
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: it-filesystem-contract
- path: mnt/ozone/target/integration
- it-client:
- name: it-client
- runs-on: ubuntu-18.04
- needs:
- - build
- steps:
- - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
- - uses: actions/checkout@master
- with:
- path: mnt/ozone
- - uses: ./mnt/ozone/.github/buildenv
- with:
- args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pclient
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: it-client
- path: mnt/ozone/target/integration
- it-hdds-om:
- name: it-hdds-om
- runs-on: ubuntu-18.04
- needs:
- - build
- steps:
- - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
- - uses: actions/checkout@master
- with:
- path: mnt/ozone
- - uses: ./mnt/ozone/.github/buildenv
- with:
- args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Phdds-om
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: it-hdds-om
- path: mnt/ozone/target/integration
- it-ozone:
- name: it-ozone
- runs-on: ubuntu-18.04
- needs:
- - build
- steps:
- - run: sudo mkdir mnt && sudo mount --bind /mnt `pwd`/mnt && sudo chmod 777 mnt
- - uses: actions/checkout@master
- with:
- path: mnt/ozone
- - uses: ./mnt/ozone/.github/buildenv
- with:
- args: ./mnt/ozone/hadoop-ozone/dev-support/checks/integration.sh -Pozone
- - uses: actions/upload-artifact@master
- if: always()
- with:
- name: it-ozone
- path: mnt/ozone/target/integration
- coverage:
- name: coverage
- runs-on: ubuntu-18.04
- needs:
- - it-ozone
- - it-hdds-om
- - it-client
- - it-filesystem-contract
- - it-filesystem
- - it-freon
- - unit
- steps:
- - uses: actions/checkout@v2
- - uses: ./.github/buildenv
- with:
- args: ./hadoop-ozone/dev-support/checks/build.sh
- - uses: actions/download-artifact@v2
- with:
- path: target/artifacts
- - run: ./.github/coverage-report.sh
- - uses: ./.github/buildenv
- if: github.repository == 'apache/hadoop-ozone'
- with:
- args: ./hadoop-ozone/dev-support/checks/sonar.sh
- env:
- SONAR_TOKEN: ${{ secrets.SONARCLOUD_TOKEN }}
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- - uses: actions/upload-artifact@master
- with:
- name: coverage
- path: target/coverage
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org