You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/02/16 21:09:35 UTC

[GitHub] [ozone] adoroszlai opened a new pull request #1928: HDDS-4741. Modularize upgrade test

adoroszlai opened a new pull request #1928:
URL: https://github.com/apache/ozone/pull/1928


   ## What changes were proposed in this pull request?
   
   As of now, `upgrade` acceptance test verifies upgrade from Ozone 0.5.0 to "current", ie. the commit being built.  It has an `OZONE_UPGRADE_TO` variable, but its value is not really relevant to which version is run after the "upgrade".
   
   This PR introduces the following changes:
   
    1. Test upgrade from multiple earlier versions, eg. from 0.5.0 or 1.0.0.
    2. Extract parts of the `upgrade` test script to two libraries (the existing main lib and an upgrade-specific one).
    3. Split `upgrade` test script:
       * `upgrade_to_current`: test upgrade from a release to the current binaries,
       * `upgrade_to_release`: test upgrade from one release to a later one.
    4. Improve the scripts which define version-specific behavior: instead of just declaring variables, define functions to activate/deactivate behavior of a specific version.
    5. Introduce "logical version" to be able to upgrade across multiple versions (eg. from 0.5.0 to 1.1.0).
   
   https://issues.apache.org/jira/browse/HDDS-4741
   
   ## How was this patch tested?
   
   Regular CI now runs `upgrade_to_current.sh` from both 0.5.0 and 1.0.0:
   https://github.com/adoroszlai/hadoop-ozone/runs/1912441301
   
   Also tested `upgrade_to_release.sh` manually.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai commented on a change in pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#discussion_r580359668



##########
File path: hadoop-ozone/dist/src/main/compose/upgrade/test.sh
##########
@@ -14,60 +14,28 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-export COMPOSE_DIR
-
-: "${OZONE_REPLICATION_FACTOR:=3}"
-: "${OZONE_UPGRADE_FROM:="0.5.0"}"
-: "${OZONE_UPGRADE_TO:="1.0.0"}"
-: "${OZONE_VOLUME:="${COMPOSE_DIR}/data"}"
-
-export OZONE_VOLUME
-
-mkdir -p "${OZONE_VOLUME}"/{dn1,dn2,dn3,om,recon,s3g,scm}
-
-if [[ -n "${OZONE_VOLUME_OWNER}" ]]; then
-  current_user=$(whoami)
-  if [[ "${OZONE_VOLUME_OWNER}" != "${current_user}" ]]; then
-    chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}" \
-      || sudo chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}"
+set -x
+SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )
+ALL_RESULT_DIR="$SCRIPT_DIR/result"
+mkdir -p "$ALL_RESULT_DIR"
+rm "$ALL_RESULT_DIR/*" || true

Review comment:
       Nice catch.  BTW, this script is copied from `hadoop-ozone/dist/src/main/compose/ozone-mr/test.sh`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] elek commented on a change in pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#discussion_r585542176



##########
File path: hadoop-ozone/dist/src/main/compose/versions/README.md
##########
@@ -13,3 +13,11 @@
 -->
 
 The scripts in this directory define version-specific behavior required for [`testlib.sh`](../../testlib.sh).  For example the `ozone admin` command was renamed from `ozone scmcli` in 1.0.0.

Review comment:
       ```suggestion
   The scripts in this directory define version-specific behavior required for [`testlib.sh`](../testlib.sh).  For example the `ozone admin` command was renamed from `ozone scmcli` in 1.0.0.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai commented on pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#issuecomment-788054762


   @avijayanhwx @elek @fapifta can you please review?  @errose28 is OK with it and would like to work on top of it in the upgrade branch.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] errose28 edited a comment on pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
errose28 edited a comment on pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#issuecomment-785162750


   Thanks for working on this @adoroszlai. Tried it out locally and it ran smoothly. Changes LGTM +1.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] avijayanhwx commented on pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#issuecomment-789181165


   Thanks for the patch @adoroszlai, and the reviews @errose28 & @elek. I am merging this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] avijayanhwx merged pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
avijayanhwx merged pull request #1928:
URL: https://github.com/apache/ozone/pull/1928


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] errose28 commented on a change in pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
errose28 commented on a change in pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#discussion_r580295465



##########
File path: hadoop-ozone/dist/src/main/compose/upgrade/test.sh
##########
@@ -14,60 +14,28 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-export COMPOSE_DIR
-
-: "${OZONE_REPLICATION_FACTOR:=3}"
-: "${OZONE_UPGRADE_FROM:="0.5.0"}"
-: "${OZONE_UPGRADE_TO:="1.0.0"}"
-: "${OZONE_VOLUME:="${COMPOSE_DIR}/data"}"
-
-export OZONE_VOLUME
-
-mkdir -p "${OZONE_VOLUME}"/{dn1,dn2,dn3,om,recon,s3g,scm}
-
-if [[ -n "${OZONE_VOLUME_OWNER}" ]]; then
-  current_user=$(whoami)
-  if [[ "${OZONE_VOLUME_OWNER}" != "${current_user}" ]]; then
-    chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}" \
-      || sudo chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}"
+set -x

Review comment:
       Printing all the commands as they run seems like a chunk of extra output that is not very helpful. Is there an advantage to this?

##########
File path: hadoop-ozone/dist/src/main/compose/upgrade/test.sh
##########
@@ -14,60 +14,28 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-export COMPOSE_DIR
-
-: "${OZONE_REPLICATION_FACTOR:=3}"
-: "${OZONE_UPGRADE_FROM:="0.5.0"}"
-: "${OZONE_UPGRADE_TO:="1.0.0"}"
-: "${OZONE_VOLUME:="${COMPOSE_DIR}/data"}"
-
-export OZONE_VOLUME
-
-mkdir -p "${OZONE_VOLUME}"/{dn1,dn2,dn3,om,recon,s3g,scm}
-
-if [[ -n "${OZONE_VOLUME_OWNER}" ]]; then
-  current_user=$(whoami)
-  if [[ "${OZONE_VOLUME_OWNER}" != "${current_user}" ]]; then
-    chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}" \
-      || sudo chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}"
+set -x
+SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )
+ALL_RESULT_DIR="$SCRIPT_DIR/result"
+mkdir -p "$ALL_RESULT_DIR"
+rm "$ALL_RESULT_DIR/*" || true

Review comment:
       Glob expansion won't happen in double quotes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] avijayanhwx commented on pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#issuecomment-780121124


   cc @errose28 for review.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai commented on a change in pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#discussion_r580360115



##########
File path: hadoop-ozone/dist/src/main/compose/upgrade/test.sh
##########
@@ -14,60 +14,28 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-export COMPOSE_DIR
-
-: "${OZONE_REPLICATION_FACTOR:=3}"
-: "${OZONE_UPGRADE_FROM:="0.5.0"}"
-: "${OZONE_UPGRADE_TO:="1.0.0"}"
-: "${OZONE_VOLUME:="${COMPOSE_DIR}/data"}"
-
-export OZONE_VOLUME
-
-mkdir -p "${OZONE_VOLUME}"/{dn1,dn2,dn3,om,recon,s3g,scm}
-
-if [[ -n "${OZONE_VOLUME_OWNER}" ]]; then
-  current_user=$(whoami)
-  if [[ "${OZONE_VOLUME_OWNER}" != "${current_user}" ]]; then
-    chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}" \
-      || sudo chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}"
+set -x

Review comment:
       No, I think it is just leftover debug.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai commented on a change in pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#discussion_r585557295



##########
File path: hadoop-ozone/dist/src/main/compose/versions/README.md
##########
@@ -13,3 +13,11 @@
 -->
 
 The scripts in this directory define version-specific behavior required for [`testlib.sh`](../../testlib.sh).  For example the `ozone admin` command was renamed from `ozone scmcli` in 1.0.0.

Review comment:
       Thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai commented on pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#issuecomment-784330923


   > Also it looks like there is an extra robot log and report getting placed in the upgrade directory (outside the results folder). Looks like log and report for 0.5.0 first gets put there, and then log and report for 1.0.0 overwrites it. Is this supposed to happen?
   
   Thanks for pointing this out.  It turns out not specific to the `upgrade` test or this change.  It happens for each "subtest" on `master`, too.  Example (from [here](https://github.com/apache/ozone/runs/1960981044#step:7:107)):
   
   ```
   Log:     /mnt/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/compatibility/result/log.html
   Report:  /mnt/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/compatibility/result/report.html
   Output:  /mnt/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/result/compatibility.xml
   Log:     /mnt/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/log.html
   Report:  /mnt/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/report.html
   ```
   
   (The second log and report are the ones created in the same way as you described.)
   
   Since the fix is [pretty simple](https://github.com/apache/ozone/pull/1928/commits/749d47121886d282e55a5e5b1f1de47031bfb330), I incuded it here, instead of creating a separate issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] errose28 commented on pull request #1928: HDDS-4741. Modularize upgrade test

Posted by GitBox <gi...@apache.org>.
errose28 commented on pull request #1928:
URL: https://github.com/apache/ozone/pull/1928#issuecomment-785162750


   Thanks for working on this @adoroszlai. Changes LGTM +1.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org