You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "adoroszlai (via GitHub)" <gi...@apache.org> on 2026/02/15 21:22:32 UTC
[PR] HDDS-14635. kubernetes check fails after restarting datanodes [ozone]
adoroszlai opened a new pull request, #9772:
URL: https://github.com/apache/ozone/pull/9772
## What changes were proposed in this pull request?
Fix failure in `kubernetes` check due to read failure after datanodes are restarted:
```
Ozone Client Key Validator | FAIL |
255 != 0
```
- Wait for pipeline to be open before trying to read.
- Increase client retry count and wait interval.
- Fix log configuration in `k8s` config to allow suppressing logger for CLI commands (respect `hadoop.root.logger` property). This is required for `ozone admin --json` command output to be valid JSON.
https://issues.apache.org/jira/browse/HDDS-14635
## How was this patch tested?
`ozone/test.sh` passed 10x:
https://github.com/adoroszlai/ozone/actions/runs/22041622888
All `kubernetes` check passed 10x:
https://github.com/adoroszlai/ozone/actions/runs/22042121647
Regular CI:
https://github.com/adoroszlai/ozone/actions/runs/22042489756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14635. kubernetes check fails after restarting datanodes [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #9772:
URL: https://github.com/apache/ozone/pull/9772
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14635. kubernetes check fails after restarting datanodes [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #9772:
URL: https://github.com/apache/ozone/pull/9772#discussion_r2815252137
##########
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh:
##########
@@ -50,6 +50,23 @@ wait_for_startup(){
fi
}
+wait_for_pipeline() {
+ retry assert_pipeline_exists
+}
+
+execute_command_in_container() {
+ CONTAINER="$1"
+ shift 1 #Remove first argument which was the container name
+
+ kubectl exec "${CONTAINER}" -- "$@"
+}
+
+assert_pipeline_exists() {
+ local count
+ count=$(execute_command_in_container scm-0 ozone admin pipeline list --state OPEN --filter-by-factor THREE --json | jq -r 'length')
Review Comment:
kubernetes examples start 3 datanodes, EC pipelines are not tested.
https://github.com/apache/ozone/blob/2a94b52d8924b093963c99bc2d9d25a43dd0641e/hadoop-ozone/dist/src/main/k8s/definitions/ozone/datanode-ss.yaml#L28
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14635. kubernetes check fails after restarting datanodes [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9772:
URL: https://github.com/apache/ozone/pull/9772#discussion_r2814070384
##########
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh:
##########
@@ -50,6 +50,23 @@ wait_for_startup(){
fi
}
+wait_for_pipeline() {
+ retry assert_pipeline_exists
+}
+
+execute_command_in_container() {
+ CONTAINER="$1"
+ shift 1 #Remove first argument which was the container name
+
+ kubectl exec "${CONTAINER}" -- "$@"
+}
+
+assert_pipeline_exists() {
+ local count
+ count=$(execute_command_in_container scm-0 ozone admin pipeline list --state OPEN --filter-by-factor THREE --json | jq -r 'length')
Review Comment:
Can it be the case that there are only EC pipelines? Any reason why we're waiting only for ratis pipelines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14635. kubernetes check fails after restarting datanodes [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #9772:
URL: https://github.com/apache/ozone/pull/9772#discussion_r2815252137
##########
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh:
##########
@@ -50,6 +50,23 @@ wait_for_startup(){
fi
}
+wait_for_pipeline() {
+ retry assert_pipeline_exists
+}
+
+execute_command_in_container() {
+ CONTAINER="$1"
+ shift 1 #Remove first argument which was the container name
+
+ kubectl exec "${CONTAINER}" -- "$@"
+}
+
+assert_pipeline_exists() {
+ local count
+ count=$(execute_command_in_container scm-0 ozone admin pipeline list --state OPEN --filter-by-factor THREE --json | jq -r 'length')
Review Comment:
kubernetes examples start 3 datanodes, EC pipelines are not tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14635. kubernetes check fails after restarting datanodes [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #9772:
URL: https://github.com/apache/ozone/pull/9772#discussion_r2815252137
##########
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh:
##########
@@ -50,6 +50,23 @@ wait_for_startup(){
fi
}
+wait_for_pipeline() {
+ retry assert_pipeline_exists
+}
+
+execute_command_in_container() {
+ CONTAINER="$1"
+ shift 1 #Remove first argument which was the container name
+
+ kubectl exec "${CONTAINER}" -- "$@"
+}
+
+assert_pipeline_exists() {
+ local count
+ count=$(execute_command_in_container scm-0 ozone admin pipeline list --state OPEN --filter-by-factor THREE --json | jq -r 'length')
Review Comment:
kubernetes examples start 1 or 3 datanodes, EC pipelines are not tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14635. kubernetes check fails after restarting datanodes [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #9772:
URL: https://github.com/apache/ozone/pull/9772#issuecomment-3913123216
Thanks @sarvekshayr, @yandrey321 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org