You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "siddhantsangwan (via GitHub)" <gi...@apache.org> on 2024/12/03 10:27:50 UTC
[PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
siddhantsangwan opened a new pull request, #7524:
URL: https://github.com/apache/ozone/pull/7524
## What changes were proposed in this pull request?
Please check the jira for a description of the problem. The solution proposed here is to change `isKey()` to `getIsKey()` in `KeyEntityInfoProtoWrapper`.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11848
## How was this patch tested?
Tested manually.
Before the fix, calling the listKeys endpoint resulted in the described bug:
```
http://localhost:9888/api/v1/keys/listKeys?startPrefix=/vol1/bucket2&limit=1000
Conflicting/ambiguous property name definitions (implicit name 'key'): found multiple explicit names: [key, isKey], but also implicit accessor: [method org.apache.hadoop.ozone.recon.api.types.KeyEntityInfoProtoWrapper#setKey(java.lang.String)][visible=true,ignore=false,explicitName=false] (through reference chain: org.apache.hadoop.ozone.recon.api.types.ListKeysResponse["keys"])
```
After the fix, it works as expected:
```
http://localhost:9888/api/v1/keys/listKeys?startPrefix=/vol1/bucket2&limit=1000
{
"status": "OK",
"path": "/vol1/bucket2",
"replicatedDataSize": 4068,
"unReplicatedDataSize": 4068,
"lastKey": "/-9223372036854775552/-9223372036854775040/-9223372036854775040/key",
"keys": [
{
"parentId": -9.223372036854775e+18,
"volumeName": "vol1",
"bucketName": "bucket2",
"keyName": "key",
"key": "/-9223372036854775552/-9223372036854775040/-9223372036854775040/key",
"path": "vol1/bucket2/key",
"replicatedSize": 4068,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"minimumNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1733221329416,
"size": 4068,
"isKey": true,
"modificationTime": 1733221330013
}
]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1869358604
##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -57,6 +59,21 @@ Check http return code
Should contain ${result} 200
END
+Check if the listKeys api responds OK
+ [Arguments] ${volume} ${bucket}
+ ${result} = Curl "${API_ENDPOINT_URL}/keys/listKeys?startPrefix=/${volume}/${bucket}&limit=1000"
+ Should contain ${result} "OK"
+ Should contain ${result} "keys"
+ Should contain ${result} "${volume}"
+ Should contain ${result} "${bucket}"
+
+Curl
+ [Arguments] ${url}
+ ${curl_command} = Set Variable curl -LSs ${url}
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Set Variable ${curl_command} curl --negotiate -u : -LSs ${url}
Review Comment:
I added code to kinit and to use curl directly without any conditions. It should hopefully pass now; I checked out a separate branch with the same code and ran CI successfully on my fork (just for testing): https://github.com/siddhantsangwan/ozone/actions/runs/12157989638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1868807358
##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -57,6 +59,21 @@ Check http return code
Should contain ${result} 200
END
+Check if the listKeys api responds OK
+ [Arguments] ${volume} ${bucket}
+ ${result} = Curl "${API_ENDPOINT_URL}/keys/listKeys?startPrefix=/${volume}/${bucket}&limit=1000"
+ Should contain ${result} "OK"
+ Should contain ${result} "keys"
+ Should contain ${result} "${volume}"
+ Should contain ${result} "${bucket}"
+
+Curl
+ [Arguments] ${url}
+ ${curl_command} = Set Variable curl -LSs ${url}
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Set Variable ${curl_command} curl --negotiate -u : -LSs ${url}
Review Comment:
[`Set Variable`](https://robotframework.org/robotframework/6.1.1/libraries/BuiltIn.html#Set%20Variable) returns the value, does not modify its first argument.
This can be fixed by using [`Set Variable If`](https://robotframework.org/robotframework/6.1.1/libraries/BuiltIn.html#Set%20Variable%20If):
```suggestion
${curl_command} = Set Variable If '${SECURITY_ENABLED}' == 'true' curl --negotiate -u : -LSs ${url} curl -LSs ${url}
```
But other test cases use `curl --negotiate -u :` unconditionally, so maybe we don't need the conditional logic at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #7524:
URL: https://github.com/apache/ozone/pull/7524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7524:
URL: https://github.com/apache/ozone/pull/7524#issuecomment-2517756485
Thanks @siddhantsangwan for the fix, @devmadhuu for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1868089834
##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -136,3 +153,13 @@ Check normal api access
kinit as non admin
Check http return code ${NON_ADMIN_API_ENDPOINT_URL} 200
+
+Check listKeys api works
+ kinit as ozone admin
+ # Create volume and bucket
+ Execute ozone sh volume create ${VOLUME}
+ Execute ozone sh bucket create ${VOLUME}/${BUCKET}
+ Freon OCKG n=2 args=-s 10 -v ${VOLUME} -b ${BUCKET}
+
+ # Wait until Recon picks up the keys
+ Wait Until Keyword Succeeds 90sec 10sec Check if the listKeys api responds OK ${VOLUME} ${BUCKET}
Review Comment:
Thanks a lot @siddhantsangwan for adding the test. Unfortunately it fails with:
```
VOLUME_ALREADY_EXISTS Volume already exists
```
Instead of starting from scratch, we could add the `listKeys` request to existing test case `Check if Recon picks up OM data`, relying on the volume, bucket and keys created there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1868766285
##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -136,3 +153,13 @@ Check normal api access
kinit as non admin
Check http return code ${NON_ADMIN_API_ENDPOINT_URL} 200
+
+Check listKeys api works
+ kinit as ozone admin
+ # Create volume and bucket
+ Execute ozone sh volume create ${VOLUME}
+ Execute ozone sh bucket create ${VOLUME}/${BUCKET}
+ Freon OCKG n=2 args=-s 10 -v ${VOLUME} -b ${BUCKET}
+
+ # Wait until Recon picks up the keys
+ Wait Until Keyword Succeeds 90sec 10sec Check if the listKeys api responds OK ${VOLUME} ${BUCKET}
Review Comment:
Thanks, I've the updated the PR with the suggested change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7524:
URL: https://github.com/apache/ozone/pull/7524#issuecomment-2514191996
Thanks @siddhantsangwan for the fix. Can we add a test case in `hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org