You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "siddhantsangwan (via GitHub)" <gi...@apache.org> on 2024/12/03 10:27:50 UTC

[PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

siddhantsangwan opened a new pull request, #7524:
URL: https://github.com/apache/ozone/pull/7524

   ## What changes were proposed in this pull request?
   
   Please check the jira for a description of the problem. The solution proposed here is to change `isKey()` to `getIsKey()` in `KeyEntityInfoProtoWrapper`.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-11848
   
   ## How was this patch tested?
   
   Tested manually. 
   
   Before the fix, calling the listKeys endpoint resulted in the described bug:
   ```
   http://localhost:9888/api/v1/keys/listKeys?startPrefix=/vol1/bucket2&limit=1000
   
   Conflicting/ambiguous property name definitions (implicit name 'key'): found multiple explicit names: [key, isKey], but also implicit accessor: [method org.apache.hadoop.ozone.recon.api.types.KeyEntityInfoProtoWrapper#setKey(java.lang.String)][visible=true,ignore=false,explicitName=false] (through reference chain: org.apache.hadoop.ozone.recon.api.types.ListKeysResponse["keys"])
   ```
   
   After the fix, it works as expected:
   ```
   http://localhost:9888/api/v1/keys/listKeys?startPrefix=/vol1/bucket2&limit=1000
   
   {
     "status": "OK",
     "path": "/vol1/bucket2",
     "replicatedDataSize": 4068,
     "unReplicatedDataSize": 4068,
     "lastKey": "/-9223372036854775552/-9223372036854775040/-9223372036854775040/key",
     "keys": [
       {
         "parentId": -9.223372036854775e+18,
         "volumeName": "vol1",
         "bucketName": "bucket2",
         "keyName": "key",
         "key": "/-9223372036854775552/-9223372036854775040/-9223372036854775040/key",
         "path": "vol1/bucket2/key",
         "replicatedSize": 4068,
         "replicationInfo": {
           "replicationFactor": "ONE",
           "requiredNodes": 1,
           "minimumNodes": 1,
           "replicationType": "RATIS"
         },
         "creationTime": 1733221329416,
         "size": 4068,
         "isKey": true,
         "modificationTime": 1733221330013
       }
     ]
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1869358604


##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -57,6 +59,21 @@ Check http return code
                             Should contain      ${result}       200
                         END
 
+Check if the listKeys api responds OK
+    [Arguments]     ${volume}    ${bucket}
+    ${result} =     Curl    "${API_ENDPOINT_URL}/keys/listKeys?startPrefix=/${volume}/${bucket}&limit=1000"
+    Should contain  ${result}   "OK"
+    Should contain  ${result}   "keys"
+    Should contain  ${result}   "${volume}"
+    Should contain  ${result}   "${bucket}"
+
+Curl
+    [Arguments]     ${url}
+    ${curl_command} =    Set Variable    curl -LSs ${url}
+    Run Keyword if      '${SECURITY_ENABLED}' == 'true'     Set Variable    ${curl_command}    curl --negotiate -u : -LSs ${url}

Review Comment:
   I added code to kinit and to use curl directly without any conditions. It should hopefully pass now; I checked out a separate branch with the same code and ran CI successfully on my fork (just for testing): https://github.com/siddhantsangwan/ozone/actions/runs/12157989638



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1868807358


##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -57,6 +59,21 @@ Check http return code
                             Should contain      ${result}       200
                         END
 
+Check if the listKeys api responds OK
+    [Arguments]     ${volume}    ${bucket}
+    ${result} =     Curl    "${API_ENDPOINT_URL}/keys/listKeys?startPrefix=/${volume}/${bucket}&limit=1000"
+    Should contain  ${result}   "OK"
+    Should contain  ${result}   "keys"
+    Should contain  ${result}   "${volume}"
+    Should contain  ${result}   "${bucket}"
+
+Curl
+    [Arguments]     ${url}
+    ${curl_command} =    Set Variable    curl -LSs ${url}
+    Run Keyword if      '${SECURITY_ENABLED}' == 'true'     Set Variable    ${curl_command}    curl --negotiate -u : -LSs ${url}

Review Comment:
   [`Set Variable`](https://robotframework.org/robotframework/6.1.1/libraries/BuiltIn.html#Set%20Variable) returns the value, does not modify its first argument.
   
   This can be fixed by using [`Set Variable If`](https://robotframework.org/robotframework/6.1.1/libraries/BuiltIn.html#Set%20Variable%20If):
   
   ```suggestion
       ${curl_command} =    Set Variable If    '${SECURITY_ENABLED}' == 'true'    curl --negotiate -u : -LSs ${url}    curl -LSs ${url}
   ```
   
   But other test cases use `curl --negotiate -u :` unconditionally, so maybe we don't need the conditional logic at all.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #7524:
URL: https://github.com/apache/ozone/pull/7524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7524:
URL: https://github.com/apache/ozone/pull/7524#issuecomment-2517756485

   Thanks @siddhantsangwan for the fix, @devmadhuu for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1868089834


##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -136,3 +153,13 @@ Check normal api access
 
     kinit as non admin
     Check http return code      ${NON_ADMIN_API_ENDPOINT_URL}   200
+
+Check listKeys api works
+   kinit as ozone admin
+   # Create volume and bucket
+   Execute    ozone sh volume create ${VOLUME}
+   Execute    ozone sh bucket create ${VOLUME}/${BUCKET}
+   Freon OCKG    n=2    args=-s 10 -v ${VOLUME} -b ${BUCKET}
+
+   # Wait until Recon picks up the keys
+   Wait Until Keyword Succeeds     90sec      10sec        Check if the listKeys api responds OK   ${VOLUME}     ${BUCKET}

Review Comment:
   Thanks a lot @siddhantsangwan for adding the test.  Unfortunately it fails with:
   
   ```
   VOLUME_ALREADY_EXISTS Volume already exists
   ```
   
   Instead of starting from scratch, we could add the `listKeys` request to existing test case `Check if Recon picks up OM data`, relying on the volume, bucket and keys created there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #7524:
URL: https://github.com/apache/ozone/pull/7524#discussion_r1868766285


##########
hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot:
##########
@@ -136,3 +153,13 @@ Check normal api access
 
     kinit as non admin
     Check http return code      ${NON_ADMIN_API_ENDPOINT_URL}   200
+
+Check listKeys api works
+   kinit as ozone admin
+   # Create volume and bucket
+   Execute    ozone sh volume create ${VOLUME}
+   Execute    ozone sh bucket create ${VOLUME}/${BUCKET}
+   Freon OCKG    n=2    args=-s 10 -v ${VOLUME} -b ${BUCKET}
+
+   # Wait until Recon picks up the keys
+   Wait Until Keyword Succeeds     90sec      10sec        Check if the listKeys api responds OK   ${VOLUME}     ${BUCKET}

Review Comment:
   Thanks, I've the updated the PR with the suggested change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11848. Serialisation bug in Recon's listKeys API [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7524:
URL: https://github.com/apache/ozone/pull/7524#issuecomment-2514191996

   Thanks @siddhantsangwan for the fix.  Can we add a test case in `hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org