You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2023/09/17 18:13:22 UTC

[GitHub] [pinot] ArnavBalyan opened a new pull request, #11608: Fix for #11588. Updated the response for reload API to be pretty printed

ArnavBalyan opened a new pull request, #11608:
URL: https://github.com/apache/pinot/pull/11608

   - Reload API returns an encoded JSON which is hard to understand.
   - Updated the response to remove the additional string.
   - Added a Pretty printed string which will print the json in an understandable format.
   `performance`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ArnavBalyan commented on pull request #11608: Updated the response for reload API to be pretty printed

Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #11608:
URL: https://github.com/apache/pinot/pull/11608#issuecomment-1723432879

   > you also need to update
   > 
   > https://github.com/apache/pinot/blob/b1b070f68dce37605a705b25834f0f5579a3b3be/pinot-controller/src/main/resources/app/pages/TenantDetails.tsx#L340
   > 
   > since that's parsing this result.
   > You should run this through the quickstart to make sure it's still working
   
   Got it thanks let me do this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Updated the response for reload API to be pretty printed [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11608:
URL: https://github.com/apache/pinot/pull/11608


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] jadami10 commented on a diff in pull request #11608: Updated the response for reload API to be pretty printed

Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1328719375


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -782,7 +782,7 @@ public SuccessResponse reloadAllSegments(
         LOGGER.error("Failed to add reload all segments job meta into zookeeper for table: {}", tableNameWithType, e);
       }
     }
-    return new SuccessResponse("Segment reload details: " + JsonUtils.objectToString(perTableMsgData));
+    return new SuccessResponse(JsonUtils.objectToPrettyString(perTableMsgData));

Review Comment:
   it doesn't need to be, but what value is `Segment reload details:` to the result of an endpoint that is clearly already the reload details. The impetus for this change was to make it easier to parse the result in java/scala so you don't have to go from request -> strip the prefix -> decode the json string -> decode the json. You should be able to go from request -> json in 1 hop.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Updated the response for reload API to be pretty printed [pinot]

Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1365404511


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java:
##########
@@ -645,9 +645,11 @@ public String reloadTableAndValidateResponse(String tableName, TableType tableTy
       throws IOException {
     String response =
         sendPostRequest(_controllerRequestURLBuilder.forTableReload(tableName, tableType, forceDownload), null);
+    response = StringEscapeUtils.unescapeJava(response);
     String tableNameWithType = TableNameBuilder.forType(tableType).tableNameWithType(tableName);
     JsonNode tableLevelDetails =
-        JsonUtils.stringToJsonNode(StringEscapeUtils.unescapeJava(response.split(": ")[1])).get(tableNameWithType);
+        JsonUtils.stringToJsonNode(response.substring(

Review Comment:
   Oh got it thank you so much!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11608: Updated the response for reload API to be pretty printed

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1328167943


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -782,7 +782,7 @@ public SuccessResponse reloadAllSegments(
         LOGGER.error("Failed to add reload all segments job meta into zookeeper for table: {}", tableNameWithType, e);
       }
     }
-    return new SuccessResponse("Segment reload details: " + JsonUtils.objectToString(perTableMsgData));
+    return new SuccessResponse(JsonUtils.objectToPrettyString(perTableMsgData));

Review Comment:
   It doesn't need to be a json, maybe simply
   ```suggestion
       return new SuccessResponse("Segment reload details: " + perTableMsgData);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] jadami10 commented on pull request #11608: Updated the response for reload API to be pretty printed

Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on PR #11608:
URL: https://github.com/apache/pinot/pull/11608#issuecomment-1726892496

   looks like there's still some tests failing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Updated the response for reload API to be pretty printed [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1348302897


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java:
##########
@@ -645,9 +645,11 @@ public String reloadTableAndValidateResponse(String tableName, TableType tableTy
       throws IOException {
     String response =
         sendPostRequest(_controllerRequestURLBuilder.forTableReload(tableName, tableType, forceDownload), null);
+    response = StringEscapeUtils.unescapeJava(response);
     String tableNameWithType = TableNameBuilder.forType(tableType).tableNameWithType(tableName);
     JsonNode tableLevelDetails =
-        JsonUtils.stringToJsonNode(StringEscapeUtils.unescapeJava(response.split(": ")[1])).get(tableNameWithType);
+        JsonUtils.stringToJsonNode(response.substring(

Review Comment:
   The json string is encoded as the value field of the top level json, so parsing the json twice should be able to extract it. See the code example above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #11608: Updated the response for reload API to be pretty printed

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11608:
URL: https://github.com/apache/pinot/pull/11608#issuecomment-1722787876

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11608?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11608](https://app.codecov.io/gh/apache/pinot/pull/11608?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (fe3f7f8) into [master](https://app.codecov.io/gh/apache/pinot/commit/8abe86b41ff74c1fc128ec4e70db4100f6ca1dd1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (8abe86b) will **decrease** coverage by `48.55%`.
   > Report is 10 commits behind head on master.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #11608       +/-   ##
   =============================================
   - Coverage     63.05%   14.51%   -48.55%     
   + Complexity     1106      201      -905     
   =============================================
     Files          2326     2323        -3     
     Lines        124974   124459      -515     
     Branches      19078    18988       -90     
   =============================================
   - Hits          78803    18064    -60739     
   - Misses        40569   104877    +64308     
   + Partials       5602     1518     -4084     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (-48.40%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (-48.55%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (-48.54%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...ler/api/resources/PinotSegmentRestletResource.java](https://app.codecov.io/gh/apache/pinot/pull/11608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90U2VnbWVudFJlc3RsZXRSZXNvdXJjZS5qYXZh) | `16.02% <0.00%> (ø)` | |
   
   ... and [1517 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11608/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Updated the response for reload API to be pretty printed [pinot]

Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1346824650


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java:
##########
@@ -645,9 +645,11 @@ public String reloadTableAndValidateResponse(String tableName, TableType tableTy
       throws IOException {
     String response =
         sendPostRequest(_controllerRequestURLBuilder.forTableReload(tableName, tableType, forceDownload), null);
+    response = StringEscapeUtils.unescapeJava(response);
     String tableNameWithType = TableNameBuilder.forType(tableType).tableNameWithType(tableName);
     JsonNode tableLevelDetails =
-        JsonUtils.stringToJsonNode(StringEscapeUtils.unescapeJava(response.split(": ")[1])).get(tableNameWithType);
+        JsonUtils.stringToJsonNode(response.substring(

Review Comment:
   Hi @Jackie-Jiang the the value corresponding to the "status" is another json string. So parsing it into json fails because the value is wrapped in "". I just did the substring logic to get the wrapped string. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Updated the response for reload API to be pretty printed [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1346568856


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java:
##########
@@ -645,9 +645,11 @@ public String reloadTableAndValidateResponse(String tableName, TableType tableTy
       throws IOException {
     String response =
         sendPostRequest(_controllerRequestURLBuilder.forTableReload(tableName, tableType, forceDownload), null);
+    response = StringEscapeUtils.unescapeJava(response);
     String tableNameWithType = TableNameBuilder.forType(tableType).tableNameWithType(tableName);
     JsonNode tableLevelDetails =
-        JsonUtils.stringToJsonNode(StringEscapeUtils.unescapeJava(response.split(": ")[1])).get(tableNameWithType);
+        JsonUtils.stringToJsonNode(response.substring(

Review Comment:
   This is very hard to understand. I think we might need to parse twice? Something like:
   ```
   JsonNode responseJson = JsonUtils.stringToJsonNode(response);
   JsonNode tableLevelDetails = JsonUtils.stringToJsonNode(responseJson.get("status").asText()).get(tableNameWithType);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Updated the response for reload API to be pretty printed [pinot]

Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1362608550


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java:
##########
@@ -645,9 +645,11 @@ public String reloadTableAndValidateResponse(String tableName, TableType tableTy
       throws IOException {
     String response =
         sendPostRequest(_controllerRequestURLBuilder.forTableReload(tableName, tableType, forceDownload), null);
+    response = StringEscapeUtils.unescapeJava(response);
     String tableNameWithType = TableNameBuilder.forType(tableType).tableNameWithType(tableName);
     JsonNode tableLevelDetails =
-        JsonUtils.stringToJsonNode(StringEscapeUtils.unescapeJava(response.split(": ")[1])).get(tableNameWithType);
+        JsonUtils.stringToJsonNode(response.substring(

Review Comment:
   Got it thanks, I did this, it seems the 6 tests fail. The result is wrapped in "", so the `JsonUtils.stringToJsonNode` fails parsing it, to fix the test we would have to extract the value and then parse it. What do you think. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Updated the response for reload API to be pretty printed [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1364679422


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java:
##########
@@ -645,9 +645,11 @@ public String reloadTableAndValidateResponse(String tableName, TableType tableTy
       throws IOException {
     String response =
         sendPostRequest(_controllerRequestURLBuilder.forTableReload(tableName, tableType, forceDownload), null);
+    response = StringEscapeUtils.unescapeJava(response);
     String tableNameWithType = TableNameBuilder.forType(tableType).tableNameWithType(tableName);
     JsonNode tableLevelDetails =
-        JsonUtils.stringToJsonNode(StringEscapeUtils.unescapeJava(response.split(": ")[1])).get(tableNameWithType);
+        JsonUtils.stringToJsonNode(response.substring(

Review Comment:
   The problem is the unnecessary unescape. Applied a commit and let's see if it fixes the problem



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ArnavBalyan commented on a diff in pull request #11608: Updated the response for reload API to be pretty printed

Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #11608:
URL: https://github.com/apache/pinot/pull/11608#discussion_r1330145418


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -782,7 +782,7 @@ public SuccessResponse reloadAllSegments(
         LOGGER.error("Failed to add reload all segments job meta into zookeeper for table: {}", tableNameWithType, e);
       }
     }
-    return new SuccessResponse("Segment reload details: " + JsonUtils.objectToString(perTableMsgData));
+    return new SuccessResponse(JsonUtils.objectToPrettyString(perTableMsgData));

Review Comment:
   For now, I've just kept the json string in the success response which makes it easier to parse/understand. Please let me know what you think Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org