You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/10/24 02:25:43 UTC

[GitHub] [pulsar] tjiuming opened a new pull request, #18138: [fix][monitor] fix metrics string encoding

tjiuming opened a new pull request, #18138:
URL: https://github.com/apache/pulsar/pull/18138

   ### Motivation
   
   For the method [SimpleTextOutputStream](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java)#write(String), it's implementation as below:
   ```
       public SimpleTextOutputStream write(String s) {
           if (s == null) {
               return this;
           }
           int len = s.length();
           for (int i = 0; i < len; i++) {
               buffer.writeByte((byte) s.charAt(i));
           }
   
           return this;
       }
   ```
   
   Generally, it works fine. But if there is a character has different length in different encoding, it would lead to an issue, say `¬`.
   
   In latin1, it's bytes is [-84], by calling `"¬".getBytes(StandardCharset.ISO_8859_1)`
   but in UTF8, it's bytes is [-62, -84], by calling `"¬".getBytes(StandardCharset.UTF_8)`
   
   if we replace `str.getBytes(charset)` with `(byte) s.charAt(i)`, `¬` will display as `�`.
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   PR in forked repository: https://github.com/tjiuming/pulsar/pull/6
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] coderzc commented on a diff in pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#discussion_r1001333319


##########
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##########
@@ -43,19 +44,16 @@ public SimpleTextOutputStream write(byte[] a, int offset, int len) {
     }
 
     public SimpleTextOutputStream write(char c) {
-        buffer.writeByte((byte) c);
+        write(String.valueOf(c));

Review Comment:
   ```suggestion
           buffer.writeChar(c);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tjiuming commented on a diff in pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
tjiuming commented on code in PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#discussion_r1001333807


##########
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##########
@@ -43,19 +44,16 @@ public SimpleTextOutputStream write(byte[] a, int offset, int len) {
     }
 
     public SimpleTextOutputStream write(char c) {
-        buffer.writeByte((byte) c);
+        write(String.valueOf(c));

Review Comment:
   I've tried to use `buffer.writeChar()`, but I doesn't work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] coderzc commented on a diff in pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#discussion_r1001333319


##########
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##########
@@ -43,19 +44,16 @@ public SimpleTextOutputStream write(byte[] a, int offset, int len) {
     }
 
     public SimpleTextOutputStream write(char c) {
-        buffer.writeByte((byte) c);
+        write(String.valueOf(c));

Review Comment:
   ```suggestion
           buffer.writeChar(c);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] asafm commented on pull request #18138: [fix][monitoring] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
asafm commented on PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#issuecomment-1285473616

   Thanks, @tjiuming for that prompt fix!
   I'll just expand a bit on the motivation:
   
   `SimpleTextOutputFormat` is the class used when writing Prometheus Text Format lines - we print those lines in this class, which inside writes as bytes to a `ByteBuf` (Netty's). That buffer is then flushed to the HTTP response upon `/metrics` endpoint (when Prometheus scrapes metrics from Pulsar).
   
   The Prometheus Text Format requires label values to be UTF-8 encoded, as written [here](https://github.com/Showmax/prometheus-docs/blob/master/content/docs/instrumenting/exposition_formats.md):
   > label_value can be any sequence of UTF-8 characters
   
   Today, as @tjiuming mentioned, the current implementation iterates over the String and each character is written in an undefined encoding - i.e. manipulating the `char` and then casting it to `byte`
   ```
   buffer.writeByte((byte) s.charAt(i));
   ```
   
   This causes Prometheus to fail the scraping as it tries to validate that the label value byte array in the response is a valid UTF-8 encoded string (which is not).
   
   The fix as @tjiuming mentioned is encoding the char using UTF-8 as provided by the `ByteBuf` class.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] coderzc commented on a diff in pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#discussion_r1001350216


##########
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##########
@@ -43,19 +44,16 @@ public SimpleTextOutputStream write(byte[] a, int offset, int len) {
     }
 
     public SimpleTextOutputStream write(char c) {
-        buffer.writeByte((byte) c);
+        write(String.valueOf(c));

Review Comment:
   Oh, the class is used for encoding other types to a UTF-8 buffer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] asafm commented on a diff in pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#discussion_r1002666681


##########
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##########
@@ -43,19 +44,16 @@ public SimpleTextOutputStream write(byte[] a, int offset, int len) {
     }
 
     public SimpleTextOutputStream write(char c) {
-        buffer.writeByte((byte) c);
+        write(String.valueOf(c));

Review Comment:
   I wouldn't do that, since we're back to the same bug if some future developer decides to use it wrong.
   I prefer to abolish that method altogether and refactor it to use String instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codecov-commenter commented on pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#issuecomment-1288346843

   # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18138?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#18138](https://codecov.io/gh/apache/pulsar/pull/18138?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c4707dc) into [master](https://codecov.io/gh/apache/pulsar/commit/6c65ca0d8a80bfaaa4d5869e0cea485f5c94369b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6c65ca0) will **increase** coverage by `36.05%`.
   > The diff coverage is `50.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pulsar/pull/18138/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pulsar/pull/18138?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #18138       +/-   ##
   =============================================
   + Coverage     34.91%   70.97%   +36.05%     
   + Complexity     5707      438     -5269     
   =============================================
     Files           607       26      -581     
     Lines         53396     2246    -51150     
     Branches       5712      245     -5467     
   =============================================
   - Hits          18644     1594    -17050     
   + Misses        32119      479    -31640     
   + Partials       2633      173     -2460     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `70.97% <50.00%> (+36.05%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18138?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...apache/pulsar/proxy/server/DirectProxyHandler.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL0RpcmVjdFByb3h5SGFuZGxlci5qYXZh) | `63.63% <50.00%> (ø)` | |
   | [...pache/pulsar/proxy/server/ProxyServiceStarter.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL1Byb3h5U2VydmljZVN0YXJ0ZXIuamF2YQ==) | `60.66% <0.00%> (-1.34%)` | :arrow_down: |
   | [.../main/java/org/apache/pulsar/PulsarStandalone.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL1B1bHNhclN0YW5kYWxvbmUuamF2YQ==) | | |
   | [...n/java/org/apache/pulsar/broker/PulsarService.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9QdWxzYXJTZXJ2aWNlLmphdmE=) | | |
   | [.../org/apache/pulsar/broker/admin/AdminResource.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9BZG1pblJlc291cmNlLmphdmE=) | | |
   | [.../apache/pulsar/broker/admin/impl/ClustersBase.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL0NsdXN0ZXJzQmFzZS5qYXZh) | | |
   | [...pache/pulsar/broker/admin/impl/NamespacesBase.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL05hbWVzcGFjZXNCYXNlLmphdmE=) | | |
   | [...pulsar/broker/admin/impl/PersistentTopicsBase.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL1BlcnNpc3RlbnRUb3BpY3NCYXNlLmphdmE=) | | |
   | [.../pulsar/broker/admin/impl/SchemasResourceBase.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL1NjaGVtYXNSZXNvdXJjZUJhc2UuamF2YQ==) | | |
   | [.../org/apache/pulsar/broker/admin/v2/Namespaces.java](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9OYW1lc3BhY2VzLmphdmE=) | | |
   | ... and [575 more](https://codecov.io/gh/apache/pulsar/pull/18138/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tjiuming commented on pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
tjiuming commented on PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#issuecomment-1286414801

   > I wonder if it would also make sense to fix this method:
   > 
   > https://github.com/apache/pulsar/blob/b59d0e9935f3943f514163b3bd14b03879fbd8dc/pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java#L46-L49
   
   `write(char)` is used to add some symbols like `,` ` ` `\n` `"`, generally, it won't lead to an issue. But, sure, fix it is better.
   @lhotari I've fixed it, PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui merged pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
codelipenghui merged PR #18138:
URL: https://github.com/apache/pulsar/pull/18138


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #18138: [fix][monitoring] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#issuecomment-1285959254

   @tjiuming Please run the tests in your fork as instructed in https://github.com/apache/pulsar/actions/runs/3288972604/attempts/1#summary-9005069188 .
   The PR title needs revisiting, please check https://github.com/apache/pulsar/actions/runs/3288972601/jobs/5419952411 .
   monitoring -> monitor .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #18138: [fix][monitoring] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#issuecomment-1285961678

   I wonder if it would also make sense to fix this method: https://github.com/apache/pulsar/blob/b59d0e9935f3943f514163b3bd14b03879fbd8dc/pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java#L46-L49
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tjiuming commented on a diff in pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
tjiuming commented on code in PR #18138:
URL: https://github.com/apache/pulsar/pull/18138#discussion_r1001333807


##########
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##########
@@ -43,19 +44,16 @@ public SimpleTextOutputStream write(byte[] a, int offset, int len) {
     }
 
     public SimpleTextOutputStream write(char c) {
-        buffer.writeByte((byte) c);
+        write(String.valueOf(c));

Review Comment:
   I've tried to use `buffer.writeChar()`, but it doesn't work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui closed pull request #18138: [fix][monitor] fix metrics string encoding

Posted by GitBox <gi...@apache.org>.
codelipenghui closed pull request #18138: [fix][monitor] fix metrics string encoding
URL: https://github.com/apache/pulsar/pull/18138


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org