You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/03 07:15:23 UTC

[GitHub] [pulsar] Nicklee007 opened a new pull request, #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Nicklee007 opened a new pull request, #16357:
URL: https://github.com/apache/pulsar/pull/16357

   ### Motivation
   Compare the namespace offload policy,  the topic offload policy command line update function use `int` and `long` type to accept param and miss some param check method, which will cause the param value set as `0`  if the param is missed.
   
   ### Modifications
   1. use `String` to accept param and add some check.
   2. Set a default value if the param is not set, and will check if is default value in `PersistentTopicsBase`.
   3. `NamespacesBase` and some check.
   
   
   ### Documentation
   - [X] `doc-not-needed`   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][client-tool]fix the topic offload policy param will be covered unexpected. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.
URL: https://github.com/apache/pulsar/pull/16357


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Nicklee007 commented on pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-1174722430

   > Here is a good example #15657.
   > 
   > Could you please help add unit tests and update the documents?
   
   @codelipenghui unit test and documents is added, please take again look~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui commented on a diff in pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#discussion_r940206001


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java:
##########
@@ -850,6 +851,29 @@ protected CompletableFuture<OffloadPoliciesImpl> internalGetOffloadPolicies(bool
         return getTopicPoliciesAsyncWithRetry(topicName, isGlobal)
             .thenCompose(op -> {
                 TopicPolicies topicPolicies = op.orElseGet(TopicPolicies::new);
+                OffloadPoliciesImpl currentOffloadPolicies = topicPolicies.getOffloadPolicies();
+                if (currentOffloadPolicies != null && offloadPolicies != null){
+                    if (Objects.equals(offloadPolicies.getManagedLedgerOffloadDeletionLagInMillis(),
+                            OffloadPoliciesImpl.DEFAULT_OFFLOAD_DELETION_LAG_IN_MILLIS)) {
+                        offloadPolicies.setManagedLedgerOffloadDeletionLagInMillis(
+                                currentOffloadPolicies.getManagedLedgerOffloadDeletionLagInMillis());

Review Comment:
   I think this will change the behavior. After this change, users are not able to set the lagInMillis to null?



##########
site2/docs/reference-pulsar-admin.md:
##########
@@ -2470,6 +2470,35 @@ Options
 |`-w`, `--wait-complete`|Wait for compaction to complete|false|
 
 
+### `set-offload-policies`
+Set the offload policy for a topic.
+
+Usage
+
+```bash
+
+$ pulsar-admin topic set-offload-policies tenant/namespace/topic options

Review Comment:
   ```suggestion
   $ pulsar-admin topics set-offload-policies tenant/namespace/topic options
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] gaozhangmin commented on a diff in pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on code in PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#discussion_r913767087


##########
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##########
@@ -1729,6 +1743,48 @@ void run() throws PulsarAdminException {
                 }
             }
 
+            int maxBlockSizeInBytes = OffloadPoliciesImpl.DEFAULT_MAX_BLOCK_SIZE_IN_BYTES;

Review Comment:
   Why use int instead of long? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Nicklee007 commented on a diff in pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on code in PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#discussion_r914659929


##########
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##########
@@ -1729,6 +1743,48 @@ void run() throws PulsarAdminException {
                 }
             }
 
+            int maxBlockSizeInBytes = OffloadPoliciesImpl.DEFAULT_MAX_BLOCK_SIZE_IN_BYTES;
+            if (StringUtils.isNotBlank(maxBlockSizeInBytesStr)) {
+                long maxBlockSize = validateSizeString(maxBlockSizeInBytesStr);
+                if (positiveCheck("MaxBlockSizeInBytes", maxBlockSize)
+                        && maxValueCheck("MaxBlockSizeInBytes", maxBlockSize, Integer.MAX_VALUE)) {

Review Comment:
   The param checks has add to cmdTopics, Thx.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-1501906174

   @Nicklee007 Please add the following content to your PR description and select a checkbox:
   ```
   - [ ] `doc` <!-- Your PR contains doc changes -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-1240141734

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] poorbarcode commented on pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-1501905489

   Since we will start the RC version of `3.0.0` on `2023-04-11`, I will change the label/milestone of PR who have not been merged.
   - The PR of type `feature` is deferred to `3.1.0`
   - The PR of type `fix` is deferred to `3.0.1`
   
   So drag this PR to `3.0.1`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Nicklee007 commented on pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-1190043055

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Nicklee007 commented on pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-1186614282

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Nicklee007 commented on a diff in pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on code in PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#discussion_r914658469


##########
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##########
@@ -1729,6 +1743,48 @@ void run() throws PulsarAdminException {
                 }
             }
 
+            int maxBlockSizeInBytes = OffloadPoliciesImpl.DEFAULT_MAX_BLOCK_SIZE_IN_BYTES;

Review Comment:
   The OffloadPoliciesImpl define the `managedLedgerOffloadMaxBlockSizeInBytes` and `managedLedgerOffloadReadBufferSizeInBytes` as Integer, as usual the block size or ReadBufferSize is less than GB, seems int is enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] gaozhangmin commented on a diff in pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on code in PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#discussion_r919706859


##########
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java:
##########
@@ -2010,20 +2011,20 @@ private class SetOffloadPolicies extends CliCommand {
         @Parameter(names = {"-m", "--maxBlockSizeInBytes"},
                 description = "ManagedLedger offload max block Size in bytes,"
                 + "s3 and google-cloud-storage requires this parameter")
-        private int maxBlockSizeInBytes;
+        private String maxBlockSizeInBytesStr;
 
         @Parameter(names = {"-rb", "--readBufferSizeInBytes"},
                 description = "ManagedLedger offload read buffer size in bytes,"
                 + "s3 and google-cloud-storage requires this parameter")
-        private int readBufferSizeInBytes;
+        private String readBufferSizeInBytesStr;
 
         @Parameter(names = {"-t", "--offloadThresholdInBytes"}
                 , description = "ManagedLedger offload threshold in bytes", required = true)
-        private long offloadThresholdInBytes;
+        private String offloadThresholdInBytesStr;
 
         @Parameter(names = {"-dl", "--offloadDeletionLagInMillis"}
                 , description = "ManagedLedger offload deletion lag in bytes")
-        private Long offloadDeletionLagInMillis;
+        private String offloadDeletionLagInMillisStr;

Review Comment:
   We'd better make this parameters and their desc unified with CmdNamespaces.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][client-tool]fix the topic offload policy param will be covered unexpected. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-6368870263

   Thanks @Nicklee007 for the fix, and sorry it sat so long. The CLI problem is now fixed on master: #20646 and #20943 made the topic-level `set-offload-policies` options in `CmdTopics` and `CmdTopicPolicies` default to the `OffloadPoliciesImpl` defaults, so options you leave out are no longer sent as `0`. We'd rather keep `set` as a full replace on the broker side, so I'm closing this as superseded. If you still see a case where a parameter gets overwritten, please open an issue with the exact command.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Nicklee007 commented on pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#issuecomment-1186174536

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] gaozhangmin commented on a diff in pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on code in PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#discussion_r913733227


##########
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##########
@@ -1729,6 +1743,48 @@ void run() throws PulsarAdminException {
                 }
             }
 
+            int maxBlockSizeInBytes = OffloadPoliciesImpl.DEFAULT_MAX_BLOCK_SIZE_IN_BYTES;
+            if (StringUtils.isNotBlank(maxBlockSizeInBytesStr)) {
+                long maxBlockSize = validateSizeString(maxBlockSizeInBytesStr);
+                if (positiveCheck("MaxBlockSizeInBytes", maxBlockSize)
+                        && maxValueCheck("MaxBlockSizeInBytes", maxBlockSize, Integer.MAX_VALUE)) {

Review Comment:
   The same problem also exists in `CmdTopics`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] gaozhangmin commented on a diff in pull request #16357: [fix][client-tool]fix the topic offload policy param will be covered unexpected.

Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on code in PR #16357:
URL: https://github.com/apache/pulsar/pull/16357#discussion_r913733227


##########
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##########
@@ -1729,6 +1743,48 @@ void run() throws PulsarAdminException {
                 }
             }
 
+            int maxBlockSizeInBytes = OffloadPoliciesImpl.DEFAULT_MAX_BLOCK_SIZE_IN_BYTES;
+            if (StringUtils.isNotBlank(maxBlockSizeInBytesStr)) {
+                long maxBlockSize = validateSizeString(maxBlockSizeInBytesStr);
+                if (positiveCheck("MaxBlockSizeInBytes", maxBlockSize)
+                        && maxValueCheck("MaxBlockSizeInBytes", maxBlockSize, Integer.MAX_VALUE)) {

Review Comment:
   The same problem also exists in `CmdTopics`.
   Bytes should use  long instead of int, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org