You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/28 09:18:54 UTC
[GitHub] [pulsar] mattisonchao opened a new pull request #13534: Offload policies support cross multiple clusters
mattisonchao opened a new pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534
Master Issue: #13444
### Motivation
Offload policies support cross multiple clusters
### Documentation
- [x] `no-need-doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1008233324
@315157973
PTAL : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 merged pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
315157973 merged pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#discussion_r776210921
##########
File path: pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java
##########
@@ -733,6 +735,135 @@ void run() throws PulsarAdminException {
}
}
+
+ @Parameters(commandDescription = "Get the offload policies for a topic")
+ private class GetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "-ap", "--applied" }, description = "Get the applied policy of the topic")
+ private boolean applied = false;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to get this policy globally. "
+ + "If set to true, broker returned global topic policies")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ print(getTopicPolicies(isGlobal).getOffloadPolicies(persistentTopic, applied));
+ }
+ }
+
+ @Parameters(commandDescription = "Remove the offload policies for a topic")
+ private class RemoveOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to remove this policy globally. "
+ + "If set to true, the removing operation will be replicate to other clusters asynchronously")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ getTopicPolicies(isGlobal).removeOffloadPolicies(persistentTopic);
+ }
+ }
+
+ @Parameters(commandDescription = "Set the offload policies for a topic")
+ private class SetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = {"-d", "--driver"}, description = "ManagedLedger offload driver", required = true)
+ private String driver;
+
+ @Parameter(names = {"-r", "--region"}
+ , description = "ManagedLedger offload region, s3 and google-cloud-storage requires this parameter")
+ private String region;
+
+ @Parameter(names = {"-b", "--bucket"}
+ , description = "ManagedLedger offload bucket, s3 and google-cloud-storage requires this parameter")
+ private String bucket;
+
+ @Parameter(names = {"-e", "--endpoint"}
+ , description = "ManagedLedger offload service endpoint, only s3 requires this parameter")
+ private String endpoint;
+
+ @Parameter(names = {"-i", "--aws-id"}
+ , description = "AWS Credential Id to use when using driver S3 or aws-s3")
+ private String awsId;
+
+ @Parameter(names = {"-s", "--aws-secret"}
+ , description = "AWS Credential Secret to use when using driver S3 or aws-s3")
+ private String awsSecret;
Review comment:
Maybe we can interact with the user, just like entering a password?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1005655236
@codelipenghui @BewareMyPower @315157973 @eolivelli @Technoboy-
PTAL :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1003325821
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1003325821
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1002083299
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#discussion_r780661665
##########
File path: pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java
##########
@@ -733,6 +735,135 @@ void run() throws PulsarAdminException {
}
}
+
+ @Parameters(commandDescription = "Get the offload policies for a topic")
+ private class GetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "-ap", "--applied" }, description = "Get the applied policy of the topic")
+ private boolean applied = false;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to get this policy globally. "
+ + "If set to true, broker returned global topic policies")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ print(getTopicPolicies(isGlobal).getOffloadPolicies(persistentTopic, applied));
+ }
+ }
+
+ @Parameters(commandDescription = "Remove the offload policies for a topic")
+ private class RemoveOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to remove this policy globally. "
+ + "If set to true, the removing operation will be replicate to other clusters asynchronously")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ getTopicPolicies(isGlobal).removeOffloadPolicies(persistentTopic);
+ }
+ }
+
+ @Parameters(commandDescription = "Set the offload policies for a topic")
+ private class SetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = {"-d", "--driver"}, description = "ManagedLedger offload driver", required = true)
+ private String driver;
+
+ @Parameter(names = {"-r", "--region"}
+ , description = "ManagedLedger offload region, s3 and google-cloud-storage requires this parameter")
+ private String region;
+
+ @Parameter(names = {"-b", "--bucket"}
+ , description = "ManagedLedger offload bucket, s3 and google-cloud-storage requires this parameter")
+ private String bucket;
+
+ @Parameter(names = {"-e", "--endpoint"}
+ , description = "ManagedLedger offload service endpoint, only s3 requires this parameter")
+ private String endpoint;
+
+ @Parameter(names = {"-i", "--aws-id"}
+ , description = "AWS Credential Id to use when using driver S3 or aws-s3")
+ private String awsId;
+
+ @Parameter(names = {"-s", "--aws-secret"}
+ , description = "AWS Credential Secret to use when using driver S3 or aws-s3")
+ private String awsSecret;
Review comment:
Do we have other cases in which we ask for secrets in this tool?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1005655236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1011256291
@mattisonchao Could you please help resolve the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1003276906
@315157973 @eolivelli
PTAL :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#discussion_r776209102
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ReplicatorTopicPoliciesTest.java
##########
@@ -296,6 +298,37 @@ public void testReplicatorMessageDispatchRatePolicies() throws Exception {
assertNull(admin3.topicPolicies(true).getDispatchRate(persistentTopicName)));
}
+ @Test
+ public void testReplicatorOffloadPolicies() throws Exception {
+ final String namespace = "pulsar/partitionedNs-" + UUID.randomUUID();
+ final String persistentTopicName = "persistent://" + namespace + "/topic" + UUID.randomUUID();
+
+ init(namespace, persistentTopicName);
+ OffloadPoliciesImpl offloadPolicies =
+ OffloadPoliciesImpl.create("s3", "region", "bucket", "endpoint", null, null, null, null,
+ 8, 9, 10L, null, OffloadedReadPriority.BOOKKEEPER_FIRST);
+
+ // set offload policies
+ try{
+ admin1.topicPolicies(true).setOffloadPolicies(persistentTopicName, offloadPolicies);
+ }catch (Exception ignore){
+ // ignore PersistentTopicsBase#internalUpdateOffloadPolicies not found nar loader exception.
Review comment:
If i do not ignore this exception, i will get not found nar loader exception.
because of this test is not mock test, should i add a nar package in project ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1002136586
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1002083299
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#discussion_r776211170
##########
File path: pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java
##########
@@ -733,6 +735,135 @@ void run() throws PulsarAdminException {
}
}
+
+ @Parameters(commandDescription = "Get the offload policies for a topic")
+ private class GetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "-ap", "--applied" }, description = "Get the applied policy of the topic")
+ private boolean applied = false;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to get this policy globally. "
+ + "If set to true, broker returned global topic policies")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ print(getTopicPolicies(isGlobal).getOffloadPolicies(persistentTopic, applied));
+ }
+ }
+
+ @Parameters(commandDescription = "Remove the offload policies for a topic")
+ private class RemoveOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to remove this policy globally. "
+ + "If set to true, the removing operation will be replicate to other clusters asynchronously")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ getTopicPolicies(isGlobal).removeOffloadPolicies(persistentTopic);
+ }
+ }
+
+ @Parameters(commandDescription = "Set the offload policies for a topic")
+ private class SetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = {"-d", "--driver"}, description = "ManagedLedger offload driver", required = true)
+ private String driver;
+
+ @Parameter(names = {"-r", "--region"}
+ , description = "ManagedLedger offload region, s3 and google-cloud-storage requires this parameter")
+ private String region;
+
+ @Parameter(names = {"-b", "--bucket"}
+ , description = "ManagedLedger offload bucket, s3 and google-cloud-storage requires this parameter")
+ private String bucket;
+
+ @Parameter(names = {"-e", "--endpoint"}
+ , description = "ManagedLedger offload service endpoint, only s3 requires this parameter")
+ private String endpoint;
+
+ @Parameter(names = {"-i", "--aws-id"}
+ , description = "AWS Credential Id to use when using driver S3 or aws-s3")
+ private String awsId;
+
+ @Parameter(names = {"-s", "--aws-secret"}
+ , description = "AWS Credential Secret to use when using driver S3 or aws-s3")
+ private String awsSecret;
Review comment:
Maybe we can interact with the user, just like entering a password?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1011910250
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1008033181
> +1
>
> We should figure out a way to not pass secrets from the cli, but this is something to do in a follow up patch probably
Thanks to you @eolivelli , i will do that in a follow up patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1011750437
> @mattisonchao Could you please help resolve the conflicts?
Conflict has been fixed. PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1003289501
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1002136586
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#discussion_r776202112
##########
File path: pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java
##########
@@ -733,6 +735,135 @@ void run() throws PulsarAdminException {
}
}
+
+ @Parameters(commandDescription = "Get the offload policies for a topic")
+ private class GetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "-ap", "--applied" }, description = "Get the applied policy of the topic")
+ private boolean applied = false;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to get this policy globally. "
+ + "If set to true, broker returned global topic policies")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ print(getTopicPolicies(isGlobal).getOffloadPolicies(persistentTopic, applied));
+ }
+ }
+
+ @Parameters(commandDescription = "Remove the offload policies for a topic")
+ private class RemoveOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to remove this policy globally. "
+ + "If set to true, the removing operation will be replicate to other clusters asynchronously")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ getTopicPolicies(isGlobal).removeOffloadPolicies(persistentTopic);
+ }
+ }
+
+ @Parameters(commandDescription = "Set the offload policies for a topic")
+ private class SetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = {"-d", "--driver"}, description = "ManagedLedger offload driver", required = true)
+ private String driver;
+
+ @Parameter(names = {"-r", "--region"}
+ , description = "ManagedLedger offload region, s3 and google-cloud-storage requires this parameter")
+ private String region;
+
+ @Parameter(names = {"-b", "--bucket"}
+ , description = "ManagedLedger offload bucket, s3 and google-cloud-storage requires this parameter")
+ private String bucket;
+
+ @Parameter(names = {"-e", "--endpoint"}
+ , description = "ManagedLedger offload service endpoint, only s3 requires this parameter")
+ private String endpoint;
+
+ @Parameter(names = {"-i", "--aws-id"}
+ , description = "AWS Credential Id to use when using driver S3 or aws-s3")
+ private String awsId;
+
+ @Parameter(names = {"-s", "--aws-secret"}
+ , description = "AWS Credential Secret to use when using driver S3 or aws-s3")
+ private String awsSecret;
Review comment:
Passing a secret as a cli argument is not usually safe (it is recorded in the shell history for instance).
Do we have a better way?
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ReplicatorTopicPoliciesTest.java
##########
@@ -296,6 +298,37 @@ public void testReplicatorMessageDispatchRatePolicies() throws Exception {
assertNull(admin3.topicPolicies(true).getDispatchRate(persistentTopicName)));
}
+ @Test
+ public void testReplicatorOffloadPolicies() throws Exception {
+ final String namespace = "pulsar/partitionedNs-" + UUID.randomUUID();
+ final String persistentTopicName = "persistent://" + namespace + "/topic" + UUID.randomUUID();
+
+ init(namespace, persistentTopicName);
+ OffloadPoliciesImpl offloadPolicies =
+ OffloadPoliciesImpl.create("s3", "region", "bucket", "endpoint", null, null, null, null,
+ 8, 9, 10L, null, OffloadedReadPriority.BOOKKEEPER_FIRST);
+
+ // set offload policies
+ try{
+ admin1.topicPolicies(true).setOffloadPolicies(persistentTopicName, offloadPolicies);
+ }catch (Exception ignore){
+ // ignore PersistentTopicsBase#internalUpdateOffloadPolicies not found nar loader exception.
Review comment:
We cannot ignore this.
In a test case like this you have 100% control over what's happening.
Please drop this catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1003289501
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1003885582
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao edited a comment on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao edited a comment on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1011750437
> @mattisonchao Could you please help resolve the conflicts?
Conflict has been fixed. PTAL :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#discussion_r783266722
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ReplicatorTopicPoliciesTest.java
##########
@@ -483,6 +485,38 @@ public void testReplicatorCompactionThresholdPolicies() throws Exception {
assertNull(admin3.topicPolicies(true).getCompactionThreshold(persistentTopicName)));
}
+ @Test
+ public void testReplicatorOffloadPolicies() throws Exception {
+ final String namespace = "pulsar/partitionedNs-" + UUID.randomUUID();
+ final String persistentTopicName = "persistent://" + namespace + "/topic" + UUID.randomUUID();
+
+ init(namespace, persistentTopicName);
+ OffloadPoliciesImpl offloadPolicies =
+ OffloadPoliciesImpl.create("s3", "region", "bucket", "endpoint", null, null, null, null,
+ 8, 9, 10L, null, OffloadedReadPriority.BOOKKEEPER_FIRST);
+
+ // set offload policies
+ try{
+ admin1.topicPolicies(true).setOffloadPolicies(persistentTopicName, offloadPolicies);
+ }catch (Exception exception){
+ // driver not found exception.
+ assertTrue(exception instanceof PulsarAdminException.ServerSideErrorException);
+ }
Review comment:
```suggestion
try {
admin1.topicPolicies(true).setOffloadPolicies(persistentTopicName, offloadPolicies);
} catch (Exception exception){
// driver not found exception.
assertTrue(exception instanceof PulsarAdminException.ServerSideErrorException);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#issuecomment-1002139547
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13534: Offload policies support cross multiple clusters
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13534:
URL: https://github.com/apache/pulsar/pull/13534#discussion_r776211170
##########
File path: pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java
##########
@@ -733,6 +735,135 @@ void run() throws PulsarAdminException {
}
}
+
+ @Parameters(commandDescription = "Get the offload policies for a topic")
+ private class GetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "-ap", "--applied" }, description = "Get the applied policy of the topic")
+ private boolean applied = false;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to get this policy globally. "
+ + "If set to true, broker returned global topic policies")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ print(getTopicPolicies(isGlobal).getOffloadPolicies(persistentTopic, applied));
+ }
+ }
+
+ @Parameters(commandDescription = "Remove the offload policies for a topic")
+ private class RemoveOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = { "--global", "-g" }, description = "Whether to remove this policy globally. "
+ + "If set to true, the removing operation will be replicate to other clusters asynchronously")
+ private boolean isGlobal = false;
+
+ @Override
+ void run() throws PulsarAdminException {
+ String persistentTopic = validatePersistentTopic(params);
+ getTopicPolicies(isGlobal).removeOffloadPolicies(persistentTopic);
+ }
+ }
+
+ @Parameters(commandDescription = "Set the offload policies for a topic")
+ private class SetOffloadPolicies extends CliCommand {
+ @Parameter(description = "persistent://tenant/namespace/topic", required = true)
+ private java.util.List<String> params;
+
+ @Parameter(names = {"-d", "--driver"}, description = "ManagedLedger offload driver", required = true)
+ private String driver;
+
+ @Parameter(names = {"-r", "--region"}
+ , description = "ManagedLedger offload region, s3 and google-cloud-storage requires this parameter")
+ private String region;
+
+ @Parameter(names = {"-b", "--bucket"}
+ , description = "ManagedLedger offload bucket, s3 and google-cloud-storage requires this parameter")
+ private String bucket;
+
+ @Parameter(names = {"-e", "--endpoint"}
+ , description = "ManagedLedger offload service endpoint, only s3 requires this parameter")
+ private String endpoint;
+
+ @Parameter(names = {"-i", "--aws-id"}
+ , description = "AWS Credential Id to use when using driver S3 or aws-s3")
+ private String awsId;
+
+ @Parameter(names = {"-s", "--aws-secret"}
+ , description = "AWS Credential Secret to use when using driver S3 or aws-s3")
+ private String awsSecret;
Review comment:
Maybe we can interact with the user, just like entering a password?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org