You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/06 05:16:51 UTC
[GitHub] [pulsar] Technoboy- opened a new pull request, #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Technoboy- opened a new pull request, #16412:
URL: https://github.com/apache/pulsar/pull/16412
Master Issue: #16070
### Motivation
Add pulsar-admin command to execute scanOffloadedLedgers, introduced with #14930
### Documentation
- [x] `doc-not-needed`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#discussion_r921183462
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/Namespaces.java:
##########
@@ -2247,49 +2242,18 @@ public void removeNamespaceResourceGroup(@PathParam("tenant") String tenant,
@ApiOperation(value = "Trigger the scan of offloaded Ledgers on the LedgerOffloader for the given namespace")
@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have admin permission"),
@ApiResponse(code = 404, message = "Namespace doesn't exist") })
- public Response scanOffloadedLedgers(@PathParam("tenant") String tenant,
- @PathParam("namespace") String namespace) {
+ public void scanOffloadedLedgers(@Suspended final AsyncResponse asyncResponse,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace) {
validateNamespaceName(tenant, namespace);
- try {
- StreamingOutput output = (outputStream) -> {
Review Comment:
@hangc0276 requested this output to be served in a streaming way.
the result maybe really huge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1186674271
@eolivelli @hangc0276 I think the streaming style you are talking about is similar to https://trino.io/docs/current/develop/client-protocol.html ? The client can get part of the data for each request and perform the next request based on the last response `nextUrl`. I'm not sure how https://github.com/apache/pulsar/pull/14930 can resolve the issue of the server-side run for a long time, but the client-side will not get a timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
shibd commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1176046954
@Technoboy- Need update docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
eolivelli commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1184481510
I had started to work on a prototype
https://github.com/apache/pulsar/compare/master...eolivelli:pulsar:impl/pulsar-admin-scanoffloaded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on code in PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#discussion_r921728220
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/Namespaces.java:
##########
@@ -2247,49 +2242,18 @@ public void removeNamespaceResourceGroup(@PathParam("tenant") String tenant,
@ApiOperation(value = "Trigger the scan of offloaded Ledgers on the LedgerOffloader for the given namespace")
@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have admin permission"),
@ApiResponse(code = 404, message = "Namespace doesn't exist") })
- public Response scanOffloadedLedgers(@PathParam("tenant") String tenant,
- @PathParam("namespace") String namespace) {
+ public void scanOffloadedLedgers(@Suspended final AsyncResponse asyncResponse,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace) {
validateNamespaceName(tenant, namespace);
- try {
- StreamingOutput output = (outputStream) -> {
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
eolivelli commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1199167233
in #14930 we send the data in a streaming fashion, without accumulating the results on the server.
now the problem is to read it in a streaming way on the client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
eolivelli commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1184497445
I suggest to not change the "StreamingOutput" part, maybe there is a better way to do it, but I did not find.
There should be some way in the client to do Streaming Reads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on code in PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#discussion_r921177150
##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Namespaces.java:
##########
@@ -4467,4 +4467,18 @@ void setIsAllowAutoUpdateSchema(String namespace, boolean isAllowAutoUpdateSchem
* @return
*/
CompletableFuture<Void> removeNamespaceResourceGroupAsync(String namespace);
+
+ /**
+ * Scan offloaded ledgers on a namespace.
+ * @param namespace
+ * @return
+ */
+ String scanOffloadedLedgers(String namespace) throws PulsarAdminException;
Review Comment:
yes, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1229667421
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#discussion_r925292652
##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Namespaces.java:
##########
@@ -4467,4 +4467,18 @@ void setIsAllowAutoUpdateSchema(String namespace, boolean isAllowAutoUpdateSchem
* @return
*/
CompletableFuture<Void> removeNamespaceResourceGroupAsync(String namespace);
+
+ /**
+ * Scan offloaded ledgers on a namespace.
+ * @param namespace
+ * @return
+ */
+ String scanOffloadedLedgers(String namespace) throws PulsarAdminException;
Review Comment:
as a short term solution this API works.
Ideally I would do it as:
`void scanOffloadedLedgers(String namespace, Consumer<String> lineReader) throws PulsarAdminException;
`
and pass to the Consumer each "line" of the response
or better:
```
void scanOffloadedLedgers(String namespace, ScanOffloadedLedgerCallback callback) throws PulsarAdminException;
interface ScanOffloadedLedgerCallback {
void begin(ScanControl control);
void value(String key, Object value); // single value, like "total"....
void ledger(OffloadedLedgerInfo info) // called for each ledger
void end(); // signal the end of the result
}
class OffloadedLedgerInfo {
long ledgerId;
....
}
interface ScanControl {
void stopScan(); // close the HTTP connection/client abruptly
}
```
I didn't implement the third option because it requires to use some JSON Streaming Scanner (and I don't have experience)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feature][CLI] Add commands to run 'scanOffloadedLedgers'. [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
URL: https://github.com/apache/pulsar/pull/16412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feature][CLI] Add commands to run 'scanOffloadedLedgers'. [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-6369511071
Hi @Technoboy-, closing this one as part of the open-PR backlog triage. The `scanOffloadedLedgers` endpoint still has no admin-client or CLI support on master, but this would need a full rework: the CLI moved to picocli, and the open change requests asked for a streaming client API instead of one `String`. Note the endpoint's output also isn't valid JSON today (missing quotes/commas). Happy to review a fresh PR if you or anyone else wants to pick it up. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#discussion_r921182657
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/Namespaces.java:
##########
@@ -2247,49 +2242,18 @@ public void removeNamespaceResourceGroup(@PathParam("tenant") String tenant,
@ApiOperation(value = "Trigger the scan of offloaded Ledgers on the LedgerOffloader for the given namespace")
@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have admin permission"),
@ApiResponse(code = 404, message = "Namespace doesn't exist") })
- public Response scanOffloadedLedgers(@PathParam("tenant") String tenant,
- @PathParam("namespace") String namespace) {
+ public void scanOffloadedLedgers(@Suspended final AsyncResponse asyncResponse,
+ @PathParam("tenant") String tenant,
+ @PathParam("namespace") String namespace) {
validateNamespaceName(tenant, namespace);
- try {
- StreamingOutput output = (outputStream) -> {
Review Comment:
why are you changing this ?
the result will be huge sometimes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1185074174
> I prefer to use the original streaming way to return the scan results. Because it will be huge and will cost very long time to scan.
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#discussion_r919597132
##########
pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Namespaces.java:
##########
@@ -4467,4 +4467,18 @@ void setIsAllowAutoUpdateSchema(String namespace, boolean isAllowAutoUpdateSchem
* @return
*/
CompletableFuture<Void> removeNamespaceResourceGroupAsync(String namespace);
+
+ /**
+ * Scan offloaded ledgers on a namespace.
+ * @param namespace
+ * @return
+ */
+ String scanOffloadedLedgers(String namespace) throws PulsarAdminException;
Review Comment:
It is better to have a Pojo for the new API? I'm not sure why #14930 makes the response return a String, not an `OffloadedLedgersStats` or `OffloadedLedgersInfo`. For the admin, returning an Object will be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #16412: [feature][CLI] Add commands to run 'scanOffloadedLedgers'.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on PR #16412:
URL: https://github.com/apache/pulsar/pull/16412#issuecomment-1176167734
> @Technoboy- Need update docs?
No need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org