You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/22 07:59:12 UTC
[GitHub] [pulsar] poorbarcode opened a new pull request #14792: [fix] [package] Fixes #1234 Auto delete the package name, when package is empty
poorbarcode opened a new pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792
<!--
### Contribution Checklist
- Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where *XYZ* should be replaced by the actual issue number.
Skip *Issue XYZ* if there is no associated github issue for this pull request.
Skip *component* if you are unsure about which is the best component. E.g. `[docs] Fix typo in produce method`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
*(If this PR fixes a github issue, please add `Fixes #<xyz>`.)*
Fixes #<xyz>
*(or if this PR is one task of a github issue, please add `Master Issue: #<xyz>` to link to the master issue.)*
Master Issue: #<xyz>
### Motivation
*Explain here the context, and why you're making that change. What is the problem you're trying to solve.*
### Modifications
*Describe the modifications you've done.*
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
- [ ] `doc-required`
(If you need help on updating docs, create a doc issue)
- [ ] `no-need-doc`
(Please explain why)
- [ ] `doc`
(If this PR contains doc changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1079649074
> >
>
> Hi, @mattisonchao
>
> If the user needs to delete all versions of the data under a package, then the API is required( I agree ).
>
> But this issue aims to solve this problem: users create a transient package and delete it, but left a dirty record.
>
> Users should not be forced to use other apis to fix this dirty record.
>
> There has no behaviour API for deleting the package.
>
> So I think this plan is reasonable.
Makes sense to me, thanks for the explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1077077147
>
If the user needs to delete all versions of the data under a package, then the API is required( I agree ).
But the issue aims to solve this problem:
users create a transient package and delete it, but left a dirty record.
Users should not be forced to use other apis to fix this dirty record.
There has no behaviour API for deleting the package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1074984338
@poorbarcode:Thanks for providing doc info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode edited a comment on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
poorbarcode edited a comment on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1077077147
>
Hi, @mattisonchao
If the user needs to delete all versions of the data under a package, then the API is required( I agree ).
But the issue aims to solve this problem:
users create a transient package and delete it, but left a dirty record.
Users should not be forced to use other apis to fix this dirty record.
There has no behaviour API for deleting the package.
So I think this plan is reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
zymap commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1078569336
Could you please try to rebase the master code? I am not sure why the CI is not running
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a change in pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on a change in pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#discussion_r835143360
##########
File path: pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java
##########
@@ -139,12 +139,30 @@ public void initialize(PackagesStorage storage) {
.thenCompose(ignore -> storage.writeAsync(packagePath(packageName), inputStream));
}
+ /***
+ * @see {@link PackagesManagement#delete}.
+ * At this implements, there is a problem:
+ * Concurrently Another command: `create node` when after "getLogs" and before "deleteLog",
+ * internal implements is recursive-delete, we could not delete every children-nodes
+ * To resolve this problem, I submit a issue&PR to bookkeeper reporitory:
+ * https://github.com/apache/bookkeeper/issues/3132
+ * https://github.com/apache/bookkeeper/pull/3133
+ * TODO After the PR merged, change this implements.
Review comment:
This pr sovle problem : "dirty zk node".
But there has one thing not perfect, the case like this:
User A create a package: "function://public/default/example@v0.1", then User A want to delete it, so he exec the command "delete function://public/default/example@v0.1"
At the same time, User B want create package "function://public/default/example@v0.2"
If the process running at this line :
````
if (list == null || list.isEmpty()){
return storage.deleteAsync(packageWithoutVersionPath(packageName)); // running at this line
} else {
return CompletableFuture.completedFuture(null);
}
````
delete command will delete `example@v0.2` also.
So package module need a api for "delete zk-node only empty", this PR "https://github.com/apache/bookkeeper/pull/3133" provided
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1078883340
> Could you please try to rebase the master code? I am not sure why the CI is not running
ok, I did
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1076451066
Hi, @poorbarcode
This change looks to change the original method behaviour.
I think we can do what @zymap mentioned.
> We'd better introduce a delete all packages rest API to handle deleting all the versions of a package. And remove the related root path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on a change in pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
zymap commented on a change in pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#discussion_r840150511
##########
File path: pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java
##########
@@ -139,12 +139,30 @@ public void initialize(PackagesStorage storage) {
.thenCompose(ignore -> storage.writeAsync(packagePath(packageName), inputStream));
}
+ /***
+ * @see {@link PackagesManagement#delete}.
+ * At this implements, there is a problem:
+ * Concurrently Another command: `create node` when after "getLogs" and before "deleteLog",
+ * internal implements is recursive-delete, we could not delete every children-nodes
+ * To resolve this problem, I submit a issue&PR to bookkeeper reporitory:
+ * https://github.com/apache/bookkeeper/issues/3132
+ * https://github.com/apache/bookkeeper/pull/3133
+ * TODO After the PR merged, change this implements.
Review comment:
I still think the TODO should be in the storage API, not the package management API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on a change in pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
zymap commented on a change in pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#discussion_r834879758
##########
File path: pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java
##########
@@ -139,12 +139,30 @@ public void initialize(PackagesStorage storage) {
.thenCompose(ignore -> storage.writeAsync(packagePath(packageName), inputStream));
}
+ /***
+ * @see {@link PackagesManagement#delete}.
+ * At this implements, there is a problem:
+ * Concurrently Another command: `create node` when after "getLogs" and before "deleteLog",
+ * internal implements is recursive-delete, we could not delete every children-nodes
+ * To resolve this problem, I submit a issue&PR to bookkeeper reporitory:
+ * https://github.com/apache/bookkeeper/issues/3132
+ * https://github.com/apache/bookkeeper/pull/3133
+ * TODO After the PR merged, change this implements.
Review comment:
I think that is a storage issue, not the package management core issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
zymap commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1078568729
> >
>
> Hi, @mattisonchao
>
> If the user needs to delete all versions of the data under a package, then the API is required( I agree ).
>
> But this issue aims to solve this problem: users create a transient package and delete it, but left a dirty record.
>
> Users should not be forced to use other apis to fix this dirty record.
>
> There has no behaviour API for deleting the package.
>
> So I think this plan is reasonable.
make sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on a change in pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
zymap commented on a change in pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#discussion_r836010060
##########
File path: pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java
##########
@@ -139,12 +139,30 @@ public void initialize(PackagesStorage storage) {
.thenCompose(ignore -> storage.writeAsync(packagePath(packageName), inputStream));
}
+ /***
+ * @see {@link PackagesManagement#delete}.
+ * At this implements, there is a problem:
+ * Concurrently Another command: `create node` when after "getLogs" and before "deleteLog",
+ * internal implements is recursive-delete, we could not delete every children-nodes
+ * To resolve this problem, I submit a issue&PR to bookkeeper reporitory:
+ * https://github.com/apache/bookkeeper/issues/3132
+ * https://github.com/apache/bookkeeper/pull/3133
+ * TODO After the PR merged, change this implements.
Review comment:
Don't we need to change the [bk storage implementation](https://github.com/apache/pulsar/blob/85c1ba50cb155111f969403a092affd536d9dcb9/pulsar-package-management/bookkeeper-storage/src/main/java/org/apache/pulsar/packages/management/storage/bookkeeper/BookKeeperPackagesStorage.java#L150)? If the delete path is not empty, we should throw exception from the delete?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao edited a comment on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
mattisonchao edited a comment on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1076451066
Hi, @poorbarcode
This change looks to change the original method behaviour.
I think we can do what @zymap mentioned.
> We'd better introduce a delete all packages rest API to handle deleting all the versions of a package. And remove the related root path.
Thank you and let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode edited a comment on pull request #14792: [fix] [package] Fix Auto delete the package name issue
Posted by GitBox <gi...@apache.org>.
poorbarcode edited a comment on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1077077147
>
Hi, @mattisonchao
If the user needs to delete all versions of the data under a package, then the API is required( I agree ).
But this issue aims to solve this problem:
users create a transient package and delete it, but left a dirty record.
Users should not be forced to use other apis to fix this dirty record.
There has no behaviour API for deleting the package.
So I think this plan is reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #14792: [fix] [package] Fixes #1234 Auto delete the package name, when package is empty
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14792:
URL: https://github.com/apache/pulsar/pull/14792#issuecomment-1074868718
@poorbarcode:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org