You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/08/26 01:03:15 UTC
[GitHub] [pulsar] freeznet opened a new pull request #11782: [package management service] fix `wrappedBuffer` always having the same block of memory
freeznet opened a new pull request #11782:
URL: https://github.com/apache/pulsar/pull/11782
Fixes #11781
### Motivation
When uploading file content to bk, the `BookKeeperPackagesStorage` always use the same block of memory to read the file content and build up a bunch of zero-copy `ByteBuf`.
This causes the `ByteBuf`s having the same block of memory and break the file content.
### Modifications
Alloc new memory block in each iteration.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
#### For contributor
For this PR, do we need to update docs?
- If yes, please update docs or create a follow-up issue if you need help.
- If no, please explain why.
#### For committer
For this PR, do we need to update docs?
- If yes,
- if you update docs in this PR, label this PR with the `doc` label.
- if you plan to update docs later, label this PR with the `doc-required` label.
- if you need help on updating docs, create a follow-up issue with the `doc-required` label.
- If no, label this PR with the `no-need-doc` label and explain why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #11782: [package management service] fix `wrappedBuffer` always using the same block of memory
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #11782:
URL: https://github.com/apache/pulsar/pull/11782#issuecomment-907007658
Good catch @freeznet ! I also added "release/2.7.4" since this looks like an important fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #11782: [package management service] fix `wrappedBuffer` always using the same block of memory
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #11782:
URL: https://github.com/apache/pulsar/pull/11782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] freeznet commented on a change in pull request #11782: [package management service] fix `wrappedBuffer` always using the same block of memory
Posted by GitBox <gi...@apache.org>.
freeznet commented on a change in pull request #11782:
URL: https://github.com/apache/pulsar/pull/11782#discussion_r696346732
##########
File path: pulsar-package-management/bookkeeper-storage/src/main/java/org/apache/pulsar/packages/management/storage/bookkeeper/DLOutputStream.java
##########
@@ -58,11 +58,12 @@ private DLOutputStream(DistributedLogManager distributedLogManager, AsyncLogWrit
try {
int read = 0;
while ((read = inputStream.read(readBuffer)) != -1) {
- log.info("write something into the ledgers " + offset);
+ log.info("write something into the ledgers offset: {}, length: {}", offset, read);
ByteBuf writeBuf = Unpooled.wrappedBuffer(readBuffer, 0, read);
Review comment:
addressed in the coming commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #11782: [package management service] fix `wrappedBuffer` always using the same block of memory
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #11782:
URL: https://github.com/apache/pulsar/pull/11782#issuecomment-907009953
or does it only apply to 2.8.x + ? I see that the change was made to DLOutputStream, but I guess that's not used asynchronously by Pulsar Functions upload, so that wouldn't be impacted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] freeznet commented on pull request #11782: [package management service] fix `wrappedBuffer` always using the same block of memory
Posted by GitBox <gi...@apache.org>.
freeznet commented on pull request #11782:
URL: https://github.com/apache/pulsar/pull/11782#issuecomment-907019122
@lhotari thanks for asking. from my understanding this fix only applies to package management service, and not with Pulsar Functions upload, so it might not affect release 2.7.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on a change in pull request #11782: [package management service] fix `wrappedBuffer` always using the same block of memory
Posted by GitBox <gi...@apache.org>.
zymap commented on a change in pull request #11782:
URL: https://github.com/apache/pulsar/pull/11782#discussion_r696244831
##########
File path: pulsar-package-management/bookkeeper-storage/src/main/java/org/apache/pulsar/packages/management/storage/bookkeeper/DLOutputStream.java
##########
@@ -58,11 +58,12 @@ private DLOutputStream(DistributedLogManager distributedLogManager, AsyncLogWrit
try {
int read = 0;
while ((read = inputStream.read(readBuffer)) != -1) {
- log.info("write something into the ledgers " + offset);
+ log.info("write something into the ledgers offset: {}, length: {}", offset, read);
ByteBuf writeBuf = Unpooled.wrappedBuffer(readBuffer, 0, read);
Review comment:
I think we can copy the bytes with `copiedBuffer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org