You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/11/03 17:41:16 UTC
[GitHub] [pulsar] lhotari opened a new pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
lhotari opened a new pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606
### Motivation
There's a concern that passing OpAddEntry across thread boundaries has thread safety issues combined to the use of recycled OpAddEntry object instances.
### Modifications
To avoid any thread safety issues caused by passing OpAddEntry, the OpAddEntry instance can be created in the same thread where it is handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari removed a comment on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari removed a comment on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959787897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959787240
I'll need to revisit the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959976341
This change makes sense to me.
Do you have any evidence of a problem that this PR is fixing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-961540500
@merlimat Please help review this PR, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959787897
The orignal PR itself is invalid since buffer would have to be retained before passing to the other thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#discussion_r742903285
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -699,10 +699,14 @@ public void asyncAddEntry(ByteBuf buffer, AddEntryCallback callback, Object ctx)
log.debug("[{}] asyncAddEntry size={} state={}", name, buffer.readableBytes(), state);
}
- OpAddEntry addOperation = OpAddEntry.create(this, buffer, callback, ctx);
+ // retain buffer in this thread
+ buffer.retain();
Review comment:
Is it necessary to retain the buffer in this thread?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#discussion_r742986699
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -699,10 +699,14 @@ public void asyncAddEntry(ByteBuf buffer, AddEntryCallback callback, Object ctx)
log.debug("[{}] asyncAddEntry size={} state={}", name, buffer.readableBytes(), state);
}
- OpAddEntry addOperation = OpAddEntry.create(this, buffer, callback, ctx);
+ // retain buffer in this thread
+ buffer.retain();
Review comment:
Yes. It also leads to test failures unless it's taken care of in the current thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] removed a comment on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
github-actions[bot] removed a comment on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959770438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-961540500
@merlimat Please help review this PR, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari removed a comment on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari removed a comment on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959787897
The orignal PR itself is invalid since buffer would have to be retained before passing to the other thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-960458045
> This change makes sense to me. Do you have any evidence of a problem that this PR is fixing?
Direct evidence in a form of a failing test is hard for a lot of thread safety issues.
In many cases, it is worth checking that code follows the rules of the Java Memory Model (JMM), specified in the Java Language Specification (JLS). In Java on 64-bit JVMs, there isn't a problem of corruption of memory pointers since "word tearing" problems are prevented. The main issue is possible data consistency and data race issues. ["Incorrectly Synchronized Programs May Exhibit Surprising Behavior"](https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4-A) is how JLS explains it. I guess the verification activity is like a model checking exercise, but at a different level.
My assumption is that the use of recycled objects and sharing them between multiple threads leads to "incorrectly synchronized programs" within Pulsar code base. It doesn't corrupt memory, but we might see "surprising behavior". One of the surprising behavior comes from the possibility of a field value changing even between subsequent reads while the field value hasn't been changed. This is explained in [Aleksey Shipilëv's Java Memory Model Unlearning Experience presentation](https://www.youtube.com/watch?v=TK-7GCCDF_I). [at about 21 minutes 50 seconds](https://www.youtube.com/watch?v=TK-7GCCDF_I&t=21m50s), there's an interesting example:
In this example, the first thread makes a single change to field x and sets it to 1. The second thread reads this field twice. On the first read, the value is 1 and on the second read, the value it 0. This is something that one would assume that "is impossible". This is one example of surprising behavior of incorrectly synchronized programs.
There are multiple ways to mitigate the issue. The safest general approach is to ensure that recycled objects are exposed only in a single thread.
The main motivation of this PR is to reduce the likelyhood of "surprising behavior" related to the use of Netty recycled object, OpAddEntry. The Netty Recycler pool is thread local. When the creation (and lookup from the Netty Recycler) of OpAddEntry is moved in the same thread where the OpAddEntry is mainly mutated and read, it reduces the likelyhood of possible data races caused by sharing objects across threads without proper synchronization. This isn't perfect, but it's an improvement. Currently ManagedLedgerImpl uses a single thread per ledger. There are gaps in the solution. The goal of my draft PR #11387 would be to continue the improvements and eventually fix the current thread safety issues that the use of recycled OpAddEntry and OpReadEntry instances introduce in Pulsar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959770920
@lhotari:Thanks for providing doc info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959770438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] removed a comment on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
github-actions[bot] removed a comment on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959770438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari removed a comment on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari removed a comment on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959787897
The orignal PR itself is invalid since buffer would have to be retained before passing to the other thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] removed a comment on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
github-actions[bot] removed a comment on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959770438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959976341
This change makes sense to me.
Do you have any evidence of a problem that this PR is fixing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959770438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959787240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959770438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-960458045
> This change makes sense to me. Do you have any evidence of a problem that this PR is fixing?
Direct evidence in a form of a failing test is hard for a lot of thread safety issues.
In many cases, it is worth checking that code follows the rules of the Java Memory Model (JMM), specified in the Java Language Specification (JLS). In Java on 64-bit JVMs, there isn't a problem of corruption of memory pointers since "word tearing" problems are prevented. The main issue is possible data consistency and data race issues. ["Incorrectly Synchronized Programs May Exhibit Surprising Behavior"](https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4-A) is how JLS explains it. I guess the verification activity is like a model checking exercise, but at a different level.
My assumption is that the use of recycled objects and sharing them between multiple threads leads to "incorrectly synchronized programs" within Pulsar code base. It doesn't corrupt memory, but we might see "surprising behavior". One of the surprising behavior comes from the possibility of a field value changing even between subsequent reads while the field value hasn't been changed. This is explained in [Aleksey Shipilëv's Java Memory Model Unlearning Experience presentation](https://www.youtube.com/watch?v=TK-7GCCDF_I). [at about 21 minutes 50 seconds](https://www.youtube.com/watch?v=TK-7GCCDF_I&t=21m50s), there's an interesting example:
In this example, the first thread makes a single change to field x and sets it to 1. The second thread reads this field twice. On the first read, the value is 1 and on the second read, the value is 0. This is something that one would assume that "is impossible". This is one example of surprising behavior of incorrectly synchronized programs.
There are multiple ways to mitigate the issue. The safest general approach is to ensure that recycled objects are exposed only in a single thread.
The main motivation of this PR is to reduce the likelyhood of "surprising behavior" related to the use of Netty recycled object, OpAddEntry. The Netty Recycler pool is thread local. When the creation (and lookup from the Netty Recycler) of OpAddEntry is moved in the same thread where the OpAddEntry is mainly mutated and read, it reduces the likelyhood of possible data races caused by sharing objects across threads without proper synchronization. This isn't perfect, but it's an improvement. Currently ManagedLedgerImpl uses a single thread per ledger. There are gaps in the solution. The goal of my draft PR #11387 would be to continue the improvements and eventually fix the current thread safety issues that the use of recycled OpAddEntry and OpReadEntry instances introduce in Pulsar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959787240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-959976341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606#issuecomment-961540500
@merlimat Please help review this PR, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat merged pull request #12606: [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry
Posted by GitBox <gi...@apache.org>.
merlimat merged pull request #12606:
URL: https://github.com/apache/pulsar/pull/12606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org