You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by pe...@apache.org on 2022/05/20 15:05:42 UTC
[pulsar] branch branch-2.10 updated (307133c59c2 -> 6415b313da9)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a change to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
from 307133c59c2 [improve][offloaders] Upgrade JClouds to 2.5.0 (#15649)
new e19cee1203f [CI] Upgrade zlib version to 1.2.12 (#14964)
new 1952a9cd876 Use tlsCertRefreshCheckDurationSec instead of 0 for refresh value (#15075)
new a6eb62286fd [enh][monitor]: add metrics for pulsar web service thread pool (#14742)
new e6b5ec9a503 Add KeyStore support in WebSocket, Function Worker HTTPS Servers (#15084)
new 82095a4129e [Flaky-test] BatchSourceExecutorTest.testLifeCycle (#10870) (#14717)
new f829ca99068 [Broker] Fix typo in enum name and handle closing of the channel properly since writeAndFlush is asynchronous (#15384)
new cb4e2161f41 [Improve][admin|client] AsyncHttpConnector doesn't use the system properties configured (#15307)
new fb0cb768728 [improve][broker-web&websocket&proxy&function-worker] Full-support set ssl provider, ciphers and protocols (#13740)
new efc53a0be83 fix bug in contructor of RocksdbMetadataStore (#15405)
new 352bb76dfab [Proxy/Client] Fix DNS server denial-of-service issue when DNS entry expires (#15403)
new d776e17dcaf [Proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler (#15415)
new 5a9b56a9865 [improve][java-client] Add pending messages information while print the producer stats (#15440)
new 46d6a7f41b7 [fix][package-management] Fix the new path `/data` introduced regression (#15367)
new 32f123c4beb [Improve][doc] Add config of IO and acceptor threads in proxy (#15340)
new c9ca16e95bb Revert "Fix: LockManagerTest.updateValue is flaky (#13911)" (#15235)
new 8b4d81e57a1 [improve][client] improve logic when ACK grouping tracker checks duplicated message id (#15465)
new d3c08dba919 Support handling single role and non-jwt-token in MultiRolesTokenAuthorizationProvider (#14857)
new 6cbead56091 [security] Remove sensitive msg from consumer/producer stats log (#15483)
new 836ed1e25fc [fix][broker] Fix MultiRolesTokenAuthorizationProvider `authorize` issue. (#15454)
new 3dcdf2a6993 Fix grant all permissions but can't list topic. (#15501)
new 810d707d8fc [fix][txn] Topic transaction buffer recover don't close reader when throw RuntimeException (#15361)
new 761c42d33ef [fix][broker]Close publishLimiter when disable it (#15520)
new e3ba66efc82 [PIP-163][Txn]Add lowWaterMark check before appending entry to TB (#15424)
new 530fafccaec Fix http produce msg redirect issue. (#15551)
new d23e251a9b3 [fix][broker] Fix to avoid TopicStatsImpl NPE even if producerName is null (#15502)
new f10f4456574 [PIP-153][optimize][txn] Optimize metadataPositions in MLPendingAckStore (#15137)
new 08219aef884 [improve][common] Use `Collection` to instead of `List` parameter type (#15329)
new a4e6e2e3651 [Java Client] Fix messages sent by producers without schema cannot be decoded (#15622)
new e3add83871c [cleanup][broker] Override close method to avoid caching exception (#15529)
new 9dead563ad0 Fix potential to add duplicated consumer (#15051)
new 6415b313da9 [fix][broker] fix MetadataStoreException$NotFoundException while doing topic lookup (#15633)
The 31 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
conf/broker.conf | 11 ++
conf/functions_worker.yml | 40 +++++
conf/proxy.conf | 38 +++++
conf/standalone.conf | 3 +
conf/websocket.conf | 35 +++++
docker/pulsar/scripts/gen-yml-from-env.py | 4 +-
.../apache/pulsar/broker/ServiceConfiguration.java | 17 ++-
.../MultiRolesTokenAuthorizationProvider.java | 35 +----
.../authorization/PulsarAuthorizationProvider.java | 2 +-
.../PrometheusMetricsGeneratorUtils.java | 4 +-
.../java/org/apache/pulsar/jetty/package-info.java | 16 +-
.../pulsar/jetty/tls/JettySslContextFactory.java | 116 ++++++++++++++
.../org/apache/pulsar/jetty/tls/package-info.java | 16 +-
.../MultiRolesTokenAuthorizationProviderTest.java | 101 ++++++++++++
.../jetty/tls/JettySslContextFactoryTest.java | 130 +++++++---------
.../JettySslContextFactoryWithKeyStoreTest.java | 47 +++---
.../src/test/resources/ssl/jetty_client_key.jks | Bin 0 -> 2679 bytes
.../src/test/resources/ssl/jetty_client_trust.jks | Bin 0 -> 1207 bytes
.../src/test/resources/ssl/jetty_server_key.jks | Bin 0 -> 2679 bytes
.../src/test/resources/ssl/jetty_server_trust.jks | Bin 0 -> 1207 bytes
.../src/test/resources/ssl/my-ca/ca.pem | 0
.../src/test/resources/ssl/my-ca/client-ca.pem | 0
.../src/test/resources/ssl/my-ca/client-key.pem | 0
.../src/test/resources/ssl/my-ca/server-ca.pem | 0
.../src/test/resources/ssl/my-ca/server-key.pem | 0
.../org/apache/pulsar/broker/PulsarService.java | 2 +
.../pulsar/broker/loadbalance/ResourceUnit.java | 5 +
.../impl/ModularLoadManagerWrapper.java | 14 +-
.../loadbalance/impl/SimpleResourceUnit.java | 22 ++-
.../pulsar/broker/namespace/NamespaceService.java | 16 +-
.../org/apache/pulsar/broker/rest/TopicsBase.java | 11 +-
.../pulsar/broker/service/AbstractTopic.java | 7 +-
.../broker/service/ConnectionController.java | 16 +-
.../broker/service/PrecisPublishLimiter.java | 2 +-
.../org/apache/pulsar/broker/service/Producer.java | 2 +-
.../pulsar/broker/service/PublishRateLimiter.java | 5 +
.../broker/service/PublishRateLimiterDisable.java | 2 +-
.../broker/service/PublishRateLimiterImpl.java | 2 +-
.../apache/pulsar/broker/service/ServerCnx.java | 60 +++++---
.../service/nonpersistent/NonPersistentTopic.java | 6 +-
.../broker/service/persistent/PersistentTopic.java | 14 +-
.../buffer/impl/TopicTransactionBuffer.java | 32 +++-
.../pendingack/impl/MLPendingAckStore.java | 170 +++++++++++----------
.../pendingack/impl/MLPendingAckStoreProvider.java | 11 +-
.../transaction/util/LogIndexLagBackoff.java | 49 ++++++
.../broker/transaction/util/package-info.java | 19 +--
.../apache/pulsar/broker/web/WebExecutorStats.java | 100 ++++++++++++
.../org/apache/pulsar/broker/web/WebService.java | 17 ++-
.../apache/pulsar/broker/admin/AdminApi2Test.java | 8 +-
.../pulsar/broker/admin/PersistentTopicsTest.java | 2 +-
.../org/apache/pulsar/broker/admin/TopicsTest.java | 8 +-
.../pulsar/broker/auth/AuthorizationTest.java | 18 +--
.../loadbalance/AdvertisedListenersTest.java | 18 ++-
.../RGUsageMTAggrWaitForAllMsgsTest.java | 6 +-
.../pulsar/broker/service/ServerCnxTest.java | 147 ++++++++++++++++++
.../pulsar/broker/stats/PrometheusMetricsTest.java | 9 +-
.../TopicTransactionBufferRecoverTest.java | 11 +-
.../pulsar/broker/transaction/TransactionTest.java | 4 +-
.../buffer/TransactionLowWaterMarkTest.java | 40 +++++
.../pendingack/PendingAckMetadataTest.java | 2 +-
.../pendingack/PendingAckPersistentTest.java | 105 +++++++++++++
.../apache/pulsar/broker/web/WebServiceTest.java | 42 +++++
.../pulsar/client/api/BrokerServiceLookupTest.java | 1 +
.../apache/pulsar/client/api/SimpleSchemaTest.java | 52 ++++++-
.../pulsar/client/impl/ConnectionPoolTest.java | 38 ++---
.../java/org/apache/pulsar/schema/SchemaTest.java | 17 ++-
.../pulsar/utils/LogIndexLagBackOffTest.java | 55 +++++++
.../proxy/ProxyPublishConsumeTlsTest.java | 2 +-
.../admin/internal/http/AsyncHttpConnector.java | 1 +
pulsar-client-cpp/docker/alpine/Dockerfile | 8 +-
.../docker/alpine/Dockerfile-alpine-3.8 | 8 +-
pulsar-client-cpp/docker/manylinux1/Dockerfile | 8 +-
pulsar-client-cpp/pkg/deb/Dockerfile | 8 +-
pulsar-client-cpp/pkg/licenses/LICENSE-zlib.txt | 4 +-
pulsar-client-cpp/pkg/rpm/Dockerfile | 8 +-
.../org/apache/pulsar/client/impl/ClientCnx.java | 2 +-
.../apache/pulsar/client/impl/ConnectionPool.java | 53 +++----
.../client/impl/ConsumerStatsRecorderImpl.java | 3 +-
.../client/impl/ControlledClusterFailover.java | 1 +
.../org/apache/pulsar/client/impl/HttpClient.java | 1 +
.../org/apache/pulsar/client/impl/MessageImpl.java | 26 +++-
.../PersistentAcknowledgmentsGroupingTracker.java | 5 +-
.../apache/pulsar/client/impl/ProducerImpl.java | 17 ++-
.../pulsar/client/impl/ProducerResponse.java | 14 +-
.../client/impl/ProducerStatsRecorderImpl.java | 9 +-
.../impl/auth/oauth2/protocol/TokenClient.java | 1 +
.../client/impl/conf/ClientConfigurationData.java | 11 ++
.../impl/conf/ClientConfigurationDataTest.java | 57 +++++++
pulsar-common/pom.xml | 11 --
.../data/stats/NonPersistentTopicStatsImpl.java | 23 +--
.../common/policies/data/stats/TopicStatsImpl.java | 13 +-
.../common/util/DefaultSslContextBuilder.java | 23 ++-
.../org/apache/pulsar/common/util/FutureUtil.java | 59 ++++++-
.../apache/pulsar/common/util/SecurityUtility.java | 74 +++------
.../JettySslContextFactoryWithAutoRefresh.java | 70 ---------
.../util/keystoretls/KeyStoreSSLContext.java | 58 +------
.../NonPersistentPartitionedTopicStatsTest.java | 40 +++++
.../policies/data/PersistentTopicStatsTest.java | 38 +++++
.../apache/pulsar/common/util/FutureUtilTest.java | 43 ++++++
pulsar-functions/instance/pom.xml | 6 +
.../source/batch/BatchSourceExecutorTest.java | 11 +-
.../pulsar/functions/worker/WorkerConfig.java | 74 ++++++++-
.../worker/WorkerApiV2ResourceConfigTest.java | 18 ++-
.../functions/worker/PulsarWorkerService.java | 9 +-
.../pulsar/functions/worker/rest/WorkerServer.java | 39 ++++-
.../metadata/impl/LocalMemoryMetadataStore.java | 26 ++--
.../pulsar/metadata/impl/RocksdbMetadataStore.java | 2 +-
.../packages/management/core/PackagesStorage.java | 21 +++
.../core/impl/PackagesManagementImpl.java | 9 +-
.../core/impl/PackagesManagementImplTest.java | 69 ++++++++-
.../filesystem/FileSystemPackagesStorage.java | 5 +
.../pulsar/proxy/server/AdminProxyHandler.java | 14 +-
.../pulsar/proxy/server/LookupProxyHandler.java | 20 +--
.../pulsar/proxy/server/ProxyConfiguration.java | 10 +-
.../pulsar/proxy/server/ProxyConnection.java | 11 +-
.../apache/pulsar/proxy/server/ProxyService.java | 12 +-
.../proxy/server/ServiceChannelInitializer.java | 2 +-
.../org/apache/pulsar/proxy/server/WebServer.java | 13 +-
.../pulsar/websocket/service/ProxyServer.java | 40 +++--
.../service/WebSocketProxyConfiguration.java | 56 +++++++
site2/docs/reference-configuration.md | 2 +
121 files changed, 2164 insertions(+), 786 deletions(-)
copy pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerResponse.java => pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/package-info.java (71%)
create mode 100644 pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/JettySslContextFactory.java
copy pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerResponse.java => pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/package-info.java (71%)
copy pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java => pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryTest.java (60%)
rename pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java => pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryWithKeyStoreTest.java (90%)
create mode 100644 pulsar-broker-common/src/test/resources/ssl/jetty_client_key.jks
create mode 100644 pulsar-broker-common/src/test/resources/ssl/jetty_client_trust.jks
create mode 100644 pulsar-broker-common/src/test/resources/ssl/jetty_server_key.jks
create mode 100644 pulsar-broker-common/src/test/resources/ssl/jetty_server_trust.jks
copy {pulsar-common => pulsar-broker-common}/src/test/resources/ssl/my-ca/ca.pem (100%)
copy {pulsar-common => pulsar-broker-common}/src/test/resources/ssl/my-ca/client-ca.pem (100%)
copy {pulsar-common => pulsar-broker-common}/src/test/resources/ssl/my-ca/client-key.pem (100%)
copy {pulsar-common => pulsar-broker-common}/src/test/resources/ssl/my-ca/server-ca.pem (100%)
copy {pulsar-common => pulsar-broker-common}/src/test/resources/ssl/my-ca/server-key.pem (100%)
create mode 100644 pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/LogIndexLagBackoff.java
copy pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerResponse.java => pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/package-info.java (71%)
create mode 100644 pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebExecutorStats.java
create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/utils/LogIndexLagBackOffTest.java
create mode 100644 pulsar-client/src/test/java/org/apache/pulsar/client/impl/conf/ClientConfigurationDataTest.java
delete mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefresh.java
[pulsar] 07/31: [Improve][admin|client] AsyncHttpConnector doesn't use the system properties configured (#15307)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit cb4e2161f4151506decad92b56cae22faa75a78a
Author: AlvaroStream <10...@users.noreply.github.com>
AuthorDate: Sat Apr 30 06:24:44 2022 +0200
[Improve][admin|client] AsyncHttpConnector doesn't use the system properties configured (#15307)
(cherry picked from commit ebf2487ab2bfb35362ffdffc520b1652426f49f1)
---
.../test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java | 1 +
.../org/apache/pulsar/client/admin/internal/http/AsyncHttpConnector.java | 1 +
.../java/org/apache/pulsar/client/impl/ControlledClusterFailover.java | 1 +
.../src/main/java/org/apache/pulsar/client/impl/HttpClient.java | 1 +
.../org/apache/pulsar/client/impl/auth/oauth2/protocol/TokenClient.java | 1 +
5 files changed, 5 insertions(+)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java
index 9ffe7dda5cc..e3bb6a92dc0 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java
@@ -885,6 +885,7 @@ public class BrokerServiceLookupTest extends ProducerConsumerBase {
private AsyncHttpClient getHttpClient(String version) {
DefaultAsyncHttpClientConfig.Builder confBuilder = new DefaultAsyncHttpClientConfig.Builder();
+ confBuilder.setUseProxyProperties(true);
confBuilder.setFollowRedirect(true);
confBuilder.setUserAgent(version);
confBuilder.setKeepAliveStrategy(new DefaultKeepAliveStrategy() {
diff --git a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/http/AsyncHttpConnector.java b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/http/AsyncHttpConnector.java
index 1f302f6586c..95ea0717b97 100644
--- a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/http/AsyncHttpConnector.java
+++ b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/http/AsyncHttpConnector.java
@@ -96,6 +96,7 @@ public class AsyncHttpConnector implements Connector {
int requestTimeoutMs,
int autoCertRefreshTimeSeconds, ClientConfigurationData conf) {
DefaultAsyncHttpClientConfig.Builder confBuilder = new DefaultAsyncHttpClientConfig.Builder();
+ confBuilder.setUseProxyProperties(true);
confBuilder.setFollowRedirect(true);
confBuilder.setRequestTimeout(conf.getRequestTimeoutMs());
confBuilder.setConnectTimeout(connectTimeoutMs);
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ControlledClusterFailover.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ControlledClusterFailover.java
index 50060a25217..98ab5362a56 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ControlledClusterFailover.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ControlledClusterFailover.java
@@ -87,6 +87,7 @@ public class ControlledClusterFailover implements ServiceUrlProvider {
private AsyncHttpClient buildHttpClient() {
DefaultAsyncHttpClientConfig.Builder confBuilder = new DefaultAsyncHttpClientConfig.Builder();
+ confBuilder.setUseProxyProperties(true);
confBuilder.setFollowRedirect(true);
confBuilder.setMaxRedirects(DEFAULT_MAX_REDIRECTS);
confBuilder.setConnectTimeout(DEFAULT_CONNECT_TIMEOUT_IN_SECONDS * 1000);
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java
index eb988528cb5..8888acc37fd 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java
@@ -72,6 +72,7 @@ public class HttpClient implements Closeable {
this.serviceNameResolver.updateServiceUrl(conf.getServiceUrl());
DefaultAsyncHttpClientConfig.Builder confBuilder = new DefaultAsyncHttpClientConfig.Builder();
+ confBuilder.setUseProxyProperties(true);
confBuilder.setFollowRedirect(true);
confBuilder.setMaxRedirects(conf.getMaxLookupRedirects());
confBuilder.setConnectTimeout(DEFAULT_CONNECT_TIMEOUT_IN_SECONDS * 1000);
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/oauth2/protocol/TokenClient.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/oauth2/protocol/TokenClient.java
index b66d7eaf23a..66961ea5c95 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/oauth2/protocol/TokenClient.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/oauth2/protocol/TokenClient.java
@@ -53,6 +53,7 @@ public class TokenClient implements ClientCredentialsExchanger {
TokenClient(URL tokenUrl, AsyncHttpClient httpClient) {
if (httpClient == null) {
DefaultAsyncHttpClientConfig.Builder confBuilder = new DefaultAsyncHttpClientConfig.Builder();
+ confBuilder.setUseProxyProperties(true);
confBuilder.setFollowRedirect(true);
confBuilder.setConnectTimeout(DEFAULT_CONNECT_TIMEOUT_IN_SECONDS * 1000);
confBuilder.setReadTimeout(DEFAULT_READ_TIMEOUT_IN_SECONDS * 1000);
[pulsar] 22/31: [fix][broker]Close publishLimiter when disable it (#15520)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 761c42d33efd3602d3a602bafb738e23c59e09ac
Author: Xiaoyu Hou <An...@gmail.com>
AuthorDate: Wed May 11 20:20:28 2022 +0800
[fix][broker]Close publishLimiter when disable it (#15520)
(cherry picked from commit e8c971a2f15d9fe79eb88f92c022216a0ca57f73)
---
.../src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java | 3 +++
1 file changed, 3 insertions(+)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index 852185d4919..a4983be4bd8 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -1102,6 +1102,9 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener<TopicP
}
} else {
log.info("Disabling publish throttling for {}", this.topic);
+ if (topicPublishRateLimiter != null) {
+ topicPublishRateLimiter.close();
+ }
this.topicPublishRateLimiter = PublishRateLimiter.DISABLED_RATE_LIMITER;
enableProducerReadForPublishRateLimiting();
}
[pulsar] 16/31: [improve][client] improve logic when ACK grouping tracker checks duplicated message id (#15465)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 8b4d81e57a16bb4f41cacada4ed0d911363dd621
Author: Qiang Zhao <ma...@gmail.com>
AuthorDate: Sat May 7 15:35:04 2022 +0800
[improve][client] improve logic when ACK grouping tracker checks duplicated message id (#15465)
(cherry picked from commit f6faeecc819de880eb4a93e4bb359bebbc0bc855)
---
.../pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
index a1831e13e09..f0f0cfd7548 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
@@ -118,10 +118,7 @@ public class PersistentAcknowledgmentsGroupingTracker implements Acknowledgments
@Override
public boolean isDuplicate(@NonNull MessageId messageId) {
final MessageId messageIdOfLastAck = lastCumulativeAck.messageId;
- if (messageIdOfLastAck == null) {
- return false;
- }
- if (messageId.compareTo(messageIdOfLastAck) <= 0) {
+ if (messageIdOfLastAck != null && messageId.compareTo(messageIdOfLastAck) <= 0) {
// Already included in a cumulative ack
return true;
} else {
[pulsar] 09/31: fix bug in contructor of RocksdbMetadataStore (#15405)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit efc53a0be83e6068dfc9c427c5cbb890422cdf08
Author: codertmy <49...@users.noreply.github.com>
AuthorDate: Mon May 2 10:33:40 2022 +0800
fix bug in contructor of RocksdbMetadataStore (#15405)
Co-authored-by: tmytangmingyang <tm...@didiglobal.com>
(cherry picked from commit 406aa43ad17cab1cd71f8667ea840a5bcb4cc74b)
---
.../main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java
index 1828eb5b878..3ca87f3f915 100644
--- a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java
+++ b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java
@@ -221,7 +221,7 @@ public class RocksdbMetadataStore extends AbstractMetadataStore {
try {
Files.createDirectories(dataPath);
} catch (IOException e) {
- e.printStackTrace();
+ throw new MetadataStoreException("Fail to create RocksDB file directory", e);
}
db = openDB(dataPath.toString(), metadataStoreConfig.getConfigFilePath());
[pulsar] 28/31: [Java Client] Fix messages sent by producers without schema cannot be decoded (#15622)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a4e6e2e3651d093cc167b445531fae0e9bc24017
Author: Yunze Xu <xy...@163.com>
AuthorDate: Thu May 19 11:49:25 2022 +0800
[Java Client] Fix messages sent by producers without schema cannot be decoded (#15622)
### Motivation
When I tried to consume a topic via a consumer with Avro schema while
the topic was produced by a producer without schema, the consumption
failed. It's because `MultiVersionSchemaInfoProvider#getSchemaByVersion`
doesn't check if `schemaVersion` is an empty byte array. If yes, a
`BytesSchemaVersion` of an empty array will be passed to `cache.get` and
then passed to `loadSchema`.
https://github.com/apache/pulsar/blob/f90ef9c6ad88c4f94ce1fcc682bbf3f3189cbf2a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionSchemaInfoProvider.java#L94-L98
However, `LookupService#getSchema` cannot accept an empty byte array as
the version, so `loadSchema` failed.
The root cause is that the schema version was set unexpectly when
messages were sent by a producer without schema. At broker side, the
returned schema version is never null. If the schema version was an
empty array, then it means the message doesn't have schema. However, at
Java client side, the empty byte array is treated as an existing schema
and the schema version field will be set. When consumer receives the
message, it will try to load schema whose version is an empty array.
### Modifications
- When a producer receives a response whose schema version is an empty
byte array, just ignore it.
- Make `MesasgeImpl#getSchemaVersion` return null if the schema version
is an empty byte array so that the consumer can consume messages
produced by older version producers without schema. And return the
internal schema for `getRegetReaderSchema` when `getSchemaVersion`
returns null.
- Fix the existing tests. Since producer without schema won't set the
`schema_version` field after this patch, some tests that rely on the
precise stats should be modified.
- Add `testConsumeAvroMessagesWithoutSchema` to cover the case that
messages without schema are compatible with the schema.
This patch also modifies the existing behavior when
`schemaValidationEnforced` is false and messages are produced by a
producer without schema and consumed by a consumer with schema.
1. If the message is incompatible with the schema
- Before: `getSchemaVersion` returns an empty array and `getValue`
fails with `SerializationException`:
> org.apache.commons.lang3.SerializationException: Failed at fetching schema info for EMPTY
- After: `getSchemaVersion` returns `null` and `getValue` fails with
`SchemaSerializationException`.
2. Otherwise (the message is compatible with the schema)
- Before: `getSchemaVersion` returns an empty array and `getValue`
fails with `SerializationException`.
- After: `getSchemaVersion` returns `null` and `getValue` returns the
correctly decoded object.
(cherry picked from commit ecd275dc21f33483a649e5b872990771257b1d45)
---
.../apache/pulsar/broker/admin/AdminApi2Test.java | 8 ++--
.../pulsar/broker/admin/PersistentTopicsTest.java | 2 +-
.../RGUsageMTAggrWaitForAllMsgsTest.java | 6 +--
.../pulsar/broker/stats/PrometheusMetricsTest.java | 2 +-
.../apache/pulsar/client/api/SimpleSchemaTest.java | 52 ++++++++++++++++++++--
.../java/org/apache/pulsar/schema/SchemaTest.java | 17 +++----
.../org/apache/pulsar/client/impl/MessageImpl.java | 26 +++++++++--
.../apache/pulsar/client/impl/ProducerImpl.java | 11 +++--
.../pulsar/client/impl/ProducerResponse.java | 14 +++++-
9 files changed, 109 insertions(+), 29 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java
index a6b89112634..e8eee716faf 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java
@@ -1291,7 +1291,7 @@ public class AdminApi2Test extends MockedPulsarServiceBaseTest {
assertEquals(topicStats.getSubscriptions().get(subName).getMsgBacklog(), 10);
topicStats = admin.topics().getStats(topic, true, true);
- assertEquals(topicStats.getSubscriptions().get(subName).getBacklogSize(), 43);
+ assertEquals(topicStats.getSubscriptions().get(subName).getBacklogSize(), 40);
assertEquals(topicStats.getSubscriptions().get(subName).getMsgBacklog(), 1);
consumer.acknowledge(message);
@@ -1545,7 +1545,7 @@ public class AdminApi2Test extends MockedPulsarServiceBaseTest {
topicStats = admin.topics().getPartitionedStats(topic, false, true, true);
assertEquals(topicStats.getSubscriptions().get(subName).getMsgBacklog(), 1);
- assertEquals(topicStats.getSubscriptions().get(subName).getBacklogSize(), 43);
+ assertEquals(topicStats.getSubscriptions().get(subName).getBacklogSize(), 40);
}
@Test(timeOut = 30000)
@@ -1584,7 +1584,7 @@ public class AdminApi2Test extends MockedPulsarServiceBaseTest {
TopicStats topicStats = admin.topics().getPartitionedStats(topic, false, true, true);
assertEquals(topicStats.getSubscriptions().get(subName).getMsgBacklog(), 10);
- assertEquals(topicStats.getSubscriptions().get(subName).getBacklogSize(), 470);
+ assertEquals(topicStats.getSubscriptions().get(subName).getBacklogSize(), 440);
assertEquals(topicStats.getSubscriptions().get(subName).getMsgBacklogNoDelayed(), 5);
for (int i = 0; i < 5; i++) {
@@ -1594,7 +1594,7 @@ public class AdminApi2Test extends MockedPulsarServiceBaseTest {
Awaitility.await().untilAsserted(() -> {
TopicStats topicStats2 = admin.topics().getPartitionedStats(topic, false, true, true);
assertEquals(topicStats2.getSubscriptions().get(subName).getMsgBacklog(), 5);
- assertEquals(topicStats2.getSubscriptions().get(subName).getBacklogSize(), 238);
+ assertEquals(topicStats2.getSubscriptions().get(subName).getBacklogSize(), 223);
assertEquals(topicStats2.getSubscriptions().get(subName).getMsgBacklogNoDelayed(), 0);
});
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java
index f1040840a09..f75d41be675 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java
@@ -775,7 +775,7 @@ public class PersistentTopicsTest extends MockedPulsarServiceBaseTest {
completableFuture = batchProducer.sendAsync("a".getBytes());
}
completableFuture.get();
- Assert.assertEquals(Optional.ofNullable(admin.topics().getBacklogSizeByMessageId(topicName + "-partition-0", MessageId.earliest)), Optional.of(350L));
+ Assert.assertEquals(Optional.ofNullable(admin.topics().getBacklogSizeByMessageId(topicName + "-partition-0", MessageId.earliest)), Optional.of(320L));
}
@Test
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/RGUsageMTAggrWaitForAllMsgsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/RGUsageMTAggrWaitForAllMsgsTest.java
index 386e8bf83b5..27f9e905262 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/RGUsageMTAggrWaitForAllMsgsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/resourcegroup/RGUsageMTAggrWaitForAllMsgsTest.java
@@ -551,7 +551,7 @@ public class RGUsageMTAggrWaitForAllMsgsTest extends ProducerConsumerBase {
log.debug("verifyProdConsStats: topicStatsMap has {} entries", topicStatsMap.size());
- // Pulsar runtime adds some additional bytes in the exchanges: a 45-byte per-message
+ // Pulsar runtime adds some additional bytes in the exchanges: a 42-byte per-message
// metadata of some kind, plus more as the number of messages increases.
// Hence the ">=" assertion with ExpectedNumBytesSent/Received in the following checks.
final int ExpectedNumBytesSent = sentNumBytes + PER_MESSAGE_METADATA_OHEAD * sentNumMsgs;
@@ -769,8 +769,8 @@ public class RGUsageMTAggrWaitForAllMsgsTest extends ProducerConsumerBase {
Assert.assertNotEquals(ninthPercentileValue, 0);
}
- // Empirically, there appears to be a 45-byte overhead for metadata, imposed by Pulsar runtime.
- private static final int PER_MESSAGE_METADATA_OHEAD = 45;
+ // Empirically, there appears to be a 42-byte overhead for metadata, imposed by Pulsar runtime.
+ private static final int PER_MESSAGE_METADATA_OHEAD = 42;
private static final int PUBLISH_INTERVAL_SECS = 10;
private static final int NUM_PRODUCERS = 4;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index c2d46e2edb3..3f64619a983 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -1361,7 +1361,7 @@ public class PrometheusMetricsTest extends BrokerTestBase {
assertEquals(cm.get(0).value, 10);
cm = (List<Metric>) metrics.get("pulsar_compaction_compacted_entries_size");
assertEquals(cm.size(), 1);
- assertEquals(cm.get(0).value, 870);
+ assertEquals(cm.get(0).value, 840);
pulsarClient.close();
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
index fd8036eaf9e..983a7f341e0 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
@@ -49,7 +49,6 @@ import org.apache.pulsar.client.impl.PulsarClientImpl;
import org.apache.pulsar.client.impl.schema.reader.AvroReader;
import org.apache.pulsar.client.impl.schema.writer.AvroWriter;
import org.apache.pulsar.common.naming.TopicName;
-import org.apache.pulsar.common.protocol.schema.SchemaVersion;
import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.KeyValueEncodingType;
import org.apache.pulsar.common.schema.SchemaInfo;
@@ -62,6 +61,7 @@ import org.testng.annotations.Factory;
import org.testng.annotations.Test;
import java.io.ByteArrayInputStream;
+import java.io.EOFException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.List;
@@ -305,7 +305,13 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
+ " if SchemaValidationEnabled is enabled");
}
Message<V2Data> msg3 = c.receive();
- Assert.assertEquals(msg3.getSchemaVersion(), SchemaVersion.Empty.bytes());
+ assertNull(msg3.getSchemaVersion());
+ try {
+ msg3.getValue();
+ fail("Schema should be incompatible");
+ } catch (SchemaSerializationException e) {
+ assertTrue(e.getCause() instanceof EOFException);
+ }
} catch (PulsarClientException e) {
if (schemaValidationEnforced) {
Assert.assertTrue(e instanceof IncompatibleSchemaException);
@@ -366,7 +372,13 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
+ " if SchemaValidationEnabled is enabled");
}
Message<V2Data> msg3 = c.receive();
- Assert.assertEquals(msg3.getSchemaVersion(), SchemaVersion.Empty.bytes());
+ assertNull(msg3.getSchemaVersion());
+ try {
+ msg3.getValue();
+ fail("Schema should be incompatible");
+ } catch (SchemaSerializationException e) {
+ assertTrue(e.getCause() instanceof EOFException);
+ }
} catch (PulsarClientException e) {
if (schemaValidationEnforced) {
Assert.assertTrue(e instanceof IncompatibleSchemaException);
@@ -1253,4 +1265,38 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
}
}
+
+ @Test
+ public void testConsumeAvroMessagesWithoutSchema() throws Exception {
+ if (schemaValidationEnforced) {
+ return;
+ }
+ final String topic = "test-consume-avro-messages-without-schema-" + UUID.randomUUID();
+ final Schema<V1Data> schema = Schema.AVRO(V1Data.class);
+ final Consumer<V1Data> consumer = pulsarClient.newConsumer(schema)
+ .topic(topic)
+ .subscriptionName("sub")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
+ final Producer<byte[]> producer = pulsarClient.newProducer()
+ .topic(topic)
+ .create();
+
+ final int numMessages = 5;
+ for (int i = 0; i < numMessages; i++) {
+ producer.send(schema.encode(new V1Data(i)));
+ }
+
+ for (int i = 0; i < numMessages; i++) {
+ final Message<V1Data> msg = consumer.receive(3, TimeUnit.SECONDS);
+ assertNotNull(msg);
+ log.info("Received {} from {}", msg.getValue().i, topic);
+ assertEquals(msg.getValue().i, i);
+ assertEquals(msg.getReaderSchema().orElse(Schema.BYTES).getSchemaInfo(), schema.getSchemaInfo());
+ consumer.acknowledge(msg);
+ }
+
+ producer.close();
+ consumer.close();
+ }
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
index 217e76820f1..c661ae07bf2 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
@@ -29,7 +29,6 @@ import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
import static org.testng.internal.junit.ArrayAsserts.assertArrayEquals;
-import com.google.common.base.Throwables;
import lombok.EqualsAndHashCode;
import org.apache.avro.Schema.Parser;
import com.fasterxml.jackson.databind.JsonNode;
@@ -1119,7 +1118,7 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
stopBroker();
isTcpLookup = false;
setup();
- testEmptySchema();
+ testIncompatibleSchema();
}
@Test
@@ -1127,10 +1126,10 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
stopBroker();
isTcpLookup = true;
setup();
- testEmptySchema();
+ testIncompatibleSchema();
}
- private void testEmptySchema() throws Exception {
+ private void testIncompatibleSchema() throws Exception {
final String namespace = "test-namespace-" + randomName(16);
String ns = PUBLIC_TENANT + "/" + namespace;
admin.namespaces().createNamespace(ns, Sets.newHashSet(CLUSTER_NAME));
@@ -1164,12 +1163,14 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
producer.send("test".getBytes(StandardCharsets.UTF_8));
Message<User> message1 = consumer.receive();
Assert.assertEquals(test, message1.getValue());
+ Message<User> message2 = consumer.receive();
try {
- Message<User> message2 = consumer.receive();
message2.getValue();
- } catch (Throwable ex) {
- Assert.assertTrue(Throwables.getRootCause(ex) instanceof SchemaSerializationException);
- Assert.assertEquals(Throwables.getRootCause(ex).getMessage(),"Empty schema version");
+ } catch (SchemaSerializationException e) {
+ final String schemaString =
+ new String(Schema.AVRO(User.class).getSchemaInfo().getSchema(), StandardCharsets.UTF_8);
+ Assert.assertTrue(e.getMessage().contains(schemaString));
+ Assert.assertTrue(e.getMessage().contains("payload (4 bytes)"));
}
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
index a184bd145b8..2fb9311b4b5 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
@@ -28,6 +28,7 @@ import io.netty.util.Recycler.Handle;
import io.netty.util.ReferenceCountUtil;
import java.io.IOException;
import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Collections;
import java.util.List;
@@ -40,6 +41,7 @@ import lombok.Getter;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SchemaSerializationException;
import org.apache.pulsar.client.impl.schema.AbstractSchema;
import org.apache.pulsar.client.impl.schema.AutoConsumeSchema;
import org.apache.pulsar.client.impl.schema.KeyValueSchemaImpl;
@@ -382,12 +384,14 @@ public class MessageImpl<T> implements Message<T> {
if (schema == null) {
return Optional.empty();
}
+ byte[] schemaVersion = getSchemaVersion();
+ if (schemaVersion == null) {
+ return Optional.of(schema);
+ }
if (schema instanceof AutoConsumeSchema) {
- byte[] schemaVersion = getSchemaVersion();
return Optional.of(((AutoConsumeSchema) schema)
.atSchemaVersion(schemaVersion));
} else if (schema instanceof AbstractSchema) {
- byte[] schemaVersion = getSchemaVersion();
return Optional.of(((AbstractSchema<?>) schema)
.atSchemaVersion(schemaVersion));
} else {
@@ -395,10 +399,13 @@ public class MessageImpl<T> implements Message<T> {
}
}
+ // For messages produced by older version producers without schema, the schema version is an empty byte array
+ // rather than null.
@Override
public byte[] getSchemaVersion() {
if (msgMetadata.hasSchemaVersion()) {
- return msgMetadata.getSchemaVersion();
+ byte[] schemaVersion = msgMetadata.getSchemaVersion();
+ return (schemaVersion.length == 0) ? null : schemaVersion;
} else {
return null;
}
@@ -464,8 +471,19 @@ public class MessageImpl<T> implements Message<T> {
}
}
-
private T decode(byte[] schemaVersion) {
+ try {
+ return decodeBySchema(schemaVersion);
+ } catch (ArrayIndexOutOfBoundsException e) {
+ // It usually means the message was produced without schema check while the message is not compatible with
+ // the current schema. Therefore, convert it to SchemaSerializationException with a better description.
+ final int payloadSize = payload.readableBytes();
+ throw new SchemaSerializationException("payload (" + payloadSize + " bytes) cannot be decoded with schema "
+ + new String(schema.getSchemaInfo().getSchema(), StandardCharsets.UTF_8));
+ }
+ }
+
+ private T decodeBySchema(byte[] schemaVersion) {
T value = poolMessage ? schema.decode(payload.nioBuffer(), schemaVersion) : null;
if (value != null) {
return value;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index 0bf37a93a3a..99197262585 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -699,9 +699,14 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
}
} else {
log.info("[{}] [{}] GetOrCreateSchema succeed", topic, producerName);
- SchemaHash schemaHash = SchemaHash.of(msg.getSchemaInternal());
- schemaCache.putIfAbsent(schemaHash, v);
- msg.getMessageBuilder().setSchemaVersion(v);
+ // In broker, if schema version is an empty byte array, it means the topic doesn't have schema. In this
+ // case, we should not cache the schema version so that the schema version of the message metadata will
+ // be null, instead of an empty array.
+ if (v.length != 0) {
+ SchemaHash schemaHash = SchemaHash.of(msg.getSchemaInternal());
+ schemaCache.putIfAbsent(schemaHash, v);
+ msg.getMessageBuilder().setSchemaVersion(v);
+ }
msg.setSchemaState(MessageImpl.SchemaState.Ready);
}
cnx.ctx().channel().eventLoop().execute(() -> {
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerResponse.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerResponse.java
index efe418d89cd..7d710815bf2 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerResponse.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerResponse.java
@@ -20,9 +20,9 @@ package org.apache.pulsar.client.impl;
import java.util.Optional;
import lombok.AllArgsConstructor;
-import lombok.Data;
+import lombok.Getter;
-@Data
+@Getter
@AllArgsConstructor
public class ProducerResponse {
private String producerName;
@@ -30,4 +30,14 @@ public class ProducerResponse {
private byte[] schemaVersion;
private Optional<Long> topicEpoch;
+
+ // Shadow the default getter generated by lombok. In broker, if the schema version is an empty byte array, it means
+ // the topic doesn't have schema.
+ public byte[] getSchemaVersion() {
+ if (schemaVersion != null && schemaVersion.length != 0) {
+ return schemaVersion;
+ } else {
+ return null;
+ }
+ }
}
[pulsar] 01/31: [CI] Upgrade zlib version to 1.2.12 (#14964)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e19cee1203ff7585cb12f804cc0bbed2af869cce
Author: Lishen Yao <ya...@gmail.com>
AuthorDate: Fri Apr 1 00:42:21 2022 +0800
[CI] Upgrade zlib version to 1.2.12 (#14964)
Co-authored-by: Zike Yang <zi...@apache.org>
(cherry picked from commit 8b2b988654887cdb9d123698d5237381704f89c3)
---
pulsar-client-cpp/docker/alpine/Dockerfile | 8 ++++----
pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8 | 8 ++++----
pulsar-client-cpp/docker/manylinux1/Dockerfile | 8 ++++----
pulsar-client-cpp/pkg/deb/Dockerfile | 8 ++++----
pulsar-client-cpp/pkg/licenses/LICENSE-zlib.txt | 4 ++--
pulsar-client-cpp/pkg/rpm/Dockerfile | 8 ++++----
6 files changed, 22 insertions(+), 22 deletions(-)
diff --git a/pulsar-client-cpp/docker/alpine/Dockerfile b/pulsar-client-cpp/docker/alpine/Dockerfile
index ef77284242c..12d1e2f9f97 100644
--- a/pulsar-client-cpp/docker/alpine/Dockerfile
+++ b/pulsar-client-cpp/docker/alpine/Dockerfile
@@ -43,12 +43,12 @@ RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.72.0/source/
rm -rf /boost_1_72_0.tar.gz /boost_1_72_0
# ZLib
-RUN curl -O -L https://zlib.net/zlib-1.2.11.tar.gz && \
- tar xfz zlib-1.2.11.tar.gz && \
- cd zlib-1.2.11 && \
+RUN curl -O -L https://zlib.net/zlib-1.2.12.tar.gz && \
+ tar xfz zlib-1.2.12.tar.gz && \
+ cd zlib-1.2.12 && \
CFLAGS="-fPIC -O3" ./configure && \
make -j4 && make install && \
- rm -rf /zlib-1.2.11.tar.gz /zlib-1.2.11
+ rm -rf /zlib-1.2.12.tar.gz /zlib-1.2.12
# Compile OpenSSL
RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_0j.tar.gz && \
diff --git a/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8 b/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8
index 0a9fbb40711..862785e9015 100644
--- a/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8
+++ b/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8
@@ -43,12 +43,12 @@ RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.72.0/source/
rm -rf /boost_1_72_0.tar.gz /boost_1_72_0
# ZLib
-RUN curl -O -L https://zlib.net/zlib-1.2.11.tar.gz && \
- tar xfz zlib-1.2.11.tar.gz && \
- cd zlib-1.2.11 && \
+RUN curl -O -L https://zlib.net/zlib-1.2.12.tar.gz && \
+ tar xfz zlib-1.2.12.tar.gz && \
+ cd zlib-1.2.12 && \
CFLAGS="-fPIC -O3" ./configure && \
make -j4 && make install && \
- rm -rf /zlib-1.2.11.tar.gz /zlib-1.2.11
+ rm -rf /zlib-1.2.12.tar.gz /zlib-1.2.12
# Compile OpenSSL
RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_0j.tar.gz && \
diff --git a/pulsar-client-cpp/docker/manylinux1/Dockerfile b/pulsar-client-cpp/docker/manylinux1/Dockerfile
index 0df92401a8f..502c6f6fe77 100644
--- a/pulsar-client-cpp/docker/manylinux1/Dockerfile
+++ b/pulsar-client-cpp/docker/manylinux1/Dockerfile
@@ -46,12 +46,12 @@ RUN curl -O -L https://www.cpan.org/src/5.0/perl-5.10.0.tar.gz && \
####################################
# ZLib
-RUN curl -O -L https://zlib.net/zlib-1.2.11.tar.gz && \
- tar xvfz zlib-1.2.11.tar.gz && \
- cd zlib-1.2.11 && \
+RUN curl -O -L https://zlib.net/zlib-1.2.12.tar.gz && \
+ tar xvfz zlib-1.2.12.tar.gz && \
+ cd zlib-1.2.12 && \
CFLAGS="-fPIC -O3" ./configure && \
make && make install && \
- rm -rf /zlib-1.2.11.tar.gz /zlib-1.2.11
+ rm -rf /zlib-1.2.12.tar.gz /zlib-1.2.12
# Compile OpenSSL
RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_0j.tar.gz && \
diff --git a/pulsar-client-cpp/pkg/deb/Dockerfile b/pulsar-client-cpp/pkg/deb/Dockerfile
index 77c3f470c06..1925827b56a 100644
--- a/pulsar-client-cpp/pkg/deb/Dockerfile
+++ b/pulsar-client-cpp/pkg/deb/Dockerfile
@@ -49,12 +49,12 @@ RUN curl -O -L https://github.com/google/protobuf/releases/download/v3.3.0/prot
rm -rf /protobuf-cpp-3.3.0.tar.gz /protobuf-3.3.0
# ZLib
-RUN curl -O -L https://github.com/madler/zlib/archive/v1.2.11.tar.gz && \
- tar xvfz v1.2.11.tar.gz && \
- cd zlib-1.2.11 && \
+RUN curl -O -L https://github.com/madler/zlib/archive/v1.2.12.tar.gz && \
+ tar xvfz v1.2.12.tar.gz && \
+ cd zlib-1.2.12 && \
CFLAGS="-fPIC -O3" ./configure && \
make && make install && \
- rm -rf /v1.2.11.tar.gz /zlib-1.2.11
+ rm -rf /v1.2.12.tar.gz /zlib-1.2.12
# Zstandard
RUN curl -O -L https://github.com/facebook/zstd/releases/download/v1.3.7/zstd-1.3.7.tar.gz && \
diff --git a/pulsar-client-cpp/pkg/licenses/LICENSE-zlib.txt b/pulsar-client-cpp/pkg/licenses/LICENSE-zlib.txt
index cf25d04b7e1..f1f93cd6a6a 100644
--- a/pulsar-client-cpp/pkg/licenses/LICENSE-zlib.txt
+++ b/pulsar-client-cpp/pkg/licenses/LICENSE-zlib.txt
@@ -1,7 +1,7 @@
zlib.h -- interface of the 'zlib' general purpose compression library
- version 1.2.11, January 15th, 2017
+ version 1.2.12, March 27th, 2022
- Copyright (C) 1995-2017 Jean-loup Gailly and Mark Adler
+ Copyright (C) 1995-2022 Jean-loup Gailly and Mark Adler
This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
diff --git a/pulsar-client-cpp/pkg/rpm/Dockerfile b/pulsar-client-cpp/pkg/rpm/Dockerfile
index c2406c5c7d3..e83290dccca 100644
--- a/pulsar-client-cpp/pkg/rpm/Dockerfile
+++ b/pulsar-client-cpp/pkg/rpm/Dockerfile
@@ -49,12 +49,12 @@ RUN curl -O -L https://github.com/google/protobuf/releases/download/v3.3.0/prot
rm -rf /protobuf-cpp-3.3.0.tar.gz /protobuf-3.3.0
# ZLib
-RUN curl -O -L https://github.com/madler/zlib/archive/v1.2.11.tar.gz && \
- tar xvfz v1.2.11.tar.gz && \
- cd zlib-1.2.11 && \
+RUN curl -O -L https://github.com/madler/zlib/archive/v1.2.12.tar.gz && \
+ tar xvfz v1.2.12.tar.gz && \
+ cd zlib-1.2.12 && \
CFLAGS="-fPIC -O3" ./configure && \
make && make install && \
- rm -rf /v1.2.11.tar.gz /zlib-1.2.11
+ rm -rf /v1.2.12.tar.gz /zlib-1.2.12
# Zstandard
RUN curl -O -L https://github.com/facebook/zstd/releases/download/v1.3.7/zstd-1.3.7.tar.gz && \
[pulsar] 05/31: [Flaky-test] BatchSourceExecutorTest.testLifeCycle (#10870) (#14717)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 82095a4129e82cae3f8d25b92d228743da32058a
Author: wuxuanqicn <89...@users.noreply.github.com>
AuthorDate: Fri Apr 22 01:46:22 2022 +0800
[Flaky-test] BatchSourceExecutorTest.testLifeCycle (#10870) (#14717)
event will be dropped at BatchSourceExecutor#triggerDiscover when discoverInProgress is true
https://github.com/apache/pulsar/blob/f0d166f36e1fbd4df1e20ae2ccc7fcae822c17b4/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/batch/BatchSourceExecutor.java#L161-L182
await util task submitted in BatchSourceExecutor#triggerDiscover execute completed and discoverInProgress update to false
Co-authored-by: xuanqi.wu <xu...@weimob.com>
(cherry picked from commit be13b2503a5bb47d672587c38408436c840f349e)
---
pulsar-functions/instance/pom.xml | 6 ++++++
.../functions/source/batch/BatchSourceExecutorTest.java | 11 +++++++++--
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/pulsar-functions/instance/pom.xml b/pulsar-functions/instance/pom.xml
index a48cdbc1d8d..2bab69025fb 100644
--- a/pulsar-functions/instance/pom.xml
+++ b/pulsar-functions/instance/pom.xml
@@ -194,6 +194,12 @@
<version>${prometheus-jmx.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.awaitility</groupId>
+ <artifactId>awaitility</artifactId>
+ <scope>test</scope>
+ </dependency>
+
</dependencies>
<build>
diff --git a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/batch/BatchSourceExecutorTest.java b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/batch/BatchSourceExecutorTest.java
index ff15a10ebbe..c98e88dea6e 100644
--- a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/batch/BatchSourceExecutorTest.java
+++ b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/batch/BatchSourceExecutorTest.java
@@ -18,7 +18,8 @@
*/
package org.apache.pulsar.functions.source.batch;
-
+import static org.awaitility.Awaitility.await;
+import static org.testng.Assert.fail;
import com.google.gson.Gson;
import lombok.Getter;
import org.apache.pulsar.client.api.ConsumerBuilder;
@@ -44,7 +45,6 @@ import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.function.Consumer;
-import static org.testng.Assert.fail;
/**
* Unit tests for {@link org.apache.pulsar.functions.source.batch.BatchSourceExecutor}
@@ -368,6 +368,8 @@ public class BatchSourceExecutorTest {
}
Assert.assertEquals(testBatchSource.getRecordCount(), 6);
Assert.assertEquals(testBatchSource.getDiscoverCount(), 1);
+
+ awaitDiscoverNotInProgress();
triggerQueue.put("trigger");
completedQueue.take();
Assert.assertTrue(testBatchSource.getDiscoverCount() == 2);
@@ -387,6 +389,8 @@ public class BatchSourceExecutorTest {
}
Assert.assertEquals(testBatchPushSource.getRecordCount(), 5);
Assert.assertEquals(testBatchPushSource.getDiscoverCount(), 1);
+
+ awaitDiscoverNotInProgress();
triggerQueue.put("trigger");
completedQueue.take();
Assert.assertEquals(testBatchPushSource.getDiscoverCount(), 2);
@@ -406,4 +410,7 @@ public class BatchSourceExecutorTest {
fail("should have thrown an exception");
}
+ private void awaitDiscoverNotInProgress() {
+ await().until(() -> !batchSourceExecutor.discoverInProgress);
+ }
}
\ No newline at end of file
[pulsar] 10/31: [Proxy/Client] Fix DNS server denial-of-service issue when DNS entry expires (#15403)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 352bb76dfabe04f4b75df31e4680e8af47e02e13
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Tue May 3 05:59:19 2022 +0300
[Proxy/Client] Fix DNS server denial-of-service issue when DNS entry expires (#15403)
(cherry picked from commit 40d71691dab2a09d3457f8fa638b19ebc2e28dd7)
---
.../pulsar/client/impl/ConnectionPoolTest.java | 38 ++++++++--------
.../apache/pulsar/client/impl/ConnectionPool.java | 53 +++++++++++-----------
.../pulsar/proxy/server/ProxyConnection.java | 11 +++--
.../apache/pulsar/proxy/server/ProxyService.java | 12 ++---
.../proxy/server/ServiceChannelInitializer.java | 2 +-
5 files changed, 60 insertions(+), 56 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ConnectionPoolTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ConnectionPoolTest.java
index 0193f592c75..9fbea7f2914 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ConnectionPoolTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ConnectionPoolTest.java
@@ -22,6 +22,10 @@ import static org.apache.pulsar.broker.BrokerTestUtil.spyWithClassAndConstructor
import com.google.common.collect.Lists;
import io.netty.channel.EventLoopGroup;
import io.netty.util.concurrent.DefaultThreadFactory;
+import java.net.InetSocketAddress;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.stream.IntStream;
import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
import org.apache.pulsar.client.impl.conf.ClientConfigurationData;
import org.apache.pulsar.common.util.netty.EventLoopUtil;
@@ -31,22 +35,18 @@ import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
-import java.net.InetAddress;
-import java.net.InetSocketAddress;
-import java.util.List;
-import java.util.concurrent.CompletableFuture;
-import java.util.stream.IntStream;
-
@Test(groups = "broker-impl")
public class ConnectionPoolTest extends MockedPulsarServiceBaseTest {
String serviceUrl;
+ int brokerPort;
@BeforeClass
@Override
protected void setup() throws Exception {
super.internalSetup();
- serviceUrl = "pulsar://non-existing-dns-name:" + pulsar.getBrokerListenPort().get();
+ brokerPort = pulsar.getBrokerListenPort().get();
+ serviceUrl = "pulsar://non-existing-dns-name:" + brokerPort;
}
@AfterClass(alwaysRun = true)
@@ -63,9 +63,11 @@ public class ConnectionPoolTest extends MockedPulsarServiceBaseTest {
conf.setServiceUrl(serviceUrl);
PulsarClientImpl client = new PulsarClientImpl(conf, eventLoop, pool);
- List<InetAddress> result = Lists.newArrayList();
- result.add(InetAddress.getByName("127.0.0.1"));
- Mockito.when(pool.resolveName("non-existing-dns-name")).thenReturn(CompletableFuture.completedFuture(result));
+ List<InetSocketAddress> result = Lists.newArrayList();
+ result.add(new InetSocketAddress("127.0.0.1", brokerPort));
+ Mockito.when(pool.resolveName(InetSocketAddress.createUnresolved("non-existing-dns-name",
+ brokerPort)))
+ .thenReturn(CompletableFuture.completedFuture(result));
client.newProducer().topic("persistent://sample/standalone/ns/my-topic").create();
@@ -75,20 +77,20 @@ public class ConnectionPoolTest extends MockedPulsarServiceBaseTest {
@Test
public void testDoubleIpAddress() throws Exception {
- String serviceUrl = "pulsar://non-existing-dns-name:" + pulsar.getBrokerListenPort().get();
-
ClientConfigurationData conf = new ClientConfigurationData();
EventLoopGroup eventLoop = EventLoopUtil.newEventLoopGroup(1, false, new DefaultThreadFactory("test"));
ConnectionPool pool = spyWithClassAndConstructorArgs(ConnectionPool.class, conf, eventLoop);
conf.setServiceUrl(serviceUrl);
PulsarClientImpl client = new PulsarClientImpl(conf, eventLoop, pool);
- List<InetAddress> result = Lists.newArrayList();
+ List<InetSocketAddress> result = Lists.newArrayList();
// Add a non existent IP to the response to check that we're trying the 2nd address as well
- result.add(InetAddress.getByName("127.0.0.99"));
- result.add(InetAddress.getByName("127.0.0.1"));
- Mockito.when(pool.resolveName("non-existing-dns-name")).thenReturn(CompletableFuture.completedFuture(result));
+ result.add(new InetSocketAddress("127.0.0.99", brokerPort));
+ result.add(new InetSocketAddress("127.0.0.1", brokerPort));
+ Mockito.when(pool.resolveName(InetSocketAddress.createUnresolved("non-existing-dns-name",
+ brokerPort)))
+ .thenReturn(CompletableFuture.completedFuture(result));
// Create producer should succeed by trying the 2nd IP
client.newProducer().topic("persistent://sample/standalone/ns/my-topic").create();
@@ -105,7 +107,7 @@ public class ConnectionPoolTest extends MockedPulsarServiceBaseTest {
ConnectionPool pool = spyWithClassAndConstructorArgs(ConnectionPool.class, conf, eventLoop);
InetSocketAddress brokerAddress =
- InetSocketAddress.createUnresolved("127.0.0.1", pulsar.getBrokerListenPort().get());
+ InetSocketAddress.createUnresolved("127.0.0.1", brokerPort);
IntStream.range(1, 5).forEach(i -> {
pool.getConnection(brokerAddress).thenAccept(cnx -> {
Assert.assertTrue(cnx.channel().isActive());
@@ -127,7 +129,7 @@ public class ConnectionPoolTest extends MockedPulsarServiceBaseTest {
ConnectionPool pool = spyWithClassAndConstructorArgs(ConnectionPool.class, conf, eventLoop);
InetSocketAddress brokerAddress =
- InetSocketAddress.createUnresolved("127.0.0.1", pulsar.getBrokerListenPort().get());
+ InetSocketAddress.createUnresolved("127.0.0.1", brokerPort);
IntStream.range(1, 10).forEach(i -> {
pool.getConnection(brokerAddress).thenAccept(cnx -> {
Assert.assertTrue(cnx.channel().isActive());
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java
index edb2a983f25..1e3331f66c2 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java
@@ -26,10 +26,10 @@ import io.netty.channel.Channel;
import io.netty.channel.ChannelException;
import io.netty.channel.ChannelOption;
import io.netty.channel.EventLoopGroup;
-import io.netty.resolver.dns.DnsNameResolver;
+import io.netty.resolver.AddressResolver;
+import io.netty.resolver.dns.DnsAddressResolverGroup;
import io.netty.resolver.dns.DnsNameResolverBuilder;
import io.netty.util.concurrent.Future;
-import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.URISyntaxException;
@@ -63,7 +63,7 @@ public class ConnectionPool implements AutoCloseable {
private final int maxConnectionsPerHosts;
private final boolean isSniProxy;
- protected final DnsNameResolver dnsResolver;
+ protected final AddressResolver<InetSocketAddress> addressResolver;
private final boolean shouldCloseDnsResolver;
public ConnectionPool(ClientConfigurationData conf, EventLoopGroup eventLoopGroup) throws PulsarClientException {
@@ -76,7 +76,8 @@ public class ConnectionPool implements AutoCloseable {
}
public ConnectionPool(ClientConfigurationData conf, EventLoopGroup eventLoopGroup,
- Supplier<ClientCnx> clientCnxSupplier, Optional<DnsNameResolver> dnsNameResolver)
+ Supplier<ClientCnx> clientCnxSupplier,
+ Optional<AddressResolver<InetSocketAddress>> addressResolver)
throws PulsarClientException {
this.eventLoopGroup = eventLoopGroup;
this.clientConfig = conf;
@@ -101,12 +102,13 @@ public class ConnectionPool implements AutoCloseable {
throw new PulsarClientException(e);
}
- this.shouldCloseDnsResolver = !dnsNameResolver.isPresent();
- this.dnsResolver = dnsNameResolver.orElseGet(() -> createDnsNameResolver(conf, eventLoopGroup));
+ this.shouldCloseDnsResolver = !addressResolver.isPresent();
+ this.addressResolver = addressResolver.orElseGet(() -> createAddressResolver(conf, eventLoopGroup));
}
- private static DnsNameResolver createDnsNameResolver(ClientConfigurationData conf, EventLoopGroup eventLoopGroup) {
- DnsNameResolverBuilder dnsNameResolverBuilder = new DnsNameResolverBuilder(eventLoopGroup.next())
+ private static AddressResolver<InetSocketAddress> createAddressResolver(ClientConfigurationData conf,
+ EventLoopGroup eventLoopGroup) {
+ DnsNameResolverBuilder dnsNameResolverBuilder = new DnsNameResolverBuilder()
.traceEnabled(true).channelType(EventLoopUtil.getDatagramChannelClass(eventLoopGroup));
if (conf.getDnsLookupBindAddress() != null) {
InetSocketAddress addr = new InetSocketAddress(conf.getDnsLookupBindAddress(),
@@ -114,7 +116,10 @@ public class ConnectionPool implements AutoCloseable {
dnsNameResolverBuilder.localAddress(addr);
}
DnsResolverUtil.applyJdkDnsCacheSettings(dnsNameResolverBuilder);
- return dnsNameResolverBuilder.build();
+ // use DnsAddressResolverGroup to create the AddressResolver since it contains a solution
+ // to prevent cache stampede / thundering herds problem when a DNS entry expires while the system
+ // is under high load
+ return new DnsAddressResolverGroup(dnsNameResolverBuilder).getResolver(eventLoopGroup.next());
}
private static final Random random = new Random();
@@ -239,19 +244,17 @@ public class ConnectionPool implements AutoCloseable {
* Resolve DNS asynchronously and attempt to connect to any IP address returned by DNS server.
*/
private CompletableFuture<Channel> createConnection(InetSocketAddress unresolvedAddress) {
- int port;
- CompletableFuture<List<InetAddress>> resolvedAddress;
+ CompletableFuture<List<InetSocketAddress>> resolvedAddress;
try {
if (isSniProxy) {
URI proxyURI = new URI(clientConfig.getProxyServiceUrl());
- port = proxyURI.getPort();
- resolvedAddress = resolveName(proxyURI.getHost());
+ resolvedAddress =
+ resolveName(InetSocketAddress.createUnresolved(proxyURI.getHost(), proxyURI.getPort()));
} else {
- port = unresolvedAddress.getPort();
- resolvedAddress = resolveName(unresolvedAddress.getHostString());
+ resolvedAddress = resolveName(unresolvedAddress);
}
return resolvedAddress.thenCompose(
- inetAddresses -> connectToResolvedAddresses(inetAddresses.iterator(), port,
+ inetAddresses -> connectToResolvedAddresses(inetAddresses.iterator(),
isSniProxy ? unresolvedAddress : null));
} catch (URISyntaxException e) {
log.error("Invalid Proxy url {}", clientConfig.getProxyServiceUrl(), e);
@@ -264,18 +267,17 @@ public class ConnectionPool implements AutoCloseable {
* Try to connect to a sequence of IP addresses until a successful connection can be made, or fail if no
* address is working.
*/
- private CompletableFuture<Channel> connectToResolvedAddresses(Iterator<InetAddress> unresolvedAddresses,
- int port,
+ private CompletableFuture<Channel> connectToResolvedAddresses(Iterator<InetSocketAddress> unresolvedAddresses,
InetSocketAddress sniHost) {
CompletableFuture<Channel> future = new CompletableFuture<>();
// Successfully connected to server
- connectToAddress(unresolvedAddresses.next(), port, sniHost)
+ connectToAddress(unresolvedAddresses.next(), sniHost)
.thenAccept(future::complete)
.exceptionally(exception -> {
if (unresolvedAddresses.hasNext()) {
// Try next IP address
- connectToResolvedAddresses(unresolvedAddresses, port, sniHost).thenAccept(future::complete)
+ connectToResolvedAddresses(unresolvedAddresses, sniHost).thenAccept(future::complete)
.exceptionally(ex -> {
// This is already unwinding the recursive call
future.completeExceptionally(ex);
@@ -291,9 +293,9 @@ public class ConnectionPool implements AutoCloseable {
return future;
}
- CompletableFuture<List<InetAddress>> resolveName(String hostname) {
- CompletableFuture<List<InetAddress>> future = new CompletableFuture<>();
- dnsResolver.resolveAll(hostname).addListener((Future<List<InetAddress>> resolveFuture) -> {
+ CompletableFuture<List<InetSocketAddress>> resolveName(InetSocketAddress unresolvedAddress) {
+ CompletableFuture<List<InetSocketAddress>> future = new CompletableFuture<>();
+ addressResolver.resolveAll(unresolvedAddress).addListener((Future<List<InetSocketAddress>> resolveFuture) -> {
if (resolveFuture.isSuccess()) {
future.complete(resolveFuture.get());
} else {
@@ -306,8 +308,7 @@ public class ConnectionPool implements AutoCloseable {
/**
* Attempt to establish a TCP connection to an already resolved single IP address.
*/
- private CompletableFuture<Channel> connectToAddress(InetAddress ipAddress, int port, InetSocketAddress sniHost) {
- InetSocketAddress remoteAddress = new InetSocketAddress(ipAddress, port);
+ private CompletableFuture<Channel> connectToAddress(InetSocketAddress remoteAddress, InetSocketAddress sniHost) {
if (clientConfig.isUseTls()) {
return toCompletableFuture(bootstrap.register())
.thenCompose(channel -> channelInitializerHandler
@@ -337,7 +338,7 @@ public class ConnectionPool implements AutoCloseable {
public void close() throws Exception {
closeAllConnections();
if (shouldCloseDnsResolver) {
- dnsResolver.close();
+ addressResolver.close();
}
}
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConnection.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConnection.java
index 33dc1abd88a..0d80c2e473a 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConnection.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConnection.java
@@ -26,7 +26,7 @@ import io.netty.channel.ChannelHandler;
import io.netty.channel.ChannelHandlerContext;
import io.netty.handler.codec.haproxy.HAProxyMessage;
import io.netty.handler.ssl.SslHandler;
-import io.netty.resolver.dns.DnsNameResolver;
+import io.netty.resolver.dns.DnsAddressResolverGroup;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.util.Collections;
@@ -77,7 +77,7 @@ public class ProxyConnection extends PulsarHandler {
private final AtomicLong requestIdGenerator =
new AtomicLong(ThreadLocalRandom.current().nextLong(0, Long.MAX_VALUE / 2));
private final ProxyService service;
- private final DnsNameResolver dnsNameResolver;
+ private final DnsAddressResolverGroup dnsAddressResolverGroup;
AuthenticationDataSource authenticationData;
private State state;
private final Supplier<SslHandler> sslHandlerSupplier;
@@ -130,10 +130,10 @@ public class ProxyConnection extends PulsarHandler {
}
public ProxyConnection(ProxyService proxyService, Supplier<SslHandler> sslHandlerSupplier,
- DnsNameResolver dnsNameResolver) {
+ DnsAddressResolverGroup dnsAddressResolverGroup) {
super(30, TimeUnit.SECONDS);
this.service = proxyService;
- this.dnsNameResolver = dnsNameResolver;
+ this.dnsAddressResolverGroup = dnsAddressResolverGroup;
this.state = State.Init;
this.sslHandlerSupplier = sslHandlerSupplier;
this.brokerProxyValidator = service.getBrokerProxyValidator();
@@ -276,7 +276,8 @@ public class ProxyConnection extends PulsarHandler {
if (this.connectionPool == null) {
this.connectionPool = new ConnectionPool(clientConf, service.getWorkerGroup(),
- clientCnxSupplier, Optional.of(dnsNameResolver));
+ clientCnxSupplier,
+ Optional.of(dnsAddressResolverGroup.getResolver(service.getWorkerGroup().next())));
} else {
LOG.error("BUG! Connection Pool has already been created for proxy connection to {} state {} role {}",
remoteAddress, state, clientAuthRole);
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyService.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyService.java
index 10b99aeaff1..1960b5143a0 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyService.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyService.java
@@ -28,7 +28,7 @@ import io.netty.channel.ChannelInitializer;
import io.netty.channel.ChannelOption;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.socket.SocketChannel;
-import io.netty.resolver.dns.DnsNameResolver;
+import io.netty.resolver.dns.DnsAddressResolverGroup;
import io.netty.resolver.dns.DnsNameResolverBuilder;
import io.netty.util.concurrent.DefaultThreadFactory;
import io.prometheus.client.Counter;
@@ -80,7 +80,7 @@ public class ProxyService implements Closeable {
private final ProxyConfiguration proxyConfig;
private final Authentication proxyClientAuthentication;
@Getter
- private final DnsNameResolver dnsNameResolver;
+ private final DnsAddressResolverGroup dnsAddressResolverGroup;
@Getter
private final BrokerProxyValidator brokerProxyValidator;
private String serviceUrl;
@@ -162,13 +162,13 @@ public class ProxyService implements Closeable {
false, workersThreadFactory);
this.authenticationService = authenticationService;
- DnsNameResolverBuilder dnsNameResolverBuilder = new DnsNameResolverBuilder(workerGroup.next())
+ DnsNameResolverBuilder dnsNameResolverBuilder = new DnsNameResolverBuilder()
.channelType(EventLoopUtil.getDatagramChannelClass(workerGroup));
DnsResolverUtil.applyJdkDnsCacheSettings(dnsNameResolverBuilder);
- dnsNameResolver = dnsNameResolverBuilder.build();
+ dnsAddressResolverGroup = new DnsAddressResolverGroup(dnsNameResolverBuilder);
- brokerProxyValidator = new BrokerProxyValidator(dnsNameResolver.asAddressResolver(),
+ brokerProxyValidator = new BrokerProxyValidator(dnsAddressResolverGroup.getResolver(workerGroup.next()),
proxyConfig.getBrokerProxyAllowedHostNames(),
proxyConfig.getBrokerProxyAllowedIPAddresses(),
proxyConfig.getBrokerProxyAllowedTargetPorts());
@@ -331,7 +331,7 @@ public class ProxyService implements Closeable {
}
public void close() throws IOException {
- dnsNameResolver.close();
+ dnsAddressResolverGroup.close();
if (discoveryProvider != null) {
discoveryProvider.close();
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java
index 1a588b481fc..f1fd98bd8f6 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java
@@ -173,7 +173,7 @@ public class ServiceChannelInitializer extends ChannelInitializer<SocketChannel>
}
ch.pipeline().addLast("handler",
- new ProxyConnection(proxyService, sslHandlerSupplier, proxyService.getDnsNameResolver()));
+ new ProxyConnection(proxyService, sslHandlerSupplier, proxyService.getDnsAddressResolverGroup()));
}
}
[pulsar] 20/31: Fix grant all permissions but can't list topic. (#15501)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 3dcdf2a69932b166975e4d0ba2eab7c65c34e14b
Author: Jiwei Guo <te...@apache.org>
AuthorDate: Mon May 9 22:05:07 2022 +0800
Fix grant all permissions but can't list topic. (#15501)
(cherry picked from commit 5155b1df876bd98d173e87753cca642b82b6595a)
---
.../authorization/PulsarAuthorizationProvider.java | 2 +-
.../apache/pulsar/broker/auth/AuthorizationTest.java | 18 +++---------------
2 files changed, 4 insertions(+), 16 deletions(-)
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java
index 1ad8fbe3e09..a01a2a735f0 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java
@@ -584,6 +584,7 @@ public class PulsarAuthorizationProvider implements AuthorizationProvider {
namespaceName, role, authData, AuthAction.packages);
case GET_TOPIC:
case GET_TOPICS:
+ case GET_BUNDLE:
return allowConsumeOrProduceOpsAsync(namespaceName, role, authData);
case UNSUBSCRIBE:
case CLEAR_BACKLOG:
@@ -592,7 +593,6 @@ public class PulsarAuthorizationProvider implements AuthorizationProvider {
case CREATE_TOPIC:
case DELETE_TOPIC:
case ADD_BUNDLE:
- case GET_BUNDLE:
case DELETE_BUNDLE:
case GRANT_PERMISSION:
case GET_PERMISSION:
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/AuthorizationTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/AuthorizationTest.java
index 574e7a14c43..39a91f72dc7 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/AuthorizationTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/AuthorizationTest.java
@@ -19,7 +19,6 @@
package org.apache.pulsar.broker.auth;
import static org.mockito.Mockito.when;
-import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
@@ -27,7 +26,6 @@ import java.util.EnumSet;
import org.apache.pulsar.broker.authorization.AuthorizationService;
import org.apache.pulsar.client.admin.PulsarAdmin;
import org.apache.pulsar.client.admin.PulsarAdminBuilder;
-import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.common.naming.TopicDomain;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.policies.data.AuthAction;
@@ -232,7 +230,7 @@ public class AuthorizationTest extends MockedPulsarServiceBaseTest {
}
@Test
- public void testGetListWithoutGetBundleOp() throws Exception {
+ public void testGetListWithGetBundleOp() throws Exception {
String tenant = "p1";
String namespaceV1 = "p1/global/ns1";
String namespaceV2 = "p1/ns2";
@@ -248,18 +246,8 @@ public class AuthorizationTest extends MockedPulsarServiceBaseTest {
.authentication(new MockAuthentication("pass.pass2"))
.build();
when(pulsar.getAdminClient()).thenReturn(admin2);
- try {
- admin2.topics().getList(namespaceV1, TopicDomain.non_persistent);
- } catch (Exception ex) {
- assertTrue(ex instanceof PulsarAdminException.NotAuthorizedException);
- assertEquals(ex.getMessage(), "Unauthorized to validateNamespaceOperation for operation [GET_BUNDLE] on namespace [p1/global/ns1]");
- }
- try {
- admin2.topics().getList(namespaceV2, TopicDomain.non_persistent);
- } catch (Exception ex) {
- assertTrue(ex instanceof PulsarAdminException.NotAuthorizedException);
- assertEquals(ex.getMessage(), "Unauthorized to validateNamespaceOperation for operation [GET_BUNDLE] on namespace [p1/ns2]");
- }
+ Assert.assertEquals(admin2.topics().getList(namespaceV1, TopicDomain.non_persistent).size(), 0);
+ Assert.assertEquals(admin2.topics().getList(namespaceV2, TopicDomain.non_persistent).size(), 0);
}
private static void waitForChange() {
[pulsar] 08/31: [improve][broker-web&websocket&proxy&function-worker] Full-support set ssl provider, ciphers and protocols (#13740)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit fb0cb768728db747dce925536729f479c966c9e9
Author: Zixuan Liu <no...@gmail.com>
AuthorDate: Sun May 1 12:35:46 2022 +0800
[improve][broker-web&websocket&proxy&function-worker] Full-support set ssl provider, ciphers and protocols (#13740)
Fixes #13734
Pulsar doesn't set ssl provider, ciphers and protocols to the web, websocket and proxy service when `tlsEnabledWithKeyStore=false`
- Add `org.apache.pulsar.jetty.tls` package in pulsar-broker-common for Jetty TLS support
- Add a new `webServiceTlsProvider=Conscrypt` to broker and proxy config
- Update `Conscrypt` as the `tlsProvider` value in websocket config
In the old version, we implicitly use the `Conscrypt` provider, now we need to set it explicitly.
(cherry picked from commit bf15e83a3f19b698c1492071792045e47824675c)
---
conf/broker.conf | 3 +
conf/functions_worker.yml | 6 +-
conf/proxy.conf | 9 +-
conf/standalone.conf | 3 +
conf/websocket.conf | 8 +-
.../apache/pulsar/broker/ServiceConfiguration.java | 6 +
.../java/org/apache/pulsar/jetty/package-info.java | 19 +++
.../pulsar/jetty/tls/JettySslContextFactory.java | 116 ++++++++++++++++++
.../org/apache/pulsar/jetty/tls/package-info.java | 19 +++
.../jetty/tls/JettySslContextFactoryTest.java | 130 +++++++++------------
.../JettySslContextFactoryWithKeyStoreTest.java | 47 ++++----
.../src/test/resources/ssl/jetty_client_key.jks | Bin 0 -> 2679 bytes
.../src/test/resources/ssl/jetty_client_trust.jks | Bin 0 -> 1207 bytes
.../src/test/resources/ssl/jetty_server_key.jks | Bin 0 -> 2679 bytes
.../src/test/resources/ssl/jetty_server_trust.jks | Bin 0 -> 1207 bytes
.../src/test/resources/ssl/my-ca/ca.pem | 18 +++
.../src/test/resources/ssl/my-ca/client-ca.pem | 19 +++
.../src/test/resources/ssl/my-ca/client-key.pem | 28 +++++
.../src/test/resources/ssl/my-ca/server-ca.pem | 19 +++
.../src/test/resources/ssl/my-ca/server-key.pem | 28 +++++
.../org/apache/pulsar/broker/PulsarService.java | 2 +
.../org/apache/pulsar/broker/web/WebService.java | 14 ++-
.../proxy/ProxyPublishConsumeTlsTest.java | 2 +-
pulsar-common/pom.xml | 11 --
.../common/util/DefaultSslContextBuilder.java | 23 +++-
.../apache/pulsar/common/util/SecurityUtility.java | 74 ++++--------
.../JettySslContextFactoryWithAutoRefresh.java | 70 -----------
.../util/keystoretls/KeyStoreSSLContext.java | 58 +--------
.../pulsar/functions/worker/rest/WorkerServer.java | 14 ++-
.../pulsar/proxy/server/AdminProxyHandler.java | 14 ++-
.../pulsar/proxy/server/ProxyConfiguration.java | 6 +
.../org/apache/pulsar/proxy/server/WebServer.java | 13 ++-
.../pulsar/websocket/service/ProxyServer.java | 11 +-
.../service/WebSocketProxyConfiguration.java | 6 +-
34 files changed, 465 insertions(+), 331 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index 5c1ca1d8b75..cd6ccb42e6c 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -623,6 +623,9 @@ tlsRequireTrustedClientCertOnConnect=false
# When using TLS authentication with KeyStore, available values can be SunJSSE, Conscrypt and etc.
tlsProvider=
+# Specify the TLS provider for the web service: SunJSSE, Conscrypt and etc.
+webServiceTlsProvider=Conscrypt
+
### --- KeyStore TLS config variables --- ###
## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
diff --git a/conf/functions_worker.yml b/conf/functions_worker.yml
index 4254d3fa8cb..f12832595c5 100644
--- a/conf/functions_worker.yml
+++ b/conf/functions_worker.yml
@@ -321,11 +321,11 @@ tlsCertRefreshCheckDurationSec: 300
# certificate isn't trusted.
tlsRequireTrustedClientCertOnConnect: false
-### --- KeyStore TLS config variables --- ###
+### --- TLS config variables --- ###
## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
-# TLS Provider for KeyStore type
-tlsProvider:
+# Specify the TLS provider for the web service: SunJSSE, Conscrypt and etc.
+tlsProvider: Conscrypt
# Enable TLS with KeyStore type configuration in function worker.
tlsEnabledWithKeyStore: false
diff --git a/conf/proxy.conf b/conf/proxy.conf
index 5c12a645054..cdaf0ff732f 100644
--- a/conf/proxy.conf
+++ b/conf/proxy.conf
@@ -71,12 +71,17 @@ webServicePort=8080
# Port to use to server HTTPS request
webServicePortTls=
-### --- KeyStore TLS config variables --- ###
+### --- TLS config variables --- ###
## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
-# TLS Provider for KeyStore type
+# Specify the TLS provider for the broker service:
+# When using TLS authentication with CACert, the valid value is either OPENSSL or JDK.
+# When using TLS authentication with KeyStore, available values can be SunJSSE, Conscrypt and etc.
tlsProvider=
+# Specify the TLS provider for the web service, available values can be SunJSSE, Conscrypt and etc.
+webServiceTlsProvider=Conscrypt
+
# Enable TLS with KeyStore type configuration in proxy.
tlsEnabledWithKeyStore=false
diff --git a/conf/standalone.conf b/conf/standalone.conf
index b107625bcc6..f3a88879f40 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -380,6 +380,9 @@ tlsRequireTrustedClientCertOnConnect=false
# When using TLS authentication with KeyStore, available values can be SunJSSE, Conscrypt and etc.
tlsProvider=
+# Specify the TLS provider for the web service: SunJSSE, Conscrypt and etc.
+webServiceTlsProvider=Conscrypt
+
### --- KeyStore TLS config variables --- ###
# Enable TLS with KeyStore type configuration in broker.
tlsEnabledWithKeyStore=false
diff --git a/conf/websocket.conf b/conf/websocket.conf
index 1c946ef4dba..797ea357176 100644
--- a/conf/websocket.conf
+++ b/conf/websocket.conf
@@ -97,6 +97,7 @@ brokerClientTrustCertsFilePath=
anonymousUserRole=
### --- TLS --- ###
+## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
# Deprecated - use webServicePortTls and brokerClientTlsEnabled instead
tlsEnabled=false
@@ -120,11 +121,8 @@ tlsRequireTrustedClientCertOnConnect=false
# Tls cert refresh duration in seconds (set 0 to check on every new connection)
tlsCertRefreshCheckDurationSec=300
-### --- KeyStore TLS config variables --- ###
-## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
-
-# TLS Provider for KeyStore type
-tlsProvider=
+# Specify the TLS provider for the WebSocket: SunJSSE, Conscrypt and etc.
+tlsProvider=Conscrypt
# Enable TLS with KeyStore type configuration in WebSocket.
tlsEnabledWithKeyStore=false
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 16bde537510..c7a5d440b8b 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -180,6 +180,12 @@ public class ServiceConfiguration implements PulsarConfiguration {
)
private Optional<Integer> webServicePortTls = Optional.empty();
+ @FieldContext(
+ category = CATEGORY_SERVER,
+ doc = "Specify the TLS provider for the web service: SunJSSE, Conscrypt and etc."
+ )
+ private String webServiceTlsProvider = "Conscrypt";
+
@FieldContext(
category = CATEGORY_TLS,
doc = "Specify the tls protocols the proxy's web service will use to negotiate during TLS Handshake.\n\n"
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/package-info.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/package-info.java
new file mode 100644
index 00000000000..f01bd9198f5
--- /dev/null
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/package-info.java
@@ -0,0 +1,19 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.jetty;
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/JettySslContextFactory.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/JettySslContextFactory.java
new file mode 100644
index 00000000000..514fa4a0725
--- /dev/null
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/JettySslContextFactory.java
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.jetty.tls;
+
+import java.util.Set;
+import javax.net.ssl.SSLContext;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.util.DefaultSslContextBuilder;
+import org.apache.pulsar.common.util.SecurityUtility;
+import org.apache.pulsar.common.util.SslContextAutoRefreshBuilder;
+import org.apache.pulsar.common.util.keystoretls.NetSslContextBuilder;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+
+@Slf4j
+public class JettySslContextFactory {
+ static {
+ // DO NOT EDIT - Load Conscrypt provider
+ if (SecurityUtility.CONSCRYPT_PROVIDER != null) {
+ }
+ }
+
+ public static SslContextFactory.Server createServerSslContextWithKeystore(String sslProviderString,
+ String keyStoreTypeString,
+ String keyStore,
+ String keyStorePassword,
+ boolean allowInsecureConnection,
+ String trustStoreTypeString,
+ String trustStore,
+ String trustStorePassword,
+ boolean requireTrustedClientCertOnConnect,
+ Set<String> ciphers,
+ Set<String> protocols,
+ long certRefreshInSec) {
+ NetSslContextBuilder sslCtxRefresher = new NetSslContextBuilder(
+ sslProviderString,
+ keyStoreTypeString,
+ keyStore,
+ keyStorePassword,
+ allowInsecureConnection,
+ trustStoreTypeString,
+ trustStore,
+ trustStorePassword,
+ requireTrustedClientCertOnConnect,
+ certRefreshInSec);
+
+ return new JettySslContextFactory.Server(sslProviderString, sslCtxRefresher,
+ requireTrustedClientCertOnConnect, ciphers, protocols);
+ }
+
+ public static SslContextFactory createServerSslContext(String sslProviderString, boolean tlsAllowInsecureConnection,
+ String tlsTrustCertsFilePath,
+ String tlsCertificateFilePath,
+ String tlsKeyFilePath,
+ boolean tlsRequireTrustedClientCertOnConnect,
+ Set<String> ciphers,
+ Set<String> protocols,
+ long certRefreshInSec) {
+ DefaultSslContextBuilder sslCtxRefresher =
+ new DefaultSslContextBuilder(tlsAllowInsecureConnection, tlsTrustCertsFilePath, tlsCertificateFilePath,
+ tlsKeyFilePath, tlsRequireTrustedClientCertOnConnect, certRefreshInSec, sslProviderString);
+
+ return new JettySslContextFactory.Server(sslProviderString, sslCtxRefresher,
+ tlsRequireTrustedClientCertOnConnect, ciphers, protocols);
+ }
+
+ private static class Server extends SslContextFactory.Server {
+ private final SslContextAutoRefreshBuilder<SSLContext> sslCtxRefresher;
+
+ public Server(String sslProviderString, SslContextAutoRefreshBuilder<SSLContext> sslCtxRefresher,
+ boolean requireTrustedClientCertOnConnect, Set<String> ciphers, Set<String> protocols) {
+ super();
+ this.sslCtxRefresher = sslCtxRefresher;
+
+ if (ciphers != null && ciphers.size() > 0) {
+ this.setIncludeCipherSuites(ciphers.toArray(new String[0]));
+ }
+
+ if (protocols != null && protocols.size() > 0) {
+ this.setIncludeProtocols(protocols.toArray(new String[0]));
+ }
+
+ if (sslProviderString != null && !sslProviderString.equals("")) {
+ setProvider(sslProviderString);
+ }
+
+ if (requireTrustedClientCertOnConnect) {
+ this.setNeedClientAuth(true);
+ this.setTrustAll(false);
+ } else {
+ this.setWantClientAuth(true);
+ this.setTrustAll(true);
+ }
+ }
+
+ @Override
+ public SSLContext getSslContext() {
+ return sslCtxRefresher.get();
+ }
+ }
+}
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/package-info.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/package-info.java
new file mode 100644
index 00000000000..8978699ff79
--- /dev/null
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/jetty/tls/package-info.java
@@ -0,0 +1,19 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.jetty.tls;
diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java b/pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryTest.java
similarity index 60%
copy from pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java
copy to pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryTest.java
index 414fbb2efd0..c1816674880 100644
--- a/pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java
+++ b/pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryTest.java
@@ -16,9 +16,17 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pulsar.common.util.keystoretls;
+
+package org.apache.pulsar.jetty.tls;
import com.google.common.io.Resources;
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLHandshakeException;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.config.RegistryBuilder;
@@ -29,40 +37,30 @@ import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
-import org.apache.logging.log4j.Level;
-import org.apache.logging.log4j.core.config.Configurator;
+import org.apache.pulsar.common.util.SecurityUtility;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.util.ssl.SslContextFactory;
import org.testng.annotations.Test;
-import javax.net.ssl.KeyManager;
-import javax.net.ssl.KeyManagerFactory;
-import javax.net.ssl.SSLContext;
-import javax.net.ssl.SSLHandshakeException;
-import javax.net.ssl.TrustManagerFactory;
-import java.io.FileInputStream;
-import java.security.KeyStore;
-import java.security.SecureRandom;
-import java.util.ArrayList;
-import java.util.HashSet;
-import java.util.List;
-
@Slf4j
-public class JettySslContextFactoryWithAutoRefreshTest {
+public class JettySslContextFactoryTest {
@Test
public void testJettyTlsServerTls() throws Exception {
- Configurator.setRootLevel(Level.INFO);
Server server = new Server();
List<ServerConnector> connectors = new ArrayList<>();
- SslContextFactory.Server factory = KeyStoreSSLContext.createSslContextFactory(null,
- "JKS", Resources.getResource("ssl/jetty_server_key.jks").getPath(),
- "jetty_server_pwd", false, "JKS",
- Resources.getResource("ssl/jetty_server_trust.jks").getPath(),
- "jetty_server_pwd", true, null,
- null, 600);
- factory.setHostnameVerifier((s, sslSession) -> true);
+ SslContextFactory factory = JettySslContextFactory.createServerSslContext(
+ null,
+ false,
+ Resources.getResource("ssl/my-ca/ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/server-ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/server-key.pem").getPath(),
+ true,
+ null,
+ null,
+ 600);
+
ServerConnector connector = new ServerConnector(server, factory);
connector.setPort(0);
connectors.add(connector);
@@ -71,7 +69,8 @@ public class JettySslContextFactoryWithAutoRefreshTest {
// client connect
HttpClientBuilder httpClientBuilder = HttpClients.custom();
RegistryBuilder<ConnectionSocketFactory> registryBuilder = RegistryBuilder.create();
- registryBuilder.register("https", new SSLConnectionSocketFactory(getClientSslContext(), new NoopHostnameVerifier()));
+ registryBuilder.register("https",
+ new SSLConnectionSocketFactory(getClientSslContext(), new NoopHostnameVerifier()));
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(registryBuilder.build());
httpClientBuilder.setConnectionManager(cm);
CloseableHttpClient httpClient = httpClientBuilder.build();
@@ -83,19 +82,22 @@ public class JettySslContextFactoryWithAutoRefreshTest {
@Test(expectedExceptions = SSLHandshakeException.class)
public void testJettyTlsServerInvalidTlsProtocol() throws Exception {
- Configurator.setRootLevel(Level.INFO);
Server server = new Server();
List<ServerConnector> connectors = new ArrayList<>();
- SslContextFactory.Server factory = KeyStoreSSLContext.createSslContextFactory(null,
- "JKS", Resources.getResource("ssl/jetty_server_key.jks").getPath(),
- "jetty_server_pwd", false, "JKS",
- Resources.getResource("ssl/jetty_server_trust.jks").getPath(),
- "jetty_server_pwd", true, null,
+ SslContextFactory factory = JettySslContextFactory.createServerSslContext(
+ null,
+ false,
+ Resources.getResource("ssl/my-ca/ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/server-ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/server-key.pem").getPath(),
+ true,
+ null,
new HashSet<String>() {
{
this.add("TLSv1.3");
}
- }, 600);
+ },
+ 600);
factory.setHostnameVerifier((s, sslSession) -> true);
ServerConnector connector = new ServerConnector(server, factory);
connector.setPort(0);
@@ -118,14 +120,16 @@ public class JettySslContextFactoryWithAutoRefreshTest {
@Test(expectedExceptions = SSLHandshakeException.class)
public void testJettyTlsServerInvalidCipher() throws Exception {
- Configurator.setRootLevel(Level.INFO);
Server server = new Server();
List<ServerConnector> connectors = new ArrayList<>();
- SslContextFactory.Server factory = KeyStoreSSLContext.createSslContextFactory(null,
- "JKS", Resources.getResource("ssl/jetty_server_key.jks").getPath(),
- "jetty_server_pwd", false, "JKS",
- Resources.getResource("ssl/jetty_server_trust.jks").getPath(),
- "jetty_server_pwd", true, new HashSet<String>() {
+ SslContextFactory factory = JettySslContextFactory.createServerSslContext(
+ null,
+ false,
+ Resources.getResource("ssl/my-ca/ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/server-ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/server-key.pem").getPath(),
+ true,
+ new HashSet<String>() {
{
this.add("TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256");
}
@@ -134,7 +138,9 @@ public class JettySslContextFactoryWithAutoRefreshTest {
{
this.add("TLSv1.2");
}
- }, 600);
+ },
+ 600);
+
factory.setHostnameVerifier((s, sslSession) -> true);
ServerConnector connector = new ServerConnector(server, factory);
connector.setPort(0);
@@ -145,7 +151,8 @@ public class JettySslContextFactoryWithAutoRefreshTest {
HttpClientBuilder httpClientBuilder = HttpClients.custom();
RegistryBuilder<ConnectionSocketFactory> registryBuilder = RegistryBuilder.create();
registryBuilder.register("https", new SSLConnectionSocketFactory(getClientSslContext(),
- new String[]{"TLSv1.2"}, new String[]{"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"}, new NoopHostnameVerifier()));
+ new String[]{"TLSv1.2"}, new String[]{"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"},
+ new NoopHostnameVerifier()));
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(registryBuilder.build());
httpClientBuilder.setConnectionManager(cm);
CloseableHttpClient httpClient = httpClientBuilder.build();
@@ -155,38 +162,13 @@ public class JettySslContextFactoryWithAutoRefreshTest {
server.stop();
}
- private static SSLContext getClientSslContext() {
- return getSslContext(Resources.getResource("ssl/jetty_client_key.jks").getPath(),
- "jetty_client_pwd",
- Resources.getResource("ssl/jetty_client_trust.jks").getPath(),
- "jetty_client_pwd");
+ private static SSLContext getClientSslContext() throws GeneralSecurityException, IOException {
+ return SecurityUtility.createSslContext(
+ false,
+ Resources.getResource("ssl/my-ca/ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/client-ca.pem").getPath(),
+ Resources.getResource("ssl/my-ca/client-key.pem").getPath(),
+ null
+ );
}
-
- private static SSLContext getSslContext(String keyStorePath, String keyStorePassword,
- String trustStorePath, String trustStorePassword) {
- try {
- SSLContext sslContext = SSLContext.getInstance("TLS");
- // key store
- KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
- KeyStore keyStore = KeyStore.getInstance("JKS");
- try (FileInputStream inputStream = new FileInputStream(keyStorePath)) {
- keyStore.load(inputStream, keyStorePassword.toCharArray());
- }
- keyManagerFactory.init(keyStore, keyStorePassword.toCharArray());
- KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();
- // trust store
- TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
- KeyStore trustStore = KeyStore.getInstance("JKS");
- try (FileInputStream inputStream = new FileInputStream(trustStorePath)) {
- trustStore.load(inputStream, trustStorePassword.toCharArray());
- }
- trustManagerFactory.init(trustStore);
- sslContext.init(keyManagers, trustManagerFactory.getTrustManagers(), new SecureRandom());
- return sslContext;
- } catch (Exception e) {
- log.error("load ssl context error ", e);
- return null;
- }
- }
-
-}
\ No newline at end of file
+}
diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java b/pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryWithKeyStoreTest.java
similarity index 90%
rename from pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java
rename to pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryWithKeyStoreTest.java
index 414fbb2efd0..292bd123fdf 100644
--- a/pulsar-common/src/test/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefreshTest.java
+++ b/pulsar-broker-common/src/test/java/org/apache/pulsar/jetty/tls/JettySslContextFactoryWithKeyStoreTest.java
@@ -16,9 +16,20 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pulsar.common.util.keystoretls;
+package org.apache.pulsar.jetty.tls;
import com.google.common.io.Resources;
+import java.io.FileInputStream;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import javax.net.ssl.KeyManager;
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLHandshakeException;
+import javax.net.ssl.TrustManagerFactory;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.config.RegistryBuilder;
@@ -36,27 +47,14 @@ import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.util.ssl.SslContextFactory;
import org.testng.annotations.Test;
-import javax.net.ssl.KeyManager;
-import javax.net.ssl.KeyManagerFactory;
-import javax.net.ssl.SSLContext;
-import javax.net.ssl.SSLHandshakeException;
-import javax.net.ssl.TrustManagerFactory;
-import java.io.FileInputStream;
-import java.security.KeyStore;
-import java.security.SecureRandom;
-import java.util.ArrayList;
-import java.util.HashSet;
-import java.util.List;
-
@Slf4j
-public class JettySslContextFactoryWithAutoRefreshTest {
+public class JettySslContextFactoryWithKeyStoreTest {
@Test
public void testJettyTlsServerTls() throws Exception {
- Configurator.setRootLevel(Level.INFO);
Server server = new Server();
List<ServerConnector> connectors = new ArrayList<>();
- SslContextFactory.Server factory = KeyStoreSSLContext.createSslContextFactory(null,
+ SslContextFactory.Server factory = JettySslContextFactory.createServerSslContextWithKeystore(null,
"JKS", Resources.getResource("ssl/jetty_server_key.jks").getPath(),
"jetty_server_pwd", false, "JKS",
Resources.getResource("ssl/jetty_server_trust.jks").getPath(),
@@ -71,7 +69,8 @@ public class JettySslContextFactoryWithAutoRefreshTest {
// client connect
HttpClientBuilder httpClientBuilder = HttpClients.custom();
RegistryBuilder<ConnectionSocketFactory> registryBuilder = RegistryBuilder.create();
- registryBuilder.register("https", new SSLConnectionSocketFactory(getClientSslContext(), new NoopHostnameVerifier()));
+ registryBuilder.register("https",
+ new SSLConnectionSocketFactory(getClientSslContext(), new NoopHostnameVerifier()));
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(registryBuilder.build());
httpClientBuilder.setConnectionManager(cm);
CloseableHttpClient httpClient = httpClientBuilder.build();
@@ -86,7 +85,7 @@ public class JettySslContextFactoryWithAutoRefreshTest {
Configurator.setRootLevel(Level.INFO);
Server server = new Server();
List<ServerConnector> connectors = new ArrayList<>();
- SslContextFactory.Server factory = KeyStoreSSLContext.createSslContextFactory(null,
+ SslContextFactory.Server factory = JettySslContextFactory.createServerSslContextWithKeystore(null,
"JKS", Resources.getResource("ssl/jetty_server_key.jks").getPath(),
"jetty_server_pwd", false, "JKS",
Resources.getResource("ssl/jetty_server_trust.jks").getPath(),
@@ -118,10 +117,9 @@ public class JettySslContextFactoryWithAutoRefreshTest {
@Test(expectedExceptions = SSLHandshakeException.class)
public void testJettyTlsServerInvalidCipher() throws Exception {
- Configurator.setRootLevel(Level.INFO);
Server server = new Server();
List<ServerConnector> connectors = new ArrayList<>();
- SslContextFactory.Server factory = KeyStoreSSLContext.createSslContextFactory(null,
+ SslContextFactory.Server factory = JettySslContextFactory.createServerSslContextWithKeystore(null,
"JKS", Resources.getResource("ssl/jetty_server_key.jks").getPath(),
"jetty_server_pwd", false, "JKS",
Resources.getResource("ssl/jetty_server_trust.jks").getPath(),
@@ -145,7 +143,8 @@ public class JettySslContextFactoryWithAutoRefreshTest {
HttpClientBuilder httpClientBuilder = HttpClients.custom();
RegistryBuilder<ConnectionSocketFactory> registryBuilder = RegistryBuilder.create();
registryBuilder.register("https", new SSLConnectionSocketFactory(getClientSslContext(),
- new String[]{"TLSv1.2"}, new String[]{"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"}, new NoopHostnameVerifier()));
+ new String[]{"TLSv1.2"}, new String[]{"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"},
+ new NoopHostnameVerifier()));
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(registryBuilder.build());
httpClientBuilder.setConnectionManager(cm);
CloseableHttpClient httpClient = httpClientBuilder.build();
@@ -167,7 +166,8 @@ public class JettySslContextFactoryWithAutoRefreshTest {
try {
SSLContext sslContext = SSLContext.getInstance("TLS");
// key store
- KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
+ KeyManagerFactory keyManagerFactory =
+ KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
KeyStore keyStore = KeyStore.getInstance("JKS");
try (FileInputStream inputStream = new FileInputStream(keyStorePath)) {
keyStore.load(inputStream, keyStorePassword.toCharArray());
@@ -175,7 +175,8 @@ public class JettySslContextFactoryWithAutoRefreshTest {
keyManagerFactory.init(keyStore, keyStorePassword.toCharArray());
KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();
// trust store
- TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
+ TrustManagerFactory trustManagerFactory =
+ TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
KeyStore trustStore = KeyStore.getInstance("JKS");
try (FileInputStream inputStream = new FileInputStream(trustStorePath)) {
trustStore.load(inputStream, trustStorePassword.toCharArray());
diff --git a/pulsar-broker-common/src/test/resources/ssl/jetty_client_key.jks b/pulsar-broker-common/src/test/resources/ssl/jetty_client_key.jks
new file mode 100644
index 00000000000..2b8ea64347d
Binary files /dev/null and b/pulsar-broker-common/src/test/resources/ssl/jetty_client_key.jks differ
diff --git a/pulsar-broker-common/src/test/resources/ssl/jetty_client_trust.jks b/pulsar-broker-common/src/test/resources/ssl/jetty_client_trust.jks
new file mode 100644
index 00000000000..166a2e00fb3
Binary files /dev/null and b/pulsar-broker-common/src/test/resources/ssl/jetty_client_trust.jks differ
diff --git a/pulsar-broker-common/src/test/resources/ssl/jetty_server_key.jks b/pulsar-broker-common/src/test/resources/ssl/jetty_server_key.jks
new file mode 100644
index 00000000000..b6189b75c8a
Binary files /dev/null and b/pulsar-broker-common/src/test/resources/ssl/jetty_server_key.jks differ
diff --git a/pulsar-broker-common/src/test/resources/ssl/jetty_server_trust.jks b/pulsar-broker-common/src/test/resources/ssl/jetty_server_trust.jks
new file mode 100644
index 00000000000..b09cc030a71
Binary files /dev/null and b/pulsar-broker-common/src/test/resources/ssl/jetty_server_trust.jks differ
diff --git a/pulsar-broker-common/src/test/resources/ssl/my-ca/ca.pem b/pulsar-broker-common/src/test/resources/ssl/my-ca/ca.pem
new file mode 100644
index 00000000000..3d5a80e2347
--- /dev/null
+++ b/pulsar-broker-common/src/test/resources/ssl/my-ca/ca.pem
@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC9DCCAdygAwIBAgIUNbNkV2+K2Hf4Q1V5gdAENZQiLokwDQYJKoZIhvcNAQEL
+BQAwETEPMA0GA1UEAxMGUHVsc2FyMCAXDTIyMDExNDA0MjgwMFoYDzIxMjIwMTE2
+MDQyODAwWjARMQ8wDQYDVQQDEwZQdWxzYXIwggEiMA0GCSqGSIb3DQEBAQUAA4IB
+DwAwggEKAoIBAQDBR2K5EKVziLqdsz78efEW4lOwKiJ32e97uxn1Z6oKgkgImpVP
+Z9aoJB4EwSnDg+6FV2YULdWPm7C6W33tDmWRaU/Hlo/cOejnK8UmiMu/EyDpE2Wj
+n0RimGmwOkBi2IWIcIzWMmPDZ9kZc65OUeEmwZedKRy62PQyfCeNU4OOHQn3PXjI
+NbXJZD5TvBmn4SJn2RP9EgmIPaBAh/Mng045ZeHHLhwMKC8EOyHc2aB7AL6brymR
+xzsiYWdcJn4mqqMvT82mVvhkgAMOcR4CXYF8eYnsG6ZbDHb13CawcvLVREJZk7AB
+XZi9Rd5xczxHILM8rdkIZfunaG1X5hbih5wJAgMBAAGjQjBAMA4GA1UdDwEB/wQE
+AwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTCC1lYG+62cUPjNk9q4jCm
+Ps65njANBgkqhkiG9w0BAQsFAAOCAQEAKV2Lpu5cH5EsG53EWsYxEKvuQZ0LTxCE
+wCDf/NxJaQbzfv0tsbZatMge0vcZ/5r8tZZoOC+pGTwk6MaRbEFH8PmvlH1LIQvu
+Y34/YQZOy8wBTWwaIfFMnYWc0iAFoFt2Lzuq+GOI+svTFp729Ae8r7UxY/f9Lioc
+ttdGr7vA6PpcIMoEIPjVp+m41uL9IDfX8eOxg4gVlwtqpbHdTzMrOz0YY+3qH/WK
+6Qffw4pwitzAEj2zCn2lvGC5cbpd13SAaqtB3xL/Aet0SS2r3g9qDo1RruQhXUng
+06U/Hqtn5K1fNQv3pivi3Jg5z1DfJWHkH37luAoIlOZHRmPK6rhp/g==
+-----END CERTIFICATE-----
diff --git a/pulsar-broker-common/src/test/resources/ssl/my-ca/client-ca.pem b/pulsar-broker-common/src/test/resources/ssl/my-ca/client-ca.pem
new file mode 100644
index 00000000000..adcae3393ad
--- /dev/null
+++ b/pulsar-broker-common/src/test/resources/ssl/my-ca/client-ca.pem
@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDHDCCAgSgAwIBAgIUJJpmKX3DnbUwJ7tUhCt8MTiwz0owDQYJKoZIhvcNAQEL
+BQAwETEPMA0GA1UEAxMGUHVsc2FyMCAXDTIyMDExNDA0MjgwMFoYDzIxMjExMjIx
+MDQyODAwWjARMQ8wDQYDVQQDEwZQdWxzYXIwggEiMA0GCSqGSIb3DQEBAQUAA4IB
+DwAwggEKAoIBAQDZN+CNZ1i1WaXulbwSASOfXErWXhGV9DHqavPp3DohgQdundfS
+648T/X80uWQlyxu4L4j0oc97jtzc1AyZFXj5nocVsveEO9aDjnYCc5NdBNJLQHgl
+IO59fEpTd55NO24g9a8/sxgn0ADCenMlngk1Ou+2QJBONw7W12/WUSUg6ICe+b+x
+qPzgApue16oGw9HxhPwa3oEvVZrEnFIWLjsSWtezhgFHMCH9/ngk0KlRyes/EZCz
+ZgkO5mgii2fmNDg+yuWUfw7Q0x6BJskGIrxisJiJBRR1+DIvJqgqxJsNmeeEQrZK
+YHBukj5RWDFOpOHgqFbPsv45sVKoLrGFrMnNAgMBAAGjajBoMA4GA1UdDwEB/wQE
+AwIFoDATBgNVHSUEDDAKBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQW
+BBSwkx93xjYP4I+dcFF3xS9NLesmFjAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJ
+KoZIhvcNAQELBQADggEBAAK3ZF63w46pT76QIOeSM3ocUm6izvW/IrxLUESfgRC4
+gg0/5VfPiHHUe6orn15KuPXHe7xCUFqc2oFn5aIU1B/6iOPeNItvMJidU0a3UAiw
+hFK9MSFgESNBiEnu1dE5tPcIIxTyCFQ/8loeY3dsdcNVoguH/2J9v/XcMMga46A1
+wudaaa1nb+ZYnXkRuyObKVJQN7EqC+4edinMOTPBbF9wtRMAMBRHXXENXb9zFthi
+Dbdn4YvadYsNHxh5ar+hQn/HSPMuCUPY/uUqxtBagb6aS0YnSoUscSLs1Jizg5NX
+d+QV8X/5E6W4xWnptUZwVxOemkdnr6A8MH1eQKKFZTM=
+-----END CERTIFICATE-----
diff --git a/pulsar-broker-common/src/test/resources/ssl/my-ca/client-key.pem b/pulsar-broker-common/src/test/resources/ssl/my-ca/client-key.pem
new file mode 100644
index 00000000000..5b08b151c80
--- /dev/null
+++ b/pulsar-broker-common/src/test/resources/ssl/my-ca/client-key.pem
@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDZN+CNZ1i1WaXu
+lbwSASOfXErWXhGV9DHqavPp3DohgQdundfS648T/X80uWQlyxu4L4j0oc97jtzc
+1AyZFXj5nocVsveEO9aDjnYCc5NdBNJLQHglIO59fEpTd55NO24g9a8/sxgn0ADC
+enMlngk1Ou+2QJBONw7W12/WUSUg6ICe+b+xqPzgApue16oGw9HxhPwa3oEvVZrE
+nFIWLjsSWtezhgFHMCH9/ngk0KlRyes/EZCzZgkO5mgii2fmNDg+yuWUfw7Q0x6B
+JskGIrxisJiJBRR1+DIvJqgqxJsNmeeEQrZKYHBukj5RWDFOpOHgqFbPsv45sVKo
+LrGFrMnNAgMBAAECggEATeVZ45uiFja16J9NuG8sJSPluoY1bD8L/3KnUcAmIImy
+7powIXVT8+k+StwI6/ywThbN2FyGmVqcHZz1f5hRr8KH0uJBHOyQetEFxM9Jk1v9
+Rfsymq36mImP5erJnAyp66vvUrqY+P4Ap71duam4x5wBBqyUk1fvPGA5vPOQiwHs
+TN9JHizGobY25fpigWKIMamyE7HWXEUzVdOo83ZiNx53ths+WcF/kqto2v5LtyfJ
+HgoPocfZI8tRz9tfgc8zOkvyjsvgdd6rLhd0r2oExnyQBJdktGFpQZMGambU328u
+NqcdJscjP/HWAHRzuSdOvCMOEn8E5GIjcWEnQqOmSQKBgQDcpb655/UdcVxrv2Ou
+8juucDJMpf6i/UcmlXVXx+3zGSuQZcCC2fupe3JcxPdK7bo65YlC3OoRihggh2sS
+cnFMNHMfyoE3G/doXIr3QyL9UAQt4yb+7Nz7jRXYcg4Ytv+FVS6BSzIDEK17v+es
+GuWDM3JwtigtzYS4tRh7lgmuBwKBgQD8BXp7yIyVv657B8OJJSoeGataziFPhZux
+WKoS3gq24169ZWXwLc+nwrdgvBNrRaHuX+cYh93RF9+2WZrRcRL41XqN938adasY
+zPsfOJa9IOgUzQtGUMSe1/WqvHfcvqZCqYq4u/LSdf+I67woP4tCqqn4E928aIZb
+6PjLH+dUiwKBgH1ntn7y1t1lEKIspPtJsaHzIqNttMvuKAJF7+t0Nkl0hM4NBt1Y
+BzDMeLNBP0vW0YGn89uMs3xEgHH8hV52rO4i4UuwTMCFpJgsAM+H2NsgHz/1WrSI
+6xANn9zk9h4V5CRjxYq2sjYLxI4RBBtNLiTjmKd24F8n78cLJl8XZ2kBAoGAGoHF
+ATH1v2ZaxqvpYApdpK7UfAeEL2YBGyUVNkjOXbAKbec1Uo6u8ZkkSnNdo4G+Z2EE
+4Gqh5PUa3YYNJ4w6D5v8eOQYJUNNDJ26p+z+xcOpRU7PqcSi+YYDW8LY5InU2NwW
+MBnsj0BD8TXCI4WTcx6aI/KK9t8TiqU1Tb/8R8MCgYANVinOLz2enB+Qzu4o88W/
+witKHI3D9+z/uWjp0Q4rwmr3OL4FD9vZWvL4qwbDgpfLirJ4e3UVfN1/FoytAKlk
+Kykf8oDWciCIdxStt/yUpgQv78IL3vM5d9B8Qb7KCRtJ0BIXGJ7Gle3xJeuduZLe
++F+hwI3Dpv5HPqa9o6ttJw==
+-----END PRIVATE KEY-----
diff --git a/pulsar-broker-common/src/test/resources/ssl/my-ca/server-ca.pem b/pulsar-broker-common/src/test/resources/ssl/my-ca/server-ca.pem
new file mode 100644
index 00000000000..df5f69298e2
--- /dev/null
+++ b/pulsar-broker-common/src/test/resources/ssl/my-ca/server-ca.pem
@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDHDCCAgSgAwIBAgIUVQHD0/oi9Ca50HA7DFLYOO2wEzYwDQYJKoZIhvcNAQEL
+BQAwETEPMA0GA1UEAxMGUHVsc2FyMCAXDTIyMDExNDA0MjgwMFoYDzIxMjExMjIx
+MDQyODAwWjARMQ8wDQYDVQQDEwZQdWxzYXIwggEiMA0GCSqGSIb3DQEBAQUAA4IB
+DwAwggEKAoIBAQDBcqDkMhjLd9ik//UQijqbajQP5t6dvVZNn9gODQrS9oB/URur
+NzCcPWYPJZfEJlTkV8mlmgq4dBjwghpy5ALOGiERk55JPIN4cy01hQ6j7YSPFvMv
+BjqZvm5dpGDNTr7GY7THegMM1wpk9EaUOm7tBOHtf6ZnANjSMcQM74RCSBt0Koqw
+06CKVDCbgJ5NNE1LgwYeVQAwtQAhY8rqqQKJvCorFbq7OiisFBnz5pRBT6N4kMo1
+9LZo3Oe2F2w9eH9vacQ0NjSOCNXqal9Xl/Pwy9JgKKppwZ/3nCgRc+yfjrnkRz0f
+b+llb2NpR5Ge+tNMakqelE8bDSw/5BPjRPftAgMBAAGjajBoMA4GA1UdDwEB/wQE
+AwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQW
+BBRXws5mmLbW+xOLflUyUZ0I0uN96zAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJ
+KoZIhvcNAQELBQADggEBAKMklpYJIkp4icz9Ea5wWQiRXWb94lGdyCA833VHeGB2
+fKvNXj1d6lEiy26pOjhDmycroKelj70WqOsqVgi4xh4Y9sj6pwb8Q423Tu3qNO1k
+qaScTar2DANSigNzqlSbLshPWQ2ZyDwkvZPuqPgHzOXekzbUGwxgCiySaQkl2mCS
+mBaG3XnESwiMIKkLphEv0MAvTVaImbSRWYEQ4OECwcHXxx+14wK8NLcdDIHcSzki
+8Eq24CxDOeL5QxciGMi5tylsdCpT+D/BXTKiu46yoRjXUsTLYL53yUZZIqQ3A4CV
+enZ/vHhP0Ev9RcRigFTqrBm7EC3b2AUpvqgRMnPwQZo=
+-----END CERTIFICATE-----
diff --git a/pulsar-broker-common/src/test/resources/ssl/my-ca/server-key.pem b/pulsar-broker-common/src/test/resources/ssl/my-ca/server-key.pem
new file mode 100644
index 00000000000..a3f3a36b73c
--- /dev/null
+++ b/pulsar-broker-common/src/test/resources/ssl/my-ca/server-key.pem
@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDBcqDkMhjLd9ik
+//UQijqbajQP5t6dvVZNn9gODQrS9oB/URurNzCcPWYPJZfEJlTkV8mlmgq4dBjw
+ghpy5ALOGiERk55JPIN4cy01hQ6j7YSPFvMvBjqZvm5dpGDNTr7GY7THegMM1wpk
+9EaUOm7tBOHtf6ZnANjSMcQM74RCSBt0Koqw06CKVDCbgJ5NNE1LgwYeVQAwtQAh
+Y8rqqQKJvCorFbq7OiisFBnz5pRBT6N4kMo19LZo3Oe2F2w9eH9vacQ0NjSOCNXq
+al9Xl/Pwy9JgKKppwZ/3nCgRc+yfjrnkRz0fb+llb2NpR5Ge+tNMakqelE8bDSw/
+5BPjRPftAgMBAAECggEBAJm2JsgMUo1ihn/dbnIdFCKoCgRUs7FtYCVADOJlVKN7
+AXGpFi4/JV4Qn4cLnQNcXfovE2iF9VzJy4NYLgH60YvJUVtxC8Yv0lukUVkEiDST
+p9A3MTa9YVUG7xVzZwPcPVTQpzYV6lSKjpTXUTm5EKk/RvJ7itKv5plmt9x7eYFb
+/JwqXo1Z6C4gfIFR85LWmrCsNUK5T9oooLz88D6+ZH3+fWlr75RDff2kqdLshMTs
+N0Ov7NXcRFeruFs/IPrgTxjBMeNa2LFdYVPeeQ41L4uOI49uVBAmSn1be+THvDoj
+Do+6wTEF/h6/VLoOaIFZZdHlqd4is+xcEg8gwVkCn2ECgYEAxqVvGKc9qaqEVwBx
+U5Ru9OFx0NqEBvkYZRbCg1REcMFd3lqFTHvHiF3pmCp0XgLJKYuy42618IJXhj6D
+Y15/p9jX0025MpnH/AdwpO6x5pv6gb/JOMnHOnq8sI3R+V6TVsv1WZj0sOj94mF0
++Od++bQkUnSlfE4X7v+cJfo/Q8UCgYEA+Uz1yOyI9Dv1dEdBMdBA8MTriYU0uJCV
+dVKzL/uC9XyguVBWu1HX0MvEKyjPRycvLB7TuQqAFLgCtC8EEuPGBpWtyXOm9Jxw
+ToCfUZFuBQeMuf4vZcFgJjiEKTdKBxrvjkhyIhPR6JAy0WUr8Ry+ZtqvmG5NOEz5
+ptm1tznYngkCgYEAlckeyV8p/uqF2biKu3QcamgoU0zB6yQfAfK0fySmasNTzZtC
+EhbvsOLnhgbVMiI1ny8ol5fedtlBuAchOWeDKIQ40as0r3QHuQG/LY6S9Im+zeFY
+kIqNwInWB+cYYkmvHe6zNXlBYLh+4BmOgzTDqPPtw4MTWXTlVSDGlFhrJeUCgYBX
+7rlS4Xt9ChkNpoRsWZROWGbr3rw1zWmqND1X01Lh28+lDZ1J/RguYXET+BUEd+G/
+oi/zuKxsomrxuxOoxgZ3FBx0TgK5jORgDCYl0zIHPB57DBkTvx123cBf+Ux3LR0K
+BqubMXp8mUATc6gIJ6dRCBmfnmhGT4BPRcM+mXy6YQKBgGEGH37VABus+Oi3g1bk
+qEAaUI1asRLJIfbY2ImxEroLIQAbTFuIQUsZTKpT7jJZubjYvy1Fev0LU/n7Kv2w
+7ym41z70ro5uxwUBfJjnF3RtgncNcftn4b3siNzvBfKEBuhegMeS5YAbBIwABUpR
+4mVpm9BLOiX4yENIT6JdUQFc
+-----END PRIVATE KEY-----
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index 08fff1eea36..73843071f98 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -1418,6 +1418,8 @@ public class PulsarService implements AutoCloseable, ShutdownService {
conf.getBrokerClientAuthenticationParameters());
if (conf.isBrokerClientTlsEnabled()) {
+ builder.tlsCiphers(config.getBrokerClientTlsCiphers())
+ .tlsProtocols(config.getBrokerClientTlsProtocols());
if (conf.isBrokerClientTlsEnabledWithKeyStore()) {
builder.useKeyStoreTls(true)
.tlsTrustStoreType(conf.getBrokerClientTlsTrustStoreType())
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
index fc800880b83..73a7cd4a0ff 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
@@ -30,8 +30,7 @@ import javax.servlet.DispatcherType;
import org.apache.pulsar.broker.PulsarServerException;
import org.apache.pulsar.broker.PulsarService;
import org.apache.pulsar.broker.ServiceConfiguration;
-import org.apache.pulsar.common.util.SecurityUtility;
-import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
+import org.apache.pulsar.jetty.tls.JettySslContextFactory;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
@@ -101,8 +100,8 @@ public class WebService implements AutoCloseable {
SslContextFactory sslCtxFactory;
ServiceConfiguration config = pulsar.getConfiguration();
if (config.isTlsEnabledWithKeyStore()) {
- sslCtxFactory = KeyStoreSSLContext.createSslContextFactory(
- config.getTlsProvider(),
+ sslCtxFactory = JettySslContextFactory.createServerSslContextWithKeystore(
+ config.getWebServiceTlsProvider(),
config.getTlsKeyStoreType(),
config.getTlsKeyStore(),
config.getTlsKeyStorePassword(),
@@ -116,12 +115,15 @@ public class WebService implements AutoCloseable {
config.getTlsCertRefreshCheckDurationSec()
);
} else {
- sslCtxFactory = SecurityUtility.createSslContextFactory(
+ sslCtxFactory = JettySslContextFactory.createServerSslContext(
+ config.getWebServiceTlsProvider(),
config.isTlsAllowInsecureConnection(),
config.getTlsTrustCertsFilePath(),
config.getTlsCertificateFilePath(),
config.getTlsKeyFilePath(),
- config.isTlsRequireTrustedClientCertOnConnect(), true,
+ config.isTlsRequireTrustedClientCertOnConnect(),
+ config.getWebServiceTlsCiphers(),
+ config.getWebServiceTlsProtocols(),
config.getTlsCertRefreshCheckDurationSec());
}
httpsConnector = new PulsarServerConnector(server, 1, 1, sslCtxFactory);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPublishConsumeTlsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPublishConsumeTlsTest.java
index b96880b5627..a8b67416107 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPublishConsumeTlsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPublishConsumeTlsTest.java
@@ -103,7 +103,7 @@ public class ProxyPublishConsumeTlsTest extends TlsProducerConsumerBase {
SslContextFactory sslContextFactory = new SslContextFactory();
sslContextFactory.setSslContext(SecurityUtility
- .createSslContext(false, SecurityUtility.loadCertificatesFromPemFile(TLS_TRUST_CERT_FILE_PATH)));
+ .createSslContext(false, SecurityUtility.loadCertificatesFromPemFile(TLS_TRUST_CERT_FILE_PATH), null));
WebSocketClient consumeClient = new WebSocketClient(sslContextFactory);
SimpleConsumerSocket consumeSocket = new SimpleConsumerSocket();
diff --git a/pulsar-common/pom.xml b/pulsar-common/pom.xml
index 643432392a0..789880a563b 100644
--- a/pulsar-common/pom.xml
+++ b/pulsar-common/pom.xml
@@ -136,11 +136,6 @@
<artifactId>netty-codec-haproxy</artifactId>
</dependency>
- <dependency>
- <groupId>org.eclipse.jetty</groupId>
- <artifactId>jetty-util</artifactId>
- </dependency>
-
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
@@ -169,12 +164,6 @@
</dependency>
<!-- test -->
- <dependency>
- <groupId>org.eclipse.jetty</groupId>
- <artifactId>jetty-server</artifactId>
- <scope>test</scope>
- </dependency>
-
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bc-fips</artifactId>
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/DefaultSslContextBuilder.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/DefaultSslContextBuilder.java
index b1e8a14ff95..2e67b02f90b 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/DefaultSslContextBuilder.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/DefaultSslContextBuilder.java
@@ -18,11 +18,8 @@
*/
package org.apache.pulsar.common.util;
-import java.io.FileNotFoundException;
-import java.io.IOException;
import java.security.GeneralSecurityException;
import javax.net.ssl.SSLContext;
-import javax.net.ssl.SSLException;
@SuppressWarnings("checkstyle:JavadocType")
public class DefaultSslContextBuilder extends SslContextAutoRefreshBuilder<SSLContext> {
@@ -31,23 +28,37 @@ public class DefaultSslContextBuilder extends SslContextAutoRefreshBuilder<SSLCo
protected final boolean tlsAllowInsecureConnection;
protected final FileModifiedTimeUpdater tlsTrustCertsFilePath, tlsCertificateFilePath, tlsKeyFilePath;
protected final boolean tlsRequireTrustedClientCertOnConnect;
+ private final String providerName;
public DefaultSslContextBuilder(boolean allowInsecure, String trustCertsFilePath, String certificateFilePath,
- String keyFilePath, boolean requireTrustedClientCertOnConnect, long certRefreshInSec)
- throws SSLException, FileNotFoundException, GeneralSecurityException, IOException {
+ String keyFilePath, boolean requireTrustedClientCertOnConnect,
+ long certRefreshInSec) {
super(certRefreshInSec);
this.tlsAllowInsecureConnection = allowInsecure;
this.tlsTrustCertsFilePath = new FileModifiedTimeUpdater(trustCertsFilePath);
this.tlsCertificateFilePath = new FileModifiedTimeUpdater(certificateFilePath);
this.tlsKeyFilePath = new FileModifiedTimeUpdater(keyFilePath);
this.tlsRequireTrustedClientCertOnConnect = requireTrustedClientCertOnConnect;
+ this.providerName = null;
+ }
+
+ public DefaultSslContextBuilder(boolean allowInsecure, String trustCertsFilePath, String certificateFilePath,
+ String keyFilePath, boolean requireTrustedClientCertOnConnect,
+ long certRefreshInSec, String providerName) {
+ super(certRefreshInSec);
+ this.tlsAllowInsecureConnection = allowInsecure;
+ this.tlsTrustCertsFilePath = new FileModifiedTimeUpdater(trustCertsFilePath);
+ this.tlsCertificateFilePath = new FileModifiedTimeUpdater(certificateFilePath);
+ this.tlsKeyFilePath = new FileModifiedTimeUpdater(keyFilePath);
+ this.tlsRequireTrustedClientCertOnConnect = requireTrustedClientCertOnConnect;
+ this.providerName = providerName;
}
@Override
public synchronized SSLContext update() throws GeneralSecurityException {
this.sslContext = SecurityUtility.createSslContext(tlsAllowInsecureConnection,
tlsTrustCertsFilePath.getFileName(), tlsCertificateFilePath.getFileName(),
- tlsKeyFilePath.getFileName());
+ tlsKeyFilePath.getFileName(), this.providerName);
return this.sslContext;
}
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
index e9266589ffc..d2905e01be2 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
@@ -64,7 +64,6 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.common.classification.InterfaceAudience;
import org.apache.pulsar.common.tls.TlsHostnameVerifier;
-import org.eclipse.jetty.util.ssl.SslContextFactory;
/**
* Helper class for the security domain.
@@ -196,9 +195,10 @@ public class SecurityUtility {
return provider;
}
- public static SSLContext createSslContext(boolean allowInsecureConnection, Certificate[] trustCertificates)
+ public static SSLContext createSslContext(boolean allowInsecureConnection, Certificate[] trustCertificates,
+ String providerName)
throws GeneralSecurityException {
- return createSslContext(allowInsecureConnection, trustCertificates, (Certificate[]) null, (PrivateKey) null);
+ return createSslContext(allowInsecureConnection, trustCertificates, null, null, providerName);
}
public static SslContext createNettySslContextForClient(SslProvider sslProvider, boolean allowInsecureConnection,
@@ -211,11 +211,11 @@ public class SecurityUtility {
}
public static SSLContext createSslContext(boolean allowInsecureConnection, String trustCertsFilePath,
- String certFilePath, String keyFilePath) throws GeneralSecurityException {
+ String certFilePath, String keyFilePath, String providerName) throws GeneralSecurityException {
X509Certificate[] trustCertificates = loadCertificatesFromPemFile(trustCertsFilePath);
X509Certificate[] certificates = loadCertificatesFromPemFile(certFilePath);
PrivateKey privateKey = loadPrivateKeyFromPemFile(keyFilePath);
- return createSslContext(allowInsecureConnection, trustCertificates, certificates, privateKey);
+ return createSslContext(allowInsecureConnection, trustCertificates, certificates, privateKey, providerName);
}
/**
@@ -323,18 +323,25 @@ public class SecurityUtility {
}
public static SSLContext createSslContext(boolean allowInsecureConnection, Certificate[] trustCertficates,
- Certificate[] certificates, PrivateKey privateKey) throws GeneralSecurityException {
+ Certificate[] certificates, PrivateKey privateKey)
+ throws GeneralSecurityException {
+ return createSslContext(allowInsecureConnection, trustCertficates, certificates, privateKey, null);
+ }
+
+ public static SSLContext createSslContext(boolean allowInsecureConnection, Certificate[] trustCertficates,
+ Certificate[] certificates, PrivateKey privateKey, String providerName)
+ throws GeneralSecurityException {
KeyStoreHolder ksh = new KeyStoreHolder();
TrustManager[] trustManagers = null;
KeyManager[] keyManagers = null;
+ Provider provider = resolveProvider(providerName);
- trustManagers = setupTrustCerts(ksh, allowInsecureConnection, trustCertficates, CONSCRYPT_PROVIDER);
+ trustManagers = setupTrustCerts(ksh, allowInsecureConnection, trustCertficates, provider);
keyManagers = setupKeyManager(ksh, privateKey, certificates);
- SSLContext sslCtx = CONSCRYPT_PROVIDER != null ? SSLContext.getInstance("TLS", CONSCRYPT_PROVIDER)
+ SSLContext sslCtx = provider != null ? SSLContext.getInstance("TLS", provider)
: SSLContext.getInstance("TLS");
sslCtx.init(keyManagers, trustManagers, new SecureRandom());
- sslCtx.getDefaultSSLParameters();
return sslCtx;
}
@@ -541,51 +548,16 @@ public class SecurityUtility {
}
}
- public static SslContextFactory createSslContextFactory(boolean tlsAllowInsecureConnection,
- String tlsTrustCertsFilePath, String tlsCertificateFilePath, String tlsKeyFilePath,
- boolean tlsRequireTrustedClientCertOnConnect, boolean autoRefresh, long certRefreshInSec)
- throws GeneralSecurityException, SSLException, FileNotFoundException, IOException {
- SslContextFactory sslCtxFactory = null;
- if (autoRefresh) {
- sslCtxFactory = new SslContextFactoryWithAutoRefresh(tlsAllowInsecureConnection, tlsTrustCertsFilePath,
- tlsCertificateFilePath, tlsKeyFilePath, tlsRequireTrustedClientCertOnConnect, certRefreshInSec);
- } else {
- sslCtxFactory = new SslContextFactory();
- SSLContext sslCtx = createSslContext(tlsAllowInsecureConnection, tlsTrustCertsFilePath,
- tlsCertificateFilePath, tlsKeyFilePath);
- sslCtxFactory.setSslContext(sslCtx);
+ public static Provider resolveProvider(String providerName) throws NoSuchAlgorithmException {
+ Provider provider = null;
+ if (!StringUtils.isEmpty(providerName)) {
+ provider = Security.getProvider(providerName);
}
- if (tlsRequireTrustedClientCertOnConnect) {
- sslCtxFactory.setNeedClientAuth(true);
- } else {
- sslCtxFactory.setWantClientAuth(true);
- }
- sslCtxFactory.setTrustAll(true);
- return sslCtxFactory;
- }
- /**
- * {@link SslContextFactory} that auto-refresh SSLContext.
- */
- static class SslContextFactoryWithAutoRefresh extends SslContextFactory {
-
- private final DefaultSslContextBuilder sslCtxRefresher;
-
- public SslContextFactoryWithAutoRefresh(boolean tlsAllowInsecureConnection, String tlsTrustCertsFilePath,
- String tlsCertificateFilePath, String tlsKeyFilePath, boolean tlsRequireTrustedClientCertOnConnect,
- long certRefreshInSec)
- throws SSLException, FileNotFoundException, GeneralSecurityException, IOException {
- super();
- sslCtxRefresher = new DefaultSslContextBuilder(tlsAllowInsecureConnection, tlsTrustCertsFilePath,
- tlsCertificateFilePath, tlsKeyFilePath, tlsRequireTrustedClientCertOnConnect, certRefreshInSec);
- if (CONSCRYPT_PROVIDER != null) {
- setProvider(CONSCRYPT_PROVIDER.getName());
- }
+ if (provider == null) {
+ provider = SSLContext.getDefault().getProvider();
}
- @Override
- public SSLContext getSslContext() {
- return sslCtxRefresher.get();
- }
+ return provider;
}
}
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefresh.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefresh.java
deleted file mode 100644
index c657905cb72..00000000000
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/JettySslContextFactoryWithAutoRefresh.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pulsar.common.util.keystoretls;
-
-import java.util.Set;
-import javax.net.ssl.SSLContext;
-import org.eclipse.jetty.util.ssl.SslContextFactory;
-
-/**
- * SslContextFactoryWithAutoRefresh that create SSLContext for web server, and refresh in time.
- */
-public class JettySslContextFactoryWithAutoRefresh extends SslContextFactory.Server {
- private final NetSslContextBuilder sslCtxRefresher;
-
- public JettySslContextFactoryWithAutoRefresh(String sslProviderString,
- String keyStoreTypeString,
- String keyStore,
- String keyStorePassword,
- boolean allowInsecureConnection,
- String trustStoreTypeString,
- String trustStore,
- String trustStorePassword,
- boolean requireTrustedClientCertOnConnect,
- Set<String> ciphers,
- Set<String> protocols,
- long certRefreshInSec) {
- super();
- sslCtxRefresher = new NetSslContextBuilder(
- sslProviderString,
- keyStoreTypeString,
- keyStore,
- keyStorePassword,
- allowInsecureConnection,
- trustStoreTypeString,
- trustStore,
- trustStorePassword,
- requireTrustedClientCertOnConnect,
- certRefreshInSec);
- if (ciphers != null && ciphers.size() > 0) {
- this.setIncludeCipherSuites(ciphers.toArray(new String[0]));
- }
- if (protocols != null && protocols.size() > 0) {
- this.setIncludeProtocols(protocols.toArray(new String[0]));
- }
- if (sslProviderString != null) {
- setProvider(sslProviderString);
- }
- }
-
- @Override
- public SSLContext getSslContext() {
- return sslCtxRefresher.get();
- }
-}
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
index 1ed310946f2..bed51e81680 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
@@ -38,7 +38,6 @@ import javax.net.ssl.TrustManagerFactory;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.common.util.SecurityUtility;
-import org.eclipse.jetty.util.ssl.SslContextFactory;
/**
* KeyStoreSSLContext that mainly wrap a SSLContext to provide SSL context for both webservice and netty.
@@ -129,8 +128,10 @@ public class KeyStoreSSLContext {
public SSLContext createSSLContext() throws GeneralSecurityException, IOException {
SSLContext sslContext;
- if (sslProviderString != null) {
- sslContext = SSLContext.getInstance(protocol, sslProviderString);
+
+ Provider provider = SecurityUtility.resolveProvider(sslProviderString);
+ if (provider != null) {
+ sslContext = SSLContext.getInstance(protocol, provider);
} else {
sslContext = SSLContext.getInstance(protocol);
}
@@ -153,8 +154,8 @@ public class KeyStoreSSLContext {
if (this.allowInsecureConnection) {
trustManagerFactory = InsecureTrustManagerFactory.INSTANCE;
} else {
- trustManagerFactory = sslProviderString != null
- ? TrustManagerFactory.getInstance(tmfAlgorithm, sslProviderString)
+ trustManagerFactory = provider != null
+ ? TrustManagerFactory.getInstance(tmfAlgorithm, provider)
: TrustManagerFactory.getInstance(tmfAlgorithm);
KeyStore trustStore = KeyStore.getInstance(trustStoreTypeString);
char[] passwordChars = trustStorePassword.toCharArray();
@@ -338,51 +339,4 @@ public class KeyStoreSSLContext {
return keyStoreSSLContext.createSSLContext();
}
-
- // for web server. autoRefresh is default true.
- public static SslContextFactory.Server createSslContextFactory(String sslProviderString,
- String keyStoreTypeString,
- String keyStore,
- String keyStorePassword,
- boolean allowInsecureConnection,
- String trustStoreTypeString,
- String trustStore,
- String trustStorePassword,
- boolean requireTrustedClientCertOnConnect,
- Set<String> ciphers,
- Set<String> protocols,
- long certRefreshInSec) {
- SslContextFactory.Server sslCtxFactory;
-
- if (sslProviderString == null) {
- Provider provider = SecurityUtility.CONSCRYPT_PROVIDER;
- if (provider != null) {
- sslProviderString = provider.getName();
- }
- }
-
- sslCtxFactory = new JettySslContextFactoryWithAutoRefresh(
- sslProviderString,
- keyStoreTypeString,
- keyStore,
- keyStorePassword,
- allowInsecureConnection,
- trustStoreTypeString,
- trustStore,
- trustStorePassword,
- requireTrustedClientCertOnConnect,
- ciphers,
- protocols,
- certRefreshInSec);
-
- if (requireTrustedClientCertOnConnect) {
- sslCtxFactory.setNeedClientAuth(true);
- } else {
- sslCtxFactory.setWantClientAuth(true);
- }
- sslCtxFactory.setTrustAll(true);
-
- return sslCtxFactory;
- }
}
-
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java
index 1339a5c4cc0..831a474a51d 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java
@@ -25,12 +25,11 @@ import org.apache.pulsar.broker.web.AuthenticationFilter;
import org.apache.pulsar.broker.web.RateLimitingFilter;
import org.apache.pulsar.broker.web.JettyRequestLogFactory;
import org.apache.pulsar.broker.web.WebExecutorThreadPool;
-import org.apache.pulsar.common.util.SecurityUtility;
-import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
import org.apache.pulsar.functions.worker.WorkerConfig;
import org.apache.pulsar.functions.worker.WorkerService;
import org.apache.pulsar.functions.worker.rest.api.v2.WorkerApiV2Resource;
import org.apache.pulsar.functions.worker.rest.api.v2.WorkerStatsApiV2Resource;
+import org.apache.pulsar.jetty.tls.JettySslContextFactory;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
@@ -125,7 +124,7 @@ public class WorkerServer {
try {
SslContextFactory sslCtxFactory;
if (workerConfig.isTlsEnabledWithKeyStore()) {
- sslCtxFactory = KeyStoreSSLContext.createSslContextFactory(
+ sslCtxFactory = JettySslContextFactory.createServerSslContextWithKeystore(
workerConfig.getTlsProvider(),
workerConfig.getTlsKeyStoreType(),
workerConfig.getTlsKeyStore(),
@@ -140,14 +139,17 @@ public class WorkerServer {
workerConfig.getTlsCertRefreshCheckDurationSec()
);
} else {
- sslCtxFactory = SecurityUtility.createSslContextFactory(
+ sslCtxFactory = JettySslContextFactory.createServerSslContext(
+ workerConfig.getTlsProvider(),
workerConfig.isTlsAllowInsecureConnection(),
workerConfig.getTlsTrustCertsFilePath(),
workerConfig.getTlsCertificateFilePath(),
workerConfig.getTlsKeyFilePath(),
workerConfig.isTlsRequireTrustedClientCertOnConnect(),
- true,
- workerConfig.getTlsCertRefreshCheckDurationSec());
+ workerConfig.getWebServiceTlsCiphers(),
+ workerConfig.getWebServiceTlsProtocols(),
+ workerConfig.getTlsCertRefreshCheckDurationSec()
+ );
}
httpsConnector = new ServerConnector(server, sslCtxFactory);
httpsConnector.setPort(this.workerConfig.getWorkerPortTls());
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
index a38e2f33b68..56d07cc14a5 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java
@@ -269,15 +269,17 @@ class AdminProxyHandler extends ProxyServlet {
AuthenticationDataProvider authData = auth.getAuthData();
if (authData.hasDataForTls()) {
sslCtx = SecurityUtility.createSslContext(
- config.isTlsAllowInsecureConnection(),
- trustCertificates,
- authData.getTlsCertificates(),
- authData.getTlsPrivateKey()
+ config.isTlsAllowInsecureConnection(),
+ trustCertificates,
+ authData.getTlsCertificates(),
+ authData.getTlsPrivateKey(),
+ config.getBrokerClientSslProvider()
);
} else {
sslCtx = SecurityUtility.createSslContext(
- config.isTlsAllowInsecureConnection(),
- trustCertificates
+ config.isTlsAllowInsecureConnection(),
+ trustCertificates,
+ config.getBrokerClientSslProvider()
);
}
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
index b79314522da..2c8204bc231 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
@@ -259,6 +259,12 @@ public class ProxyConfiguration implements PulsarConfiguration {
)
private Optional<Integer> webServicePortTls = Optional.empty();
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "Specify the TLS provider for the web service, available values can be SunJSSE, Conscrypt and etc."
+ )
+ private String webServiceTlsProvider = "Conscrypt";
+
@FieldContext(
category = CATEGORY_TLS,
doc = "Specify the tls protocols the proxy's web service will use to negotiate during TLS Handshake.\n\n"
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/WebServer.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/WebServer.java
index 84b1153b386..35f717361f2 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/WebServer.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/WebServer.java
@@ -35,8 +35,7 @@ import org.apache.pulsar.broker.web.JettyRequestLogFactory;
import org.apache.pulsar.broker.web.JsonMapperProvider;
import org.apache.pulsar.broker.web.RateLimitingFilter;
import org.apache.pulsar.broker.web.WebExecutorThreadPool;
-import org.apache.pulsar.common.util.SecurityUtility;
-import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
+import org.apache.pulsar.jetty.tls.JettySslContextFactory;
import org.eclipse.jetty.server.Connector;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.HttpConfiguration;
@@ -98,8 +97,8 @@ public class WebServer {
try {
SslContextFactory sslCtxFactory;
if (config.isTlsEnabledWithKeyStore()) {
- sslCtxFactory = KeyStoreSSLContext.createSslContextFactory(
- config.getTlsProvider(),
+ sslCtxFactory = JettySslContextFactory.createServerSslContextWithKeystore(
+ config.getWebServiceTlsProvider(),
config.getTlsKeyStoreType(),
config.getTlsKeyStore(),
config.getTlsKeyStorePassword(),
@@ -113,13 +112,15 @@ public class WebServer {
config.getTlsCertRefreshCheckDurationSec()
);
} else {
- sslCtxFactory = SecurityUtility.createSslContextFactory(
+ sslCtxFactory = JettySslContextFactory.createServerSslContext(
+ config.getWebServiceTlsProvider(),
config.isTlsAllowInsecureConnection(),
config.getTlsTrustCertsFilePath(),
config.getTlsCertificateFilePath(),
config.getTlsKeyFilePath(),
config.isTlsRequireTrustedClientCertOnConnect(),
- true,
+ config.getWebServiceTlsCiphers(),
+ config.getWebServiceTlsProtocols(),
config.getTlsCertRefreshCheckDurationSec());
}
connectorTls = new ServerConnector(server, 1, 1, sslCtxFactory);
diff --git a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java
index e3a30ad4ca7..efaee06c8f7 100644
--- a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java
+++ b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java
@@ -32,8 +32,7 @@ import org.apache.pulsar.broker.web.JettyRequestLogFactory;
import org.apache.pulsar.broker.web.JsonMapperProvider;
import org.apache.pulsar.broker.web.WebExecutorThreadPool;
import org.apache.pulsar.client.api.PulsarClientException;
-import org.apache.pulsar.common.util.SecurityUtility;
-import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
+import org.apache.pulsar.jetty.tls.JettySslContextFactory;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
@@ -75,7 +74,7 @@ public class ProxyServer {
try {
SslContextFactory sslCtxFactory;
if (config.isTlsEnabledWithKeyStore()) {
- sslCtxFactory = KeyStoreSSLContext.createSslContextFactory(
+ sslCtxFactory = JettySslContextFactory.createServerSslContextWithKeystore(
config.getTlsProvider(),
config.getTlsKeyStoreType(),
config.getTlsKeyStore(),
@@ -90,13 +89,15 @@ public class ProxyServer {
config.getTlsCertRefreshCheckDurationSec()
);
} else {
- sslCtxFactory = SecurityUtility.createSslContextFactory(
+ sslCtxFactory = JettySslContextFactory.createServerSslContext(
+ config.getTlsProvider(),
config.isTlsAllowInsecureConnection(),
config.getTlsTrustCertsFilePath(),
config.getTlsCertificateFilePath(),
config.getTlsKeyFilePath(),
config.isTlsRequireTrustedClientCertOnConnect(),
- true,
+ config.getWebServiceTlsCiphers(),
+ config.getWebServiceTlsProtocols(),
config.getTlsCertRefreshCheckDurationSec());
}
connectorTls = new ServerConnector(server, sslCtxFactory);
diff --git a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java
index 4480d4525e5..3585ee9b8ca 100644
--- a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java
+++ b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java
@@ -183,11 +183,9 @@ public class WebSocketProxyConfiguration implements PulsarConfiguration {
private boolean tlsEnabledWithKeyStore = false;
@FieldContext(
- doc = "Specify the TLS provider for the WebSocket service: \n"
- + "When using TLS authentication with CACert, the valid value is either OPENSSL or JDK.\n"
- + "When using TLS authentication with KeyStore, available values can be SunJSSE, Conscrypt and etc."
+ doc = "Specify the TLS provider for the WebSocket service: SunJSSE, Conscrypt and etc."
)
- private String tlsProvider = null;
+ private String tlsProvider = "Conscrypt";
@FieldContext(
doc = "TLS KeyStore type configuration in WebSocket: JKS, PKCS12"
[pulsar] 18/31: [security] Remove sensitive msg from consumer/producer stats log (#15483)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 6cbead560916fda42cde88f872ad79827c941ab9
Author: ZhangJian He <sh...@gmail.com>
AuthorDate: Mon May 9 07:39:31 2022 +0800
[security] Remove sensitive msg from consumer/producer stats log (#15483)
Currently, we are print password field to consumer/producer stats log
- add missed `@JsonIgnore` on field and getMethod
- delete unused `withoutAttribute` call
(cherry picked from commit 8b2f3dd095f365fdb22c71078d5a3e0bf6cc9626)
---
.../client/impl/ConsumerStatsRecorderImpl.java | 3 +-
.../client/impl/ProducerStatsRecorderImpl.java | 3 +-
.../client/impl/conf/ClientConfigurationData.java | 11 +++++
.../impl/conf/ClientConfigurationDataTest.java | 57 ++++++++++++++++++++++
4 files changed, 70 insertions(+), 4 deletions(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerStatsRecorderImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerStatsRecorderImpl.java
index 7850babf466..a41e036e516 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerStatsRecorderImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerStatsRecorderImpl.java
@@ -112,8 +112,7 @@ public class ConsumerStatsRecorderImpl implements ConsumerStatsRecorder {
try {
log.info("Starting Pulsar consumer status recorder with config: {}", w.writeValueAsString(conf));
- log.info("Pulsar client config: {}", w.withoutAttribute("authentication")
- .writeValueAsString(pulsarClient.getConfiguration()));
+ log.info("Pulsar client config: {}", w.writeValueAsString(pulsarClient.getConfiguration()));
} catch (IOException e) {
log.error("Failed to dump config info", e);
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java
index 1f1b5b2efe7..5a60dafb2b5 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java
@@ -104,8 +104,7 @@ public class ProducerStatsRecorderImpl implements ProducerStatsRecorder {
try {
log.info("Starting Pulsar producer perf with config: {}", w.writeValueAsString(conf));
- log.info("Pulsar client config: {}",
- w.withoutAttribute("authentication").writeValueAsString(pulsarClient.getConfiguration()));
+ log.info("Pulsar client config: {}", w.writeValueAsString(pulsarClient.getConfiguration()));
} catch (IOException e) {
log.error("Failed to dump config info", e);
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ClientConfigurationData.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ClientConfigurationData.java
index d691bbeed2a..c599a65c5d6 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ClientConfigurationData.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ClientConfigurationData.java
@@ -32,6 +32,7 @@ import java.util.TreeSet;
import java.util.concurrent.TimeUnit;
import lombok.AllArgsConstructor;
import lombok.Data;
+import lombok.Getter;
import lombok.NoArgsConstructor;
import org.apache.pulsar.client.api.Authentication;
import org.apache.pulsar.client.api.ProxyProtocol;
@@ -58,6 +59,7 @@ public class ClientConfigurationData implements Serializable, Cloneable {
value = "The implementation class of ServiceUrlProvider used to generate ServiceUrl."
)
@JsonIgnore
+ @Getter(onMethod_ = @JsonIgnore)
private transient ServiceUrlProvider serviceUrlProvider;
@ApiModelProperty(
@@ -253,6 +255,10 @@ public class ClientConfigurationData implements Serializable, Cloneable {
name = "tlsTrustStorePassword",
value = "Password of TLS TrustStore."
)
+
+ @Secret
+ @JsonIgnore
+ @Getter(onMethod_ = @JsonIgnore)
private String tlsTrustStorePassword = null;
@ApiModelProperty(
@@ -324,8 +330,12 @@ public class ClientConfigurationData implements Serializable, Cloneable {
name = "socks5ProxyUsername",
value = "Password of SOCKS5 proxy."
)
+
+ @Secret
+ @JsonIgnore
private String socks5ProxyPassword;
+ @JsonIgnore
public Authentication getAuthentication() {
if (authentication == null) {
this.authentication = AuthenticationDisabled.INSTANCE;
@@ -383,6 +393,7 @@ public class ClientConfigurationData implements Serializable, Cloneable {
return Objects.nonNull(socks5ProxyUsername) ? socks5ProxyUsername : System.getProperty("socks5Proxy.username");
}
+ @JsonIgnore
public String getSocks5ProxyPassword() {
return Objects.nonNull(socks5ProxyPassword) ? socks5ProxyPassword : System.getProperty("socks5Proxy.password");
}
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/conf/ClientConfigurationDataTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/conf/ClientConfigurationDataTest.java
new file mode 100644
index 00000000000..b5c30c9a7c6
--- /dev/null
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/conf/ClientConfigurationDataTest.java
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.client.impl.conf;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectWriter;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import org.apache.pulsar.client.impl.auth.AuthenticationToken;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+/**
+ * Unit test {@link ClientConfigurationData}.
+ */
+public class ClientConfigurationDataTest {
+
+ private final ObjectWriter w;
+
+ {
+ ObjectMapper m = new ObjectMapper();
+ m.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false);
+ w = m.writer();
+ }
+
+
+ @Test
+ public void testDoNotPrintSensitiveInfo() throws JsonProcessingException {
+ ClientConfigurationData clientConfigurationData = new ClientConfigurationData();
+ clientConfigurationData.setTlsTrustStorePassword("xxxx");
+ clientConfigurationData.setSocks5ProxyPassword("yyyy");
+ clientConfigurationData.setAuthentication(new AuthenticationToken("zzzz"));
+ String s = w.writeValueAsString(clientConfigurationData);
+ Assert.assertFalse(s.contains("Password"));
+ Assert.assertFalse(s.contains("xxxx"));
+ Assert.assertFalse(s.contains("yyyy"));
+ Assert.assertFalse(s.contains("zzzz"));
+ }
+
+}
[pulsar] 26/31: [PIP-153][optimize][txn] Optimize metadataPositions in MLPendingAckStore (#15137)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit f10f4456574ef0c03c6caf542ecbca1a24998ec2
Author: Xiangying Meng <55...@users.noreply.github.com>
AuthorDate: Mon May 16 09:48:57 2022 +0800
[PIP-153][optimize][txn] Optimize metadataPositions in MLPendingAckStore (#15137)
Master Issue: https://github.com/apache/pulsar/issues/15073
Reduce the memory occupied by metadataPositions and avoid OOM
Regularly store a small amount of data according to certain rules, the detailed implementation can be found in [PIP153](https://github.com/apache/pulsar/issues/15073)
(cherry picked from commit ebca19b522fd9f4496689ca7d32ede345d28511a)
---
conf/broker.conf | 6 +
.../apache/pulsar/broker/ServiceConfiguration.java | 9 ++
.../pendingack/impl/MLPendingAckStore.java | 170 +++++++++++----------
.../pendingack/impl/MLPendingAckStoreProvider.java | 11 +-
.../transaction/util/LogIndexLagBackoff.java | 49 ++++++
.../broker/transaction/util/package-info.java | 22 +++
.../pulsar/broker/transaction/TransactionTest.java | 4 +-
.../pendingack/PendingAckMetadataTest.java | 2 +-
.../pendingack/PendingAckPersistentTest.java | 105 +++++++++++++
.../pulsar/utils/LogIndexLagBackOffTest.java | 55 +++++++
10 files changed, 344 insertions(+), 89 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index cd6ccb42e6c..779a70ddef2 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -1382,6 +1382,12 @@ transactionBufferSnapshotMinTimeInMillis=5000
# The max concurrent requests for transaction buffer client, default is 1000
transactionBufferClientMaxConcurrentRequests=1000
+# MLPendingAckStore maintains a ConcurrentSkipListMap pendingAckLogIndex,
+# It stores the position in pendingAckStore as its value and saves a position used to determine
+# whether the previous data can be cleaned up as a key.
+# transactionPendingAckLogIndexMinLag is used to configure the minimum lag between indexes
+transactionPendingAckLogIndexMinLag=500
+
### --- Packages management service configuration variables (begin) --- ###
# Enable the packages management service or not
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index c7a5d440b8b..0c3f227d0cc 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -2483,6 +2483,15 @@ public class ServiceConfiguration implements PulsarConfiguration {
)
private long transactionBufferClientOperationTimeoutInMills = 3000L;
+ @FieldContext(
+ category = CATEGORY_TRANSACTION,
+ doc = "MLPendingAckStore maintain a ConcurrentSkipListMap pendingAckLogIndex`,"
+ + "it store the position in pendingAckStore as value and save a position used to determine"
+ + "whether the previous data can be cleaned up as a key."
+ + "transactionPendingAckLogIndexMinLag is used to configure the minimum lag between indexes"
+ )
+ private long transactionPendingAckLogIndexMinLag = 500L;
+
/**** --- KeyStore TLS config variables. --- ****/
@FieldContext(
category = CATEGORY_KEYSTORE_TLS,
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java
index ee5cc22f340..4ac0cd9b82c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java
@@ -42,6 +42,7 @@ import org.apache.pulsar.broker.transaction.pendingack.PendingAckStore;
import org.apache.pulsar.broker.transaction.pendingack.proto.PendingAckMetadata;
import org.apache.pulsar.broker.transaction.pendingack.proto.PendingAckMetadataEntry;
import org.apache.pulsar.broker.transaction.pendingack.proto.PendingAckOp;
+import org.apache.pulsar.broker.transaction.util.LogIndexLagBackoff;
import org.apache.pulsar.client.api.transaction.TxnID;
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.api.proto.CommandAck.AckType;
@@ -72,33 +73,40 @@ public class MLPendingAckStore implements PendingAckStore {
private PositionImpl currentLoadPosition;
+ private final AtomicLong currentIndexLag = new AtomicLong(0);
+ private volatile long maxIndexLag;
+
+ protected PositionImpl maxAckPosition = PositionImpl.EARLIEST;
+ private final LogIndexLagBackoff logIndexBackoff;
+
/**
* The map is for pending ack store clear useless data.
* <p>
- * When ack message append to pending ack store, it will store the position which is persistent as key.
+ * key:the largest ack position of origin topic, corresponds to the value position.
* <p>
- * When ack message append to pending ack store, it will store the position which is the max position of this
- * ack by the original topic as value.
+ * value:the position persistent by pendingAck log.
* <p>
- * It will judge the position with the max sub cursor position whether smaller than the subCursor mark
+ * It will judge the position with the max sub cursor position (key) whether smaller than the subCursor mark
* delete position.
* <p>
- * If the max position is smaller than the subCursor mark delete position, the log cursor will mark delete
- * the position.
+ * If the max position (key) is smaller than the subCursor mark delete position,
+ * the log cursor will mark delete the position before log position (value).
*/
- private final ConcurrentSkipListMap<PositionImpl, PositionImpl> metadataPositions;
+ private final ConcurrentSkipListMap<PositionImpl, PositionImpl> pendingAckLogIndex;
private final ManagedCursor subManagedCursor;
public MLPendingAckStore(ManagedLedger managedLedger, ManagedCursor cursor,
- ManagedCursor subManagedCursor) {
+ ManagedCursor subManagedCursor, long transactionPendingAckLogIndexMinLag) {
this.managedLedger = managedLedger;
this.cursor = cursor;
this.currentLoadPosition = (PositionImpl) this.cursor.getMarkDeletedPosition();
this.entryQueue = new SpscArrayQueue<>(2000);
this.lastConfirmedEntry = (PositionImpl) managedLedger.getLastConfirmedEntry();
- this.metadataPositions = new ConcurrentSkipListMap<>();
+ this.pendingAckLogIndex = new ConcurrentSkipListMap<>();
this.subManagedCursor = subManagedCursor;
+ this.logIndexBackoff = new LogIndexLagBackoff(transactionPendingAckLogIndexMinLag, Long.MAX_VALUE, 1);
+ this.maxIndexLag = logIndexBackoff.next(0);
}
@Override
@@ -219,62 +227,12 @@ public class MLPendingAckStore implements PendingAckStore {
log.debug("[{}][{}] MLPendingAckStore message append success at {} txnId: {}, operation : {}",
managedLedger.getName(), ctx, position, txnID, pendingAckMetadataEntry.getPendingAckOp());
}
- // store the persistent position in to memory
- if (pendingAckMetadataEntry.getPendingAckOp() != PendingAckOp.ABORT
- && pendingAckMetadataEntry.getPendingAckOp() != PendingAckOp.COMMIT) {
- Optional<PendingAckMetadata> optional = pendingAckMetadataEntry.getPendingAckMetadatasList()
- .stream().max((o1, o2) -> ComparisonChain.start().compare(o1.getLedgerId(),
- o2.getLedgerId()).compare(o1.getEntryId(), o2.getEntryId()).result());
- optional.ifPresent(pendingAckMetadata ->
- metadataPositions.compute((PositionImpl) position, (thisPosition, otherPosition) -> {
- PositionImpl nowPosition = PositionImpl.get(pendingAckMetadata.getLedgerId(),
- pendingAckMetadata.getEntryId());
- if (otherPosition == null) {
- return nowPosition;
- } else {
- return nowPosition.compareTo(otherPosition) > 0 ? nowPosition : otherPosition;
- }
- }));
- }
-
+ currentIndexLag.incrementAndGet();
+ handleMetadataEntry((PositionImpl) position, pendingAckMetadataEntry);
buf.release();
completableFuture.complete(null);
- if (!metadataPositions.isEmpty()) {
- PositionImpl deletePosition = null;
- while (!metadataPositions.isEmpty()
- && metadataPositions.firstKey() != null
- && subManagedCursor.getPersistentMarkDeletedPosition() != null
- && metadataPositions.firstEntry().getValue()
- .compareTo((PositionImpl) subManagedCursor.getPersistentMarkDeletedPosition()) <= 0) {
- deletePosition = metadataPositions.firstKey();
- metadataPositions.remove(metadataPositions.firstKey());
- }
-
- if (deletePosition != null) {
- PositionImpl finalDeletePosition = deletePosition;
- cursor.asyncMarkDelete(deletePosition,
- new AsyncCallbacks.MarkDeleteCallback() {
- @Override
- public void markDeleteComplete(Object ctx) {
- if (log.isDebugEnabled()) {
- log.debug("[{}] Transaction pending ack store mark delete position : "
- + "[{}] success", managedLedger.getName(),
- finalDeletePosition);
- }
- }
-
- @Override
- public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {
- if (log.isDebugEnabled()) {
- log.error("[{}] Transaction pending ack store mark delete position : "
- + "[{}] fail!", managedLedger.getName(),
- finalDeletePosition, exception);
- }
- }
- }, null);
- }
- }
+ clearUselessLogData();
}
@Override
@@ -292,6 +250,68 @@ public class MLPendingAckStore implements PendingAckStore {
return completableFuture;
}
+ private void handleMetadataEntry(PositionImpl logPosition, PendingAckMetadataEntry pendingAckMetadataEntry) {
+ // store the persistent position in to memory
+ // store the max position of this entry retain
+ if (pendingAckMetadataEntry.getPendingAckOp() != PendingAckOp.ABORT
+ && pendingAckMetadataEntry.getPendingAckOp() != PendingAckOp.COMMIT) {
+ Optional<PendingAckMetadata> optional = pendingAckMetadataEntry.getPendingAckMetadatasList()
+ .stream().max((o1, o2) -> ComparisonChain.start().compare(o1.getLedgerId(),
+ o2.getLedgerId()).compare(o1.getEntryId(), o2.getEntryId()).result());
+
+ optional.ifPresent(pendingAckMetadata -> {
+ PositionImpl nowPosition = PositionImpl.get(pendingAckMetadata.getLedgerId(),
+ pendingAckMetadata.getEntryId());
+
+ if (nowPosition.compareTo(maxAckPosition) > 0) {
+ maxAckPosition = nowPosition;
+ }
+ if (currentIndexLag.get() >= maxIndexLag) {
+ pendingAckLogIndex.compute(maxAckPosition,
+ (thisPosition, otherPosition) -> logPosition);
+ maxIndexLag = logIndexBackoff.next(pendingAckLogIndex.size());
+ currentIndexLag.set(0);
+ }
+ });
+ }
+ }
+
+ private void clearUselessLogData() {
+ if (!pendingAckLogIndex.isEmpty()) {
+ PositionImpl deletePosition = null;
+ while (!pendingAckLogIndex.isEmpty()
+ && pendingAckLogIndex.firstKey() != null
+ && subManagedCursor.getPersistentMarkDeletedPosition() != null
+ && pendingAckLogIndex.firstEntry().getKey()
+ .compareTo((PositionImpl) subManagedCursor.getPersistentMarkDeletedPosition()) <= 0) {
+ deletePosition = pendingAckLogIndex.remove(pendingAckLogIndex.firstKey());
+ }
+
+ if (deletePosition != null) {
+ maxIndexLag = logIndexBackoff.next(pendingAckLogIndex.size());
+ PositionImpl finalDeletePosition = deletePosition;
+ cursor.asyncMarkDelete(deletePosition,
+ new AsyncCallbacks.MarkDeleteCallback() {
+ @Override
+ public void markDeleteComplete(Object ctx) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Transaction pending ack store mark delete position : "
+ + "[{}] success", managedLedger.getName(),
+ finalDeletePosition);
+ }
+ }
+
+ @Override
+ public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Transaction pending ack store mark delete position : "
+ + "[{}] fail!", managedLedger.getName(),
+ finalDeletePosition, exception);
+ }
+ }, null);
+ }
+ }
+ }
+
class PendingAckReplay implements Runnable {
private final FillEntryQueueCallback fillEntryQueueCallback;
@@ -319,30 +339,12 @@ public class MLPendingAckStore implements PendingAckStore {
currentLoadPosition = PositionImpl.get(entry.getLedgerId(), entry.getEntryId());
PendingAckMetadataEntry pendingAckMetadataEntry = new PendingAckMetadataEntry();
pendingAckMetadataEntry.parseFrom(buffer, buffer.readableBytes());
- // store the persistent position in to memory
- // store the max position of this entry retain
- if (pendingAckMetadataEntry.getPendingAckOp() != PendingAckOp.ABORT
- && pendingAckMetadataEntry.getPendingAckOp() != PendingAckOp.COMMIT) {
- Optional<PendingAckMetadata> optional = pendingAckMetadataEntry.getPendingAckMetadatasList()
- .stream().max((o1, o2) -> ComparisonChain.start().compare(o1.getLedgerId(),
- o2.getLedgerId()).compare(o1.getEntryId(), o2.getEntryId()).result());
-
- optional.ifPresent(pendingAckMetadata ->
- metadataPositions.compute(PositionImpl.get(entry.getLedgerId(), entry.getEntryId()),
- (thisPosition, otherPosition) -> {
- PositionImpl nowPosition = PositionImpl
- .get(pendingAckMetadata.getLedgerId(),
- pendingAckMetadata.getEntryId());
- if (otherPosition == null) {
- return nowPosition;
- } else {
- return nowPosition.compareTo(otherPosition) > 0 ? nowPosition
- : otherPosition;
- }
- }));
- }
+ currentIndexLag.incrementAndGet();
+ handleMetadataEntry(new PositionImpl(entry.getLedgerId(), entry.getEntryId()),
+ pendingAckMetadataEntry);
pendingAckReplyCallBack.handleMetadataEntry(pendingAckMetadataEntry);
entry.release();
+ clearUselessLogData();
} else {
try {
Thread.sleep(1);
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java
index 5417caec3d7..6b84d6e329a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java
@@ -74,9 +74,14 @@ public class MLPendingAckStoreProvider implements TransactionPendingAckStoreProv
InitialPosition.Earliest, new AsyncCallbacks.OpenCursorCallback() {
@Override
public void openCursorComplete(ManagedCursor cursor, Object ctx) {
- pendingAckStoreFuture
- .complete(new MLPendingAckStore(ledger, cursor,
- subscription.getCursor()));
+ pendingAckStoreFuture.complete(new MLPendingAckStore(ledger,
+ cursor,
+ subscription.getCursor(),
+ originPersistentTopic
+ .getBrokerService()
+ .getPulsar()
+ .getConfiguration()
+ .getTransactionPendingAckLogIndexMinLag()));
if (log.isDebugEnabled()) {
log.debug("{},{} open MLPendingAckStore cursor success",
originPersistentTopic.getName(),
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/LogIndexLagBackoff.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/LogIndexLagBackoff.java
new file mode 100644
index 00000000000..145381814ba
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/LogIndexLagBackoff.java
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.util;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import lombok.Getter;
+
+public class LogIndexLagBackoff {
+
+ @Getter
+ private final long minLag;
+ @Getter
+ private final long maxLag;
+ @Getter
+ private final double exponent;
+
+ public LogIndexLagBackoff(long minLag, long maxLag, double exponent) {
+ checkArgument(minLag > 0, "min lag must be > 0");
+ checkArgument(maxLag >= minLag, "maxLag should be >= minLag");
+ checkArgument(exponent > 0, "exponent must be > 0");
+ this.minLag = minLag;
+ this.maxLag = maxLag;
+ this.exponent = exponent;
+ }
+
+
+ public long next(int indexCount) {
+ if (indexCount <= 0) {
+ return minLag;
+ }
+ return (long) Math.min(this.maxLag, minLag * Math.pow(indexCount, exponent));
+ }
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/package-info.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/package-info.java
new file mode 100644
index 00000000000..58cb1c24b19
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util/package-info.java
@@ -0,0 +1,22 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/**
+ * Implementation of a transaction tools.
+ */
+package org.apache.pulsar.broker.transaction.util;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
index 7ab49481466..f78bfde7406 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java
@@ -616,7 +616,7 @@ public class TransactionTest extends TransactionTestBase {
TransactionPendingAckStoreProvider pendingAckStoreProvider = mock(TransactionPendingAckStoreProvider.class);
doReturn(CompletableFuture.completedFuture(
- new MLPendingAckStore(persistentTopic.getManagedLedger(), managedCursor, null)))
+ new MLPendingAckStore(persistentTopic.getManagedLedger(), managedCursor, null, 500)))
.when(pendingAckStoreProvider).newPendingAckStore(any());
doReturn(CompletableFuture.completedFuture(true)).when(pendingAckStoreProvider).checkInitializedBefore(any());
@@ -897,6 +897,8 @@ public class TransactionTest extends TransactionTestBase {
@Test
public void testPendingAckMarkDeletePosition() throws Exception {
+ getPulsarServiceList().get(0).getConfig().setTransactionPendingAckLogIndexMinLag(1);
+ getPulsarServiceList().get(0).getConfiguration().setManagedLedgerDefaultMarkDeleteRateLimit(5);
String topic = NAMESPACE1 + "/test1";
@Cleanup
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckMetadataTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckMetadataTest.java
index c99eee62463..14dbcdb8897 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckMetadataTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckMetadataTest.java
@@ -72,7 +72,7 @@ public class PendingAckMetadataTest extends MockedBookKeeperTestCase {
ManagedCursor cursor = completableFuture.get().openCursor("test");
ManagedCursor subCursor = completableFuture.get().openCursor("test");
MLPendingAckStore pendingAckStore =
- new MLPendingAckStore(completableFuture.get(), cursor, subCursor);
+ new MLPendingAckStore(completableFuture.get(), cursor, subCursor, 500);
Field field = MLPendingAckStore.class.getDeclaredField("managedLedger");
field.setAccessible(true);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckPersistentTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckPersistentTest.java
index 7bc562ca62b..96b2b9c14bf 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckPersistentTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/PendingAckPersistentTest.java
@@ -28,6 +28,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentSkipListMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import lombok.Cleanup;
@@ -47,6 +48,7 @@ import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.SubscriptionType;
import org.apache.pulsar.client.api.transaction.Transaction;
+import org.apache.pulsar.client.impl.MessageIdImpl;
import org.apache.pulsar.client.api.transaction.TxnID;
import org.apache.pulsar.client.impl.transaction.TransactionImpl;
import org.apache.pulsar.common.naming.NamespaceName;
@@ -54,6 +56,7 @@ import org.apache.pulsar.common.naming.TopicDomain;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.policies.data.TopicStats;
import org.awaitility.Awaitility;
+import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
@@ -197,6 +200,8 @@ public class PendingAckPersistentTest extends TransactionTestBase {
@Test
public void cumulativePendingAckReplayTest() throws Exception {
int messageCount = 1000;
+ getPulsarServiceList().get(0).getConfig().setTransactionPendingAckLogIndexMinLag(4 * messageCount + 2);
+ getPulsarServiceList().get(0).getConfiguration().setManagedLedgerDefaultMarkDeleteRateLimit(10);
String subName = "cumulative-test";
@Cleanup
@@ -351,6 +356,106 @@ public class PendingAckPersistentTest extends TransactionTestBase {
assertFalse(topics.contains(topic));
}
+ @Test
+ public void testDeleteUselessLogDataWhenSubCursorMoved() throws Exception {
+ getPulsarServiceList().get(0).getConfig().setTransactionPendingAckLogIndexMinLag(5);
+ getPulsarServiceList().get(0).getConfiguration().setManagedLedgerDefaultMarkDeleteRateLimit(5);
+ String subName = "test-log-delete";
+ String topic = TopicName.get(TopicDomain.persistent.toString(),
+ NamespaceName.get(NAMESPACE1), "test-log-delete").toString();
+
+ @Cleanup
+ Consumer<byte[]> consumer = pulsarClient.newConsumer()
+ .topic(topic)
+ .subscriptionName(subName)
+ .subscribe();
+ @Cleanup
+ Producer<byte[]> producer = pulsarClient.newProducer()
+ .topic(topic)
+ .sendTimeout(0, TimeUnit.SECONDS)
+ .enableBatching(false)
+ .create();
+
+ for (int i = 0; i < 20; i++) {
+ producer.newMessage().send();
+ }
+ // init
+ Message<byte[]> message = consumer.receive(5, TimeUnit.SECONDS);
+ Transaction transaction = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.SECONDS)
+ .build()
+ .get();
+ consumer.acknowledgeAsync(message.getMessageId(), transaction).get();
+
+ PersistentTopic persistentTopic = (PersistentTopic) getPulsarServiceList().get(0)
+ .getBrokerService().getTopic(topic, false).get().get();
+
+ PersistentSubscription persistentSubscription = persistentTopic.getSubscription(subName);
+ Field field = PersistentSubscription.class.getDeclaredField("pendingAckHandle");
+ field.setAccessible(true);
+ PendingAckHandleImpl pendingAckHandle = (PendingAckHandleImpl) field.get(persistentSubscription);
+ Field field1 = PendingAckHandleImpl.class.getDeclaredField("pendingAckStoreFuture");
+ field1.setAccessible(true);
+ PendingAckStore pendingAckStore = ((CompletableFuture<PendingAckStore>) field1.get(pendingAckHandle)).get();
+
+ Field field3 = MLPendingAckStore.class.getDeclaredField("pendingAckLogIndex");
+ Field field4 = MLPendingAckStore.class.getDeclaredField("maxIndexLag");
+
+ field3.setAccessible(true);
+ field4.setAccessible(true);
+
+ ConcurrentSkipListMap<PositionImpl, PositionImpl> pendingAckLogIndex =
+ (ConcurrentSkipListMap<PositionImpl, PositionImpl>) field3.get(pendingAckStore);
+ long maxIndexLag = (long) field4.get(pendingAckStore);
+ Assert.assertEquals(pendingAckLogIndex.size(), 0);
+ Assert.assertEquals(maxIndexLag, 5);
+ transaction.commit().get();
+
+ Awaitility.await().untilAsserted(() ->
+ Assert.assertEquals(persistentSubscription.getCursor().getPersistentMarkDeletedPosition().getEntryId(),
+ ((MessageIdImpl)message.getMessageId()).getEntryId()));
+ // 7 more acks. Will find that there are still only two records in the map.
+ Transaction transaction1 = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.SECONDS)
+ .build()
+ .get();
+ Message<byte[]> message0 = null;
+ //remove previous index
+ for (int i = 0; i < 4; i++) {
+ message0 = consumer.receive(5, TimeUnit.SECONDS);
+ consumer.acknowledgeAsync(message0.getMessageId(), transaction1).get();
+ }
+ Assert.assertEquals(pendingAckLogIndex.size(), 1);
+ maxIndexLag = (long) field4.get(pendingAckStore);
+ Assert.assertEquals(maxIndexLag, 5);
+ //add new index
+ for (int i = 0; i < 9; i++) {
+ message0= consumer.receive(5, TimeUnit.SECONDS);
+ consumer.acknowledgeAsync(message0.getMessageId(), transaction1).get();
+ }
+
+ Assert.assertEquals(pendingAckLogIndex.size(), 2);
+ maxIndexLag = (long) field4.get(pendingAckStore);
+ Assert.assertEquals(maxIndexLag, 10);
+
+ transaction1.commit().get();
+ Message<byte[]> message1 = message0;
+ Awaitility.await().untilAsserted(() ->
+ Assert.assertEquals(persistentSubscription.getCursor().getPersistentMarkDeletedPosition().getEntryId(),
+ ((MessageIdImpl)message1.getMessageId()).getEntryId()));
+
+ Transaction transaction2 = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.SECONDS)
+ .build()
+ .get();
+ Message<byte[]> message2 = consumer.receive(5, TimeUnit.SECONDS);
+ consumer.acknowledgeAsync(message2.getMessageId(), transaction2).get();
+
+ Assert.assertEquals(pendingAckLogIndex.size(), 0);
+ maxIndexLag = (long) field4.get(pendingAckStore);
+ Assert.assertEquals(maxIndexLag, 5);
+ }
+
@Test
public void testPendingAckLowWaterMarkRemoveFirstTxn() throws Exception {
String topic = TopicName.get(TopicDomain.persistent.toString(),
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/utils/LogIndexLagBackOffTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/utils/LogIndexLagBackOffTest.java
new file mode 100644
index 00000000000..8d4f2c356a6
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/utils/LogIndexLagBackOffTest.java
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.utils;
+
+import org.apache.pulsar.broker.transaction.util.LogIndexLagBackoff;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+@Test(groups = "utils")
+public class LogIndexLagBackOffTest {
+ @Test
+ public void testGenerateNextLogIndexLag() {
+ LogIndexLagBackoff logIndexLagBackoff = new LogIndexLagBackoff(1, 10, 1);
+ Assert.assertEquals(logIndexLagBackoff.next(0), 1);
+ Assert.assertEquals(logIndexLagBackoff.next(6), 6);
+
+ Assert.assertEquals(logIndexLagBackoff.next(77), 10);
+
+ logIndexLagBackoff = new LogIndexLagBackoff(1, 10, 2);
+ Assert.assertEquals(logIndexLagBackoff.next(3), 9);
+
+ try {
+ new LogIndexLagBackoff(-1, 2, 3);
+ } catch (IllegalArgumentException e) {
+ Assert.assertEquals(e.getMessage(), "min lag must be > 0");
+ }
+ try {
+ new LogIndexLagBackoff(2, 1, 3);
+ } catch (IllegalArgumentException e) {
+ Assert.assertEquals(e.getMessage(), "maxLag should be >= minLag");
+ }
+ try {
+ new LogIndexLagBackoff(1, 1, 0.2);
+ } catch (IllegalArgumentException e) {
+ Assert.assertEquals(e.getMessage(), "exponent must be > 0");
+ }
+
+ }
+}
[pulsar] 25/31: [fix][broker] Fix to avoid TopicStatsImpl NPE even if producerName is null (#15502)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit d23e251a9b3070d2c7716afb01170667957588ba
Author: Yuri Mizushima <yu...@yahoo-corp.jp>
AuthorDate: Sat May 14 14:40:46 2022 +0900
[fix][broker] Fix to avoid TopicStatsImpl NPE even if producerName is null (#15502)
(cherry picked from commit 7bdfa3a3e5e71c44a174b28d0a843fb6730865fa)
---
.../org/apache/pulsar/broker/service/Producer.java | 2 +-
.../data/stats/NonPersistentTopicStatsImpl.java | 23 +++++++------
.../common/policies/data/stats/TopicStatsImpl.java | 13 ++++---
.../NonPersistentPartitionedTopicStatsTest.java | 40 ++++++++++++++++++++++
.../policies/data/PersistentTopicStatsTest.java | 38 ++++++++++++++++++++
5 files changed, 101 insertions(+), 15 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
index 99d21db569b..f8591a8447a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
@@ -128,7 +128,7 @@ public class Producer {
stats.setClientVersion(cnx.getClientVersion());
stats.setProducerName(producerName);
stats.producerId = producerId;
- if (serviceConf.isAggregatePublisherStatsByProducerName()) {
+ if (serviceConf.isAggregatePublisherStatsByProducerName() && stats.getProducerName() != null) {
// If true and the client supports partial producer,
// aggregate publisher stats of PartitionedTopicStats by producerName.
// Otherwise, aggregate it by list index.
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/NonPersistentTopicStatsImpl.java b/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/NonPersistentTopicStatsImpl.java
index 35c665f881d..23e603ea028 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/NonPersistentTopicStatsImpl.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/NonPersistentTopicStatsImpl.java
@@ -18,6 +18,8 @@
*/
package org.apache.pulsar.common.policies.data.stats;
+import static java.util.Comparator.naturalOrder;
+import static java.util.Comparator.nullsLast;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
@@ -62,10 +64,10 @@ public class NonPersistentTopicStatsImpl extends TopicStatsImpl implements NonPe
@JsonProperty("publishers")
public List<NonPersistentPublisherStats> getNonPersistentPublishers() {
- return Stream.concat(nonPersistentPublishers.stream()
- .sorted(Comparator.comparing(NonPersistentPublisherStats::getProducerName)),
- nonPersistentPublishersMap.values().stream()
- .sorted(Comparator.comparing(NonPersistentPublisherStats::getProducerName)))
+ return Stream.concat(nonPersistentPublishers.stream().sorted(
+ Comparator.comparing(NonPersistentPublisherStats::getProducerName, nullsLast(naturalOrder()))),
+ nonPersistentPublishersMap.values().stream().sorted(
+ Comparator.comparing(NonPersistentPublisherStats::getProducerName, nullsLast(naturalOrder()))))
.collect(Collectors.toList());
}
@@ -92,10 +94,10 @@ public class NonPersistentTopicStatsImpl extends TopicStatsImpl implements NonPe
@SuppressFBWarnings(value = "MF_CLASS_MASKS_FIELD", justification = "expected to override")
public List<NonPersistentPublisherStats> getPublishers() {
- return Stream.concat(nonPersistentPublishers.stream()
- .sorted(Comparator.comparing(NonPersistentPublisherStats::getProducerName)),
- nonPersistentPublishersMap.values()
- .stream().sorted(Comparator.comparing(NonPersistentPublisherStats::getProducerName)))
+ return Stream.concat(nonPersistentPublishers.stream().sorted(
+ Comparator.comparing(NonPersistentPublisherStats::getProducerName, nullsLast(naturalOrder()))),
+ nonPersistentPublishersMap.values().stream().sorted(
+ Comparator.comparing(NonPersistentPublisherStats::getProducerName, nullsLast(naturalOrder()))))
.collect(Collectors.toList());
}
@@ -106,9 +108,10 @@ public class NonPersistentTopicStatsImpl extends TopicStatsImpl implements NonPe
}
public void addPublisher(NonPersistentPublisherStatsImpl stats) {
- if (stats.isSupportsPartialProducer()) {
+ if (stats.isSupportsPartialProducer() && stats.getProducerName() != null) {
nonPersistentPublishersMap.put(stats.getProducerName(), stats);
} else {
+ stats.setSupportsPartialProducer(false); // setter method with side effect
nonPersistentPublishers.add(stats);
}
}
@@ -153,7 +156,7 @@ public class NonPersistentTopicStatsImpl extends TopicStatsImpl implements NonPe
this.msgDropRate += stats.msgDropRate;
stats.getNonPersistentPublishers().forEach(s -> {
- if (s.isSupportsPartialProducer()) {
+ if (s.isSupportsPartialProducer() && s.getProducerName() != null) {
((NonPersistentPublisherStatsImpl) this.nonPersistentPublishersMap
.computeIfAbsent(s.getProducerName(), key -> {
final NonPersistentPublisherStatsImpl newStats = new NonPersistentPublisherStatsImpl();
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/TopicStatsImpl.java b/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/TopicStatsImpl.java
index 18e7d60054f..3e90ca9be94 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/TopicStatsImpl.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/stats/TopicStatsImpl.java
@@ -18,6 +18,8 @@
*/
package org.apache.pulsar.common.policies.data.stats;
+import static java.util.Comparator.naturalOrder;
+import static java.util.Comparator.nullsLast;
import com.fasterxml.jackson.annotation.JsonIgnore;
import java.util.ArrayList;
import java.util.Comparator;
@@ -133,8 +135,10 @@ public class TopicStatsImpl implements TopicStats {
public CompactionStatsImpl compaction;
public List<? extends PublisherStats> getPublishers() {
- return Stream.concat(publishers.stream().sorted(Comparator.comparing(PublisherStatsImpl::getProducerName)),
- publishersMap.values().stream().sorted(Comparator.comparing(PublisherStatsImpl::getProducerName)))
+ return Stream.concat(publishers.stream().sorted(
+ Comparator.comparing(PublisherStatsImpl::getProducerName, nullsLast(naturalOrder()))),
+ publishersMap.values().stream().sorted(
+ Comparator.comparing(PublisherStatsImpl::getProducerName, nullsLast(naturalOrder()))))
.collect(Collectors.toList());
}
@@ -145,9 +149,10 @@ public class TopicStatsImpl implements TopicStats {
}
public void addPublisher(PublisherStatsImpl stats) {
- if (stats.isSupportsPartialProducer()) {
+ if (stats.isSupportsPartialProducer() && stats.getProducerName() != null) {
publishersMap.put(stats.getProducerName(), stats);
} else {
+ stats.setSupportsPartialProducer(false); // setter method with side effect
publishers.add(stats);
}
}
@@ -223,7 +228,7 @@ public class TopicStatsImpl implements TopicStats {
this.nonContiguousDeletedMessagesRangesSerializedSize += stats.nonContiguousDeletedMessagesRangesSerializedSize;
stats.getPublishers().forEach(s -> {
- if (s.isSupportsPartialProducer()) {
+ if (s.isSupportsPartialProducer() && s.getProducerName() != null) {
this.publishersMap.computeIfAbsent(s.getProducerName(), key -> {
final PublisherStatsImpl newStats = new PublisherStatsImpl();
newStats.setSupportsPartialProducer(true);
diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/NonPersistentPartitionedTopicStatsTest.java b/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/NonPersistentPartitionedTopicStatsTest.java
index 311ed432524..001eaec901f 100644
--- a/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/NonPersistentPartitionedTopicStatsTest.java
+++ b/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/NonPersistentPartitionedTopicStatsTest.java
@@ -22,9 +22,11 @@ import org.apache.pulsar.common.policies.data.stats.NonPersistentPartitionedTopi
import org.apache.pulsar.common.policies.data.stats.NonPersistentPublisherStatsImpl;
import org.apache.pulsar.common.policies.data.stats.NonPersistentReplicatorStatsImpl;
import org.apache.pulsar.common.policies.data.stats.NonPersistentSubscriptionStatsImpl;
+import org.apache.pulsar.common.policies.data.stats.NonPersistentTopicStatsImpl;
import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
public class NonPersistentPartitionedTopicStatsTest {
@@ -55,4 +57,42 @@ public class NonPersistentPartitionedTopicStatsTest {
assertEquals(nonPersistentPartitionedTopicStats.metadata.partitions, 0);
assertEquals(nonPersistentPartitionedTopicStats.partitions.size(), 0);
}
+
+ @Test
+ public void testPartitionedTopicStatsByNullProducerName() {
+ final NonPersistentTopicStatsImpl topicStats1 = new NonPersistentTopicStatsImpl();
+ final NonPersistentPublisherStatsImpl publisherStats1 = new NonPersistentPublisherStatsImpl();
+ publisherStats1.setSupportsPartialProducer(false);
+ publisherStats1.setProducerName(null);
+ final NonPersistentPublisherStatsImpl publisherStats2 = new NonPersistentPublisherStatsImpl();
+ publisherStats2.setSupportsPartialProducer(false);
+ publisherStats2.setProducerName(null);
+ topicStats1.addPublisher(publisherStats1);
+ topicStats1.addPublisher(publisherStats2);
+
+ assertEquals(topicStats1.getPublishers().size(), 2);
+ assertFalse(topicStats1.getPublishers().get(0).isSupportsPartialProducer());
+ assertFalse(topicStats1.getPublishers().get(1).isSupportsPartialProducer());
+
+ final NonPersistentTopicStatsImpl topicStats2 = new NonPersistentTopicStatsImpl();
+ final NonPersistentPublisherStatsImpl publisherStats3 = new NonPersistentPublisherStatsImpl();
+ publisherStats3.setSupportsPartialProducer(true);
+ publisherStats3.setProducerName(null);
+ final NonPersistentPublisherStatsImpl publisherStats4 = new NonPersistentPublisherStatsImpl();
+ publisherStats4.setSupportsPartialProducer(true);
+ publisherStats4.setProducerName(null);
+ topicStats2.addPublisher(publisherStats3);
+ topicStats2.addPublisher(publisherStats4);
+
+ assertEquals(topicStats2.getPublishers().size(), 2);
+ // when the producerName is null, fall back to false
+ assertFalse(topicStats2.getPublishers().get(0).isSupportsPartialProducer());
+ assertFalse(topicStats2.getPublishers().get(1).isSupportsPartialProducer());
+
+ final NonPersistentPartitionedTopicStatsImpl target = new NonPersistentPartitionedTopicStatsImpl();
+ target.add(topicStats1);
+ target.add(topicStats2);
+
+ assertEquals(target.getPublishers().size(), 2);
+ }
}
diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/PersistentTopicStatsTest.java b/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/PersistentTopicStatsTest.java
index 0eb1997ffd2..0da2b9f8abb 100644
--- a/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/PersistentTopicStatsTest.java
+++ b/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/PersistentTopicStatsTest.java
@@ -19,6 +19,7 @@
package org.apache.pulsar.common.policies.data;
import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
import org.apache.pulsar.common.policies.data.stats.PublisherStatsImpl;
import org.apache.pulsar.common.policies.data.stats.ReplicatorStatsImpl;
@@ -235,4 +236,41 @@ public class PersistentTopicStatsTest {
assertEquals(target.replication.size(), 1);
}
+ @Test
+ public void testPersistentTopicStatsByNullProducerName() {
+ final TopicStatsImpl topicStats1 = new TopicStatsImpl();
+ final PublisherStatsImpl publisherStats1 = new PublisherStatsImpl();
+ publisherStats1.setSupportsPartialProducer(false);
+ publisherStats1.setProducerName(null);
+ final PublisherStatsImpl publisherStats2 = new PublisherStatsImpl();
+ publisherStats2.setSupportsPartialProducer(false);
+ publisherStats2.setProducerName(null);
+ topicStats1.addPublisher(publisherStats1);
+ topicStats1.addPublisher(publisherStats2);
+
+ assertEquals(topicStats1.getPublishers().size(), 2);
+ assertFalse(topicStats1.getPublishers().get(0).isSupportsPartialProducer());
+ assertFalse(topicStats1.getPublishers().get(1).isSupportsPartialProducer());
+
+ final TopicStatsImpl topicStats2 = new TopicStatsImpl();
+ final PublisherStatsImpl publisherStats3 = new PublisherStatsImpl();
+ publisherStats3.setSupportsPartialProducer(true);
+ publisherStats3.setProducerName(null);
+ final PublisherStatsImpl publisherStats4 = new PublisherStatsImpl();
+ publisherStats4.setSupportsPartialProducer(true);
+ publisherStats4.setProducerName(null);
+ topicStats2.addPublisher(publisherStats3);
+ topicStats2.addPublisher(publisherStats4);
+
+ assertEquals(topicStats2.getPublishers().size(), 2);
+ // when the producerName is null, fall back to false
+ assertFalse(topicStats2.getPublishers().get(0).isSupportsPartialProducer());
+ assertFalse(topicStats2.getPublishers().get(1).isSupportsPartialProducer());
+
+ final TopicStatsImpl target = new TopicStatsImpl();
+ target.add(topicStats1);
+ target.add(topicStats2);
+
+ assertEquals(target.getPublishers().size(), 2);
+ }
}
[pulsar] 04/31: Add KeyStore support in WebSocket, Function Worker HTTPS Servers (#15084)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e6b5ec9a503c6b13ed108e68a74e6fea97326013
Author: Michael Marshall <mm...@apache.org>
AuthorDate: Wed Apr 20 22:25:17 2022 -0500
Add KeyStore support in WebSocket, Function Worker HTTPS Servers (#15084)
* Add KeyStore support in WebSocket, Function Worker HTTPS Servers
* Avoid leaking worker config password
* Fix checkstyle
* Replace broker with appropriate text in annotations
* Update python script for new configs
Co-authored-by: Lari Hotari <lh...@apache.org>
We support configuring KeyStores for the broker and the proxy, but not the WebSocket or the Function Worker. By adding this support, users are able to provide KeyStores of type PCKS12 or JKS, which adds flexibility. Further, these KeyStores simplify support for additional algorithms because we rely on the TLS provider to load the KeyStore instead of loading keys ourselves.
* Add `KeyStoreSSLContext`s to the function worker server
* Add `KeyStoreSSLContext`s to the web socket server
* Add configurations to the function worker, the web socket, and the proxy configuration files to simply configuration
* Rely on `toString`, not `ObjectMapper`, when converting the `WorkerConfig` to a string so that we don't log the KeyStore password. (Add a test to verify this logic. Note that we don't want the `ObjectMapper` to ignore the field because we use mappers when converting configuration classes.)
I manually verified that this change works in a minikube cluster. The underlying method named `KeyStoreSSLContext#createSslContextFactory` is already used and tested, so I don't believe we need additional testing on that component.
This change adds a new way to configure TLS in the WebSocket and Function Worker HTTPS Servers. As such, it adds new configuration. This configuration is named in the same way that the broker and proxy configuration is named, so it is consistent.
I've documented the new configuration in the appropriate configuration files.
(cherry picked from commit a4103960a4eb7803af6fabd9fe54a3137f82aff0)
---
conf/broker.conf | 2 +
conf/functions_worker.yml | 40 ++++++++++++
conf/proxy.conf | 27 ++++++++
conf/websocket.conf | 37 +++++++++++
docker/pulsar/scripts/gen-yml-from-env.py | 4 +-
.../apache/pulsar/broker/ServiceConfiguration.java | 2 +-
.../pulsar/broker/stats/PrometheusMetricsTest.java | 7 +-
.../pulsar/functions/worker/WorkerConfig.java | 74 +++++++++++++++++++++-
.../worker/WorkerApiV2ResourceConfigTest.java | 18 +++++-
.../functions/worker/PulsarWorkerService.java | 9 +--
.../pulsar/functions/worker/rest/WorkerServer.java | 35 ++++++++--
.../pulsar/websocket/service/ProxyServer.java | 37 ++++++++---
.../service/WebSocketProxyConfiguration.java | 58 +++++++++++++++++
13 files changed, 318 insertions(+), 32 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index 6efa908813c..5c1ca1d8b75 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -624,6 +624,8 @@ tlsRequireTrustedClientCertOnConnect=false
tlsProvider=
### --- KeyStore TLS config variables --- ###
+## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
+
# Enable TLS with KeyStore type configuration in broker.
tlsEnabledWithKeyStore=false
diff --git a/conf/functions_worker.yml b/conf/functions_worker.yml
index 24d46e1b61a..4254d3fa8cb 100644
--- a/conf/functions_worker.yml
+++ b/conf/functions_worker.yml
@@ -317,6 +317,46 @@ tlsAllowInsecureConnection: false
tlsEnableHostnameVerification: false
# Tls cert refresh duration in seconds (set 0 to check on every new connection)
tlsCertRefreshCheckDurationSec: 300
+# Whether client certificates are required for TLS. Connections are rejected if the client
+# certificate isn't trusted.
+tlsRequireTrustedClientCertOnConnect: false
+
+### --- KeyStore TLS config variables --- ###
+## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
+
+# TLS Provider for KeyStore type
+tlsProvider:
+
+# Enable TLS with KeyStore type configuration in function worker.
+tlsEnabledWithKeyStore: false
+
+# TLS KeyStore type configuration in function worker: JKS, PKCS12
+tlsKeyStoreType: JKS
+
+# TLS KeyStore path in function worker
+tlsKeyStore:
+
+# TLS KeyStore password for function worker
+tlsKeyStorePassword:
+
+# TLS TrustStore type configuration in function worker: JKS, PKCS12
+tlsTrustStoreType: JKS
+
+# TLS TrustStore path in function worker
+tlsTrustStore:
+
+# TLS TrustStore password in function worker, default value is empty password
+tlsTrustStorePassword:
+
+# Specify the tls protocols the function worker's web service will use to negotiate during TLS handshake
+# (a comma-separated list of protocol names).
+# Examples:- [TLSv1.3, TLSv1.2]
+webServiceTlsProtocols:
+
+# Specify the tls cipher the function worker will use to negotiate during TLS Handshake
+# (a comma-separated list of ciphers).
+# Examples:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
+webServiceTlsCiphers:
########################
# State Management
diff --git a/conf/proxy.conf b/conf/proxy.conf
index e4cf8b565dd..5c12a645054 100644
--- a/conf/proxy.conf
+++ b/conf/proxy.conf
@@ -71,6 +71,33 @@ webServicePort=8080
# Port to use to server HTTPS request
webServicePortTls=
+### --- KeyStore TLS config variables --- ###
+## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
+
+# TLS Provider for KeyStore type
+tlsProvider=
+
+# Enable TLS with KeyStore type configuration in proxy.
+tlsEnabledWithKeyStore=false
+
+# TLS KeyStore type configuration in proxy: JKS, PKCS12
+tlsKeyStoreType=JKS
+
+# TLS KeyStore path in proxy
+tlsKeyStore=
+
+# TLS KeyStore password for proxy
+tlsKeyStorePassword=
+
+# TLS TrustStore type configuration in proxy: JKS, PKCS12
+tlsTrustStoreType=JKS
+
+# TLS TrustStore path in proxy
+tlsTrustStore=
+
+# TLS TrustStore password in proxy, default value is empty password
+tlsTrustStorePassword=
+
# Specify the tls protocols the proxy's web service will use to negotiate during TLS handshake
# (a comma-separated list of protocol names).
# Examples:- [TLSv1.3, TLSv1.2]
diff --git a/conf/websocket.conf b/conf/websocket.conf
index 7acbf01d83c..1c946ef4dba 100644
--- a/conf/websocket.conf
+++ b/conf/websocket.conf
@@ -120,6 +120,43 @@ tlsRequireTrustedClientCertOnConnect=false
# Tls cert refresh duration in seconds (set 0 to check on every new connection)
tlsCertRefreshCheckDurationSec=300
+### --- KeyStore TLS config variables --- ###
+## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
+
+# TLS Provider for KeyStore type
+tlsProvider=
+
+# Enable TLS with KeyStore type configuration in WebSocket.
+tlsEnabledWithKeyStore=false
+
+# TLS KeyStore type configuration in WebSocket: JKS, PKCS12
+tlsKeyStoreType=JKS
+
+# TLS KeyStore path in WebSocket
+tlsKeyStore=
+
+# TLS KeyStore password for WebSocket
+tlsKeyStorePassword=
+
+# TLS TrustStore type configuration in WebSocket: JKS, PKCS12
+tlsTrustStoreType=JKS
+
+# TLS TrustStore path in WebSocket
+tlsTrustStore=
+
+# TLS TrustStore password in WebSocket, default value is empty password
+tlsTrustStorePassword=
+
+# Specify the tls protocols the proxy's web service will use to negotiate during TLS handshake
+# (a comma-separated list of protocol names).
+# Examples:- [TLSv1.3, TLSv1.2]
+webServiceTlsProtocols=
+
+# Specify the tls cipher the proxy will use to negotiate during TLS Handshake
+# (a comma-separated list of ciphers).
+# Examples:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
+webServiceTlsCiphers=
+
### --- Deprecated config variables --- ###
# Deprecated. Use configurationStoreServers
diff --git a/docker/pulsar/scripts/gen-yml-from-env.py b/docker/pulsar/scripts/gen-yml-from-env.py
index f2e10312a3a..4534ea9c168 100755
--- a/docker/pulsar/scripts/gen-yml-from-env.py
+++ b/docker/pulsar/scripts/gen-yml-from-env.py
@@ -47,7 +47,9 @@ SET_KEYS = [
'proxyRoles',
'schemaRegistryCompatibilityCheckers',
'brokerClientTlsCiphers',
- 'brokerClientTlsProtocols'
+ 'brokerClientTlsProtocols',
+ 'webServiceTlsCiphers',
+ 'webServiceTlsProtocols',
]
PF_ENV_PREFIX = 'PF_'
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 61aca76c199..16bde537510 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -2486,7 +2486,7 @@ public class ServiceConfiguration implements PulsarConfiguration {
@FieldContext(
category = CATEGORY_KEYSTORE_TLS,
- doc = "TLS Provider for Specify the SSL provider for the broker service: \n"
+ doc = "Specify the TLS provider for the broker service: \n"
+ "When using TLS authentication with CACert, the valid value is either OPENSSL or JDK.\n"
+ "When using TLS authentication with KeyStore, available values can be SunJSSE, Conscrypt and etc."
)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 144a5c604c3..c2d46e2edb3 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -1460,10 +1460,9 @@ public class PrometheusMetricsTest extends BrokerTestBase {
return parsed;
}
-
- static class Metric {
- Map<String, String> tags = new TreeMap<>();
- double value;
+ public static class Metric {
+ public Map<String, String> tags = new TreeMap<>();
+ public double value;
@Override
public String toString() {
diff --git a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
index 7a367afaf0f..a55f49bfc20 100644
--- a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
+++ b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
@@ -38,9 +38,11 @@ import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
-
+import java.util.TreeSet;
import lombok.AccessLevel;
import lombok.Getter;
+import lombok.ToString;
+import lombok.experimental.Accessors;
import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider;
import org.apache.pulsar.common.configuration.Category;
@@ -78,6 +80,8 @@ public class WorkerConfig implements Serializable, PulsarConfiguration {
@Category
private static final String CATEGORY_SECURITY = "Common Security Settings (applied for both worker and client)";
@Category
+ private static final String CATEGORY_KEYSTORE_TLS = "KeyStoreTLS";
+ @Category
private static final String CATEGORY_WORKER_SECURITY = "Worker Security Settings";
@Category
private static final String CATEGORY_CLIENT_SECURITY = "Security settings for clients talking to brokers";
@@ -427,6 +431,74 @@ public class WorkerConfig implements Serializable, PulsarConfiguration {
doc = "Tls cert refresh duration in seconds (set 0 to check on every new connection)"
)
private long tlsCertRefreshCheckDurationSec = 300;
+
+ /**** --- KeyStore TLS config variables. --- ****/
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "Enable TLS with KeyStore type configuration in function worker"
+ )
+ private boolean tlsEnabledWithKeyStore = false;
+
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "Specify the TLS provider for the function worker service: \n"
+ + "When using TLS authentication with CACert, the valid value is either OPENSSL or JDK.\n"
+ + "When using TLS authentication with KeyStore, available values can be SunJSSE, Conscrypt and etc."
+ )
+ private String tlsProvider = null;
+
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "TLS KeyStore type configuration in function worker: JKS, PKCS12"
+ )
+ private String tlsKeyStoreType = "JKS";
+
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "TLS KeyStore path in function worker"
+ )
+ private String tlsKeyStore = null;
+
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "TLS KeyStore password for function worker"
+ )
+ @ToString.Exclude
+ private String tlsKeyStorePassword = null;
+
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "TLS TrustStore type configuration in function worker: JKS, PKCS12"
+ )
+ private String tlsTrustStoreType = "JKS";
+
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "TLS TrustStore path in function worker"
+ )
+ private String tlsTrustStore = null;
+
+ @FieldContext(
+ category = CATEGORY_KEYSTORE_TLS,
+ doc = "TLS TrustStore password for function worker, null means empty password."
+ )
+ @ToString.Exclude
+ private String tlsTrustStorePassword = null;
+
+ @FieldContext(
+ category = CATEGORY_WORKER_SECURITY,
+ doc = "Specify the tls protocols the proxy's web service will use to negotiate during TLS Handshake.\n\n"
+ + "Example:- [TLSv1.3, TLSv1.2]"
+ )
+ private Set<String> webServiceTlsProtocols = new TreeSet<>();
+
+ @FieldContext(
+ category = CATEGORY_WORKER_SECURITY,
+ doc = "Specify the tls cipher the proxy's web service will use to negotiate during TLS Handshake.\n\n"
+ + "Example:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]"
+ )
+ private Set<String> webServiceTlsCiphers = new TreeSet<>();
+
@FieldContext(
category = CATEGORY_WORKER_SECURITY,
doc = "Enforce authentication"
diff --git a/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/worker/WorkerApiV2ResourceConfigTest.java b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/worker/WorkerApiV2ResourceConfigTest.java
index 6cfaea9e242..bf45dd958c8 100644
--- a/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/worker/WorkerApiV2ResourceConfigTest.java
+++ b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/worker/WorkerApiV2ResourceConfigTest.java
@@ -25,10 +25,11 @@ import static org.testng.Assert.assertNull;
import static org.testng.Assert.assertTrue;
import java.net.URL;
+import java.util.Locale;
+import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider;
import org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory;
-import org.apache.pulsar.functions.worker.WorkerConfig;
import org.testng.annotations.Test;
/**
@@ -121,4 +122,19 @@ public class WorkerApiV2ResourceConfigTest {
assertTrue(newK8SWc.isFunctionInstanceResourceChangeInLockStep());
}
+
+ @Test
+ public void testPasswordsNotLeakedOnToString() throws Exception {
+ URL yamlUrl = getClass().getClassLoader().getResource("test_worker_config.yml");
+ WorkerConfig wc = WorkerConfig.load(yamlUrl.toURI().getPath());
+ assertFalse(wc.toString().toLowerCase(Locale.ROOT).contains("password"), "Stringified config must not contain password");
+ }
+
+ @Test
+ public void testPasswordsPresentOnObjectMapping() throws Exception {
+ URL yamlUrl = getClass().getClassLoader().getResource("test_worker_config.yml");
+ WorkerConfig wc = WorkerConfig.load(yamlUrl.toURI().getPath());
+ assertTrue((new ObjectMapper().writeValueAsString(wc)).toLowerCase(Locale.ROOT).contains("password"),
+ "ObjectMapper output must include passwords for proper serialization");
+ }
}
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/PulsarWorkerService.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/PulsarWorkerService.java
index d0e9ada1c66..93ce035e94f 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/PulsarWorkerService.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/PulsarWorkerService.java
@@ -19,8 +19,6 @@
package org.apache.pulsar.functions.worker;
import static org.apache.pulsar.common.policies.data.PoliciesUtil.getBundles;
-import com.fasterxml.jackson.core.JsonProcessingException;
-import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Sets;
import io.netty.util.concurrent.DefaultThreadFactory;
@@ -399,12 +397,7 @@ public class PulsarWorkerService implements WorkerService {
workerStatsManager.startupTimeStart();
log.info("/** Starting worker id={} **/", workerConfig.getWorkerId());
-
- try {
- log.info("Worker Configs: {}", new ObjectMapper().writeValueAsString(workerConfig));
- } catch (JsonProcessingException e) {
- log.warn("Failed to print worker configs with error {}", e.getMessage(), e);
- }
+ log.info("Worker Configs: {}", workerConfig);
try {
DistributedLogConfiguration dlogConf = WorkerUtils.getDlogConf(workerConfig);
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java
index e5310844424..1339a5c4cc0 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java
@@ -26,6 +26,7 @@ import org.apache.pulsar.broker.web.RateLimitingFilter;
import org.apache.pulsar.broker.web.JettyRequestLogFactory;
import org.apache.pulsar.broker.web.WebExecutorThreadPool;
import org.apache.pulsar.common.util.SecurityUtility;
+import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
import org.apache.pulsar.functions.worker.WorkerConfig;
import org.apache.pulsar.functions.worker.WorkerService;
import org.apache.pulsar.functions.worker.rest.api.v2.WorkerApiV2Resource;
@@ -122,13 +123,33 @@ public class WorkerServer {
if (this.workerConfig.getTlsEnabled()) {
try {
- SslContextFactory sslCtxFactory = SecurityUtility.createSslContextFactory(
- this.workerConfig.isTlsAllowInsecureConnection(), this.workerConfig.getTlsTrustCertsFilePath(),
- this.workerConfig.getTlsCertificateFilePath(), this.workerConfig.getTlsKeyFilePath(),
- this.workerConfig.isTlsRequireTrustedClientCertOnConnect(),
- true,
- this.workerConfig.getTlsCertRefreshCheckDurationSec());
- httpsConnector = new ServerConnector(server, 1, 1, sslCtxFactory);
+ SslContextFactory sslCtxFactory;
+ if (workerConfig.isTlsEnabledWithKeyStore()) {
+ sslCtxFactory = KeyStoreSSLContext.createSslContextFactory(
+ workerConfig.getTlsProvider(),
+ workerConfig.getTlsKeyStoreType(),
+ workerConfig.getTlsKeyStore(),
+ workerConfig.getTlsKeyStorePassword(),
+ workerConfig.isTlsAllowInsecureConnection(),
+ workerConfig.getTlsTrustStoreType(),
+ workerConfig.getTlsTrustStore(),
+ workerConfig.getTlsTrustStorePassword(),
+ workerConfig.isTlsRequireTrustedClientCertOnConnect(),
+ workerConfig.getWebServiceTlsCiphers(),
+ workerConfig.getWebServiceTlsProtocols(),
+ workerConfig.getTlsCertRefreshCheckDurationSec()
+ );
+ } else {
+ sslCtxFactory = SecurityUtility.createSslContextFactory(
+ workerConfig.isTlsAllowInsecureConnection(),
+ workerConfig.getTlsTrustCertsFilePath(),
+ workerConfig.getTlsCertificateFilePath(),
+ workerConfig.getTlsKeyFilePath(),
+ workerConfig.isTlsRequireTrustedClientCertOnConnect(),
+ true,
+ workerConfig.getTlsCertRefreshCheckDurationSec());
+ }
+ httpsConnector = new ServerConnector(server, sslCtxFactory);
httpsConnector.setPort(this.workerConfig.getWorkerPortTls());
connectors.add(httpsConnector);
} catch (Exception e) {
diff --git a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java
index 6a22e984827..e3a30ad4ca7 100644
--- a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java
+++ b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/ProxyServer.java
@@ -33,6 +33,7 @@ import org.apache.pulsar.broker.web.JsonMapperProvider;
import org.apache.pulsar.broker.web.WebExecutorThreadPool;
import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.common.util.SecurityUtility;
+import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
@@ -72,15 +73,33 @@ public class ProxyServer {
// TLS enabled connector
if (config.getWebServicePortTls().isPresent()) {
try {
- SslContextFactory sslCtxFactory = SecurityUtility.createSslContextFactory(
- config.isTlsAllowInsecureConnection(),
- config.getTlsTrustCertsFilePath(),
- config.getTlsCertificateFilePath(),
- config.getTlsKeyFilePath(),
- config.isTlsRequireTrustedClientCertOnConnect(),
- true,
- config.getTlsCertRefreshCheckDurationSec());
- connectorTls = new ServerConnector(server, -1, -1, sslCtxFactory);
+ SslContextFactory sslCtxFactory;
+ if (config.isTlsEnabledWithKeyStore()) {
+ sslCtxFactory = KeyStoreSSLContext.createSslContextFactory(
+ config.getTlsProvider(),
+ config.getTlsKeyStoreType(),
+ config.getTlsKeyStore(),
+ config.getTlsKeyStorePassword(),
+ config.isTlsAllowInsecureConnection(),
+ config.getTlsTrustStoreType(),
+ config.getTlsTrustStore(),
+ config.getTlsTrustStorePassword(),
+ config.isTlsRequireTrustedClientCertOnConnect(),
+ config.getWebServiceTlsCiphers(),
+ config.getWebServiceTlsProtocols(),
+ config.getTlsCertRefreshCheckDurationSec()
+ );
+ } else {
+ sslCtxFactory = SecurityUtility.createSslContextFactory(
+ config.isTlsAllowInsecureConnection(),
+ config.getTlsTrustCertsFilePath(),
+ config.getTlsCertificateFilePath(),
+ config.getTlsKeyFilePath(),
+ config.isTlsRequireTrustedClientCertOnConnect(),
+ true,
+ config.getTlsCertRefreshCheckDurationSec());
+ }
+ connectorTls = new ServerConnector(server, sslCtxFactory);
connectorTls.setPort(config.getWebServicePortTls().get());
connectors.add(connectorTls);
} catch (Exception e) {
diff --git a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java
index ad9b4be0228..4480d4525e5 100644
--- a/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java
+++ b/pulsar-websocket/src/main/java/org/apache/pulsar/websocket/service/WebSocketProxyConfiguration.java
@@ -24,6 +24,7 @@ import java.util.Set;
import java.util.TreeSet;
import lombok.Getter;
import lombok.Setter;
+import lombok.ToString;
import org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider;
import org.apache.pulsar.common.configuration.FieldContext;
import org.apache.pulsar.common.configuration.PulsarConfiguration;
@@ -175,6 +176,63 @@ public class WebSocketProxyConfiguration implements PulsarConfiguration {
@FieldContext(doc = "TLS cert refresh duration (in seconds). 0 means checking every new connection.")
private long tlsCertRefreshCheckDurationSec = 300;
+ /**** --- KeyStore TLS config variables. --- ****/
+ @FieldContext(
+ doc = "Enable TLS with KeyStore type configuration for WebSocket"
+ )
+ private boolean tlsEnabledWithKeyStore = false;
+
+ @FieldContext(
+ doc = "Specify the TLS provider for the WebSocket service: \n"
+ + "When using TLS authentication with CACert, the valid value is either OPENSSL or JDK.\n"
+ + "When using TLS authentication with KeyStore, available values can be SunJSSE, Conscrypt and etc."
+ )
+ private String tlsProvider = null;
+
+ @FieldContext(
+ doc = "TLS KeyStore type configuration in WebSocket: JKS, PKCS12"
+ )
+ private String tlsKeyStoreType = "JKS";
+
+ @FieldContext(
+ doc = "TLS KeyStore path in WebSocket"
+ )
+ private String tlsKeyStore = null;
+
+ @FieldContext(
+ doc = "TLS KeyStore password for WebSocket"
+ )
+ @ToString.Exclude
+ private String tlsKeyStorePassword = null;
+
+ @FieldContext(
+ doc = "TLS TrustStore type configuration in WebSocket: JKS, PKCS12"
+ )
+ private String tlsTrustStoreType = "JKS";
+
+ @FieldContext(
+ doc = "TLS TrustStore path in WebSocket"
+ )
+ private String tlsTrustStore = null;
+
+ @FieldContext(
+ doc = "TLS TrustStore password for WebSocket, null means empty password."
+ )
+ @ToString.Exclude
+ private String tlsTrustStorePassword = null;
+
+ @FieldContext(
+ doc = "Specify the tls protocols the proxy's web service will use to negotiate during TLS Handshake.\n\n"
+ + "Example:- [TLSv1.3, TLSv1.2]"
+ )
+ private Set<String> webServiceTlsProtocols = new TreeSet<>();
+
+ @FieldContext(
+ doc = "Specify the tls cipher the proxy's web service will use to negotiate during TLS Handshake.\n\n"
+ + "Example:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]"
+ )
+ private Set<String> webServiceTlsCiphers = new TreeSet<>();
+
@FieldContext(doc = "Key-value properties. Types are all String")
private Properties properties = new Properties();
[pulsar] 23/31: [PIP-163][Txn]Add lowWaterMark check before appending entry to TB (#15424)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e3ba66efc82e722dfa370a0dbaabcb1e673b5406
Author: Xiangying Meng <55...@users.noreply.github.com>
AuthorDate: Fri May 13 09:38:32 2022 +0800
[PIP-163][Txn]Add lowWaterMark check before appending entry to TB (#15424)
Master Issue: [#15423](https://github.com/apache/pulsar/issues/15423)
### Motivation & Modification
Details can be found at https://github.com/apache/pulsar/issues/15423.
(cherry picked from commit 15d6907153007ffbf94a351a19df31763b0c6d5a)
---
.../broker/service/persistent/PersistentTopic.java | 8 +++--
.../buffer/impl/TopicTransactionBuffer.java | 24 +++++++++++++
.../buffer/TransactionLowWaterMarkTest.java | 40 ++++++++++++++++++++++
.../org/apache/pulsar/client/impl/ClientCnx.java | 2 +-
.../apache/pulsar/client/impl/ProducerImpl.java | 6 ++--
5 files changed, 72 insertions(+), 8 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index a36176cdb45..1ef90c35841 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -2956,8 +2956,7 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
return;
}
if (isExceedMaximumMessageSize(headersAndPayload.readableBytes(), publishContext)) {
- publishContext.completed(new NotAllowedException("Exceed maximum message size")
- , -1, -1);
+ publishContext.completed(new NotAllowedException("Exceed maximum message size"), -1, -1);
decrementPendingWriteOpsAndCheck();
return;
}
@@ -2978,7 +2977,10 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
})
.exceptionally(throwable -> {
throwable = throwable.getCause();
- if (!(throwable instanceof ManagedLedgerException)) {
+ if (throwable instanceof NotAllowedException) {
+ publishContext.completed((NotAllowedException) throwable, -1, -1);
+ return null;
+ } else if (!(throwable instanceof ManagedLedgerException)) {
throwable = new ManagedLedgerException(throwable);
}
addFailed((ManagedLedgerException) throwable, publishContext);
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
index a432b76f5fa..c9cde544c80 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
@@ -25,6 +25,7 @@ import io.netty.util.TimerTask;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import lombok.SneakyThrows;
@@ -95,6 +96,8 @@ public class TopicTransactionBuffer extends TopicTransactionBufferState implemen
private final CompletableFuture<Void> transactionBufferFuture = new CompletableFuture<>();
+ private final ConcurrentHashMap<Long, Long> lowWaterMarks = new ConcurrentHashMap<>();
+
public TopicTransactionBuffer(PersistentTopic topic) {
super(State.None);
this.topic = topic;
@@ -240,6 +243,13 @@ public class TopicTransactionBuffer extends TopicTransactionBufferState implemen
@Override
public CompletableFuture<Position> appendBufferToTxn(TxnID txnId, long sequenceId, ByteBuf buffer) {
CompletableFuture<Position> completableFuture = new CompletableFuture<>();
+ Long lowWaterMark = lowWaterMarks.get(txnId.getMostSigBits());
+ if (lowWaterMark != null && lowWaterMark >= txnId.getLeastSigBits()) {
+ completableFuture.completeExceptionally(new BrokerServiceException
+ .NotAllowedException("Transaction [" + txnId + "] has been ended. "
+ + "Please use a new transaction to send message."));
+ return completableFuture;
+ }
topic.getManagedLedger().asyncAddEntry(buffer, new AsyncCallbacks.AddEntryCallback() {
@Override
public void addComplete(Position position, ByteBuf entryData, Object ctx) {
@@ -275,6 +285,13 @@ public class TopicTransactionBuffer extends TopicTransactionBufferState implemen
@Override
public CompletableFuture<Void> commitTxn(TxnID txnID, long lowWaterMark) {
+ lowWaterMarks.compute(txnID.getMostSigBits(), (tcId, oldLowWaterMark) -> {
+ if (oldLowWaterMark == null || oldLowWaterMark < lowWaterMark) {
+ return lowWaterMark;
+ } else {
+ return oldLowWaterMark;
+ }
+ });
if (log.isDebugEnabled()) {
log.debug("Transaction {} commit on topic {}.", txnID.toString(), topic.getName());
}
@@ -315,6 +332,13 @@ public class TopicTransactionBuffer extends TopicTransactionBufferState implemen
@Override
public CompletableFuture<Void> abortTxn(TxnID txnID, long lowWaterMark) {
+ lowWaterMarks.compute(txnID.getMostSigBits(), (tcId, oldLowWaterMark) -> {
+ if (oldLowWaterMark == null || oldLowWaterMark < lowWaterMark) {
+ return lowWaterMark;
+ } else {
+ return oldLowWaterMark;
+ }
+ });
if (log.isDebugEnabled()) {
log.debug("Transaction {} abort on topic {}.", txnID.toString(), topic.getName());
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionLowWaterMarkTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionLowWaterMarkTest.java
index 873509ff6bf..ba0659892b4 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionLowWaterMarkTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionLowWaterMarkTest.java
@@ -42,6 +42,7 @@ import org.apache.pulsar.broker.transaction.pendingack.impl.PendingAckHandleImpl
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;
import org.apache.pulsar.client.api.SubscriptionType;
import org.apache.pulsar.client.api.transaction.Transaction;
@@ -287,4 +288,43 @@ public class TransactionLowWaterMarkTest extends TransactionTestBase {
fail();
}
}
+
+ @Test
+ public void testTBLowWaterMarkEndToEnd() throws Exception {
+ Transaction txn1 = pulsarClient.newTransaction()
+ .withTransactionTimeout(500, TimeUnit.SECONDS)
+ .build().get();
+ Transaction txn2 = pulsarClient.newTransaction()
+ .withTransactionTimeout(500, TimeUnit.SECONDS)
+ .build().get();
+ while (txn2.getTxnID().getMostSigBits() != txn1.getTxnID().getMostSigBits()) {
+ txn2 = pulsarClient.newTransaction()
+ .withTransactionTimeout(500, TimeUnit.SECONDS)
+ .build().get();
+ }
+
+ @Cleanup
+ Producer<byte[]> producer = pulsarClient
+ .newProducer()
+ .topic(TOPIC)
+ .sendTimeout(0, TimeUnit.SECONDS)
+ .enableBatching(false)
+ .create();
+
+ producer.newMessage(txn1).send();
+ producer.newMessage(txn2).send();
+
+ txn1.commit().get();
+ txn2.commit().get();
+
+ Field field = TransactionImpl.class.getDeclaredField("state");
+ field.setAccessible(true);
+ field.set(txn1, TransactionImpl.State.OPEN);
+ try {
+ producer.newMessage(txn1).send();
+ fail();
+ } catch (PulsarClientException.NotAllowedException ignore) {
+ // no-op
+ }
+ }
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
index 29d86d58d1d..edefb2fe3a4 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
@@ -692,7 +692,7 @@ public class ClientCnx extends PulsarHandler {
producers.get(producerId).terminated(this);
break;
case NotAllowedError:
- producers.get(producerId).recoverNotAllowedError(sequenceId);
+ producers.get(producerId).recoverNotAllowedError(sequenceId, sendError.getMessage());
break;
default:
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index 45d1416f898..0bf37a93a3a 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -1186,16 +1186,14 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
resendMessages(cnx, this.connectionHandler.getEpoch());
}
- protected synchronized void recoverNotAllowedError(long sequenceId) {
+ protected synchronized void recoverNotAllowedError(long sequenceId, String errorMsg) {
OpSendMsg op = pendingMessages.peek();
if (op != null && sequenceId == getHighestSequenceId(op)) {
pendingMessages.remove();
releaseSemaphoreForSendOp(op);
try {
op.sendComplete(
- new PulsarClientException.NotAllowedException(
- format("The size of the message which is produced by producer %s to the topic "
- + "%s is not allowed", producerName, topic)));
+ new PulsarClientException.NotAllowedException(errorMsg));
} catch (Throwable t) {
log.warn("[{}] [{}] Got exception while completing the callback for msg {}:", topic,
producerName, sequenceId, t);
[pulsar] 11/31: [Proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler (#15415)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit d776e17dcaff01afcefee42c83e35e35cc624b2a
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Tue May 3 22:49:43 2022 +0300
[Proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler (#15415)
* [Proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler
* Use existing code pattern for creating address
(cherry picked from commit 7373a51690d728475d47846bfbcca4fa64f2e228)
---
.../pulsar/proxy/server/LookupProxyHandler.java | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
index 69d60fcabd0..c242242d0c9 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
@@ -215,20 +215,16 @@ public class LookupProxyHandler {
private void handlePartitionMetadataResponse(CommandPartitionedTopicMetadata partitionMetadata,
long clientRequestId) {
TopicName topicName = TopicName.get(partitionMetadata.getTopic());
- URI brokerURI;
- try {
- String availableBrokerServiceURL = getBrokerServiceUrl(clientRequestId);
- if (availableBrokerServiceURL == null) {
- log.warn("No available broker for {} to lookup partition metadata", topicName);
- return;
- }
- brokerURI = new URI(availableBrokerServiceURL);
- } catch (URISyntaxException e) {
- proxyConnection.ctx().writeAndFlush(Commands.newPartitionMetadataResponse(ServerError.MetadataError,
- e.getMessage(), clientRequestId));
+
+ String serviceUrl = getBrokerServiceUrl(clientRequestId);
+ if (serviceUrl == null) {
+ log.warn("No available broker for {} to lookup partition metadata", topicName);
+ return;
+ }
+ InetSocketAddress addr = getAddr(serviceUrl, clientRequestId);
+ if (addr == null) {
return;
}
- InetSocketAddress addr = new InetSocketAddress(brokerURI.getHost(), brokerURI.getPort());
if (log.isDebugEnabled()) {
log.debug("Getting connections to '{}' for Looking up topic '{}' with clientReq Id '{}'", addr,
[pulsar] 31/31: [fix][broker] fix MetadataStoreException$NotFoundException while doing topic lookup (#15633)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 6415b313da91f9ef2bb1e386e17281bbbf8a2522
Author: lipenghui <pe...@apache.org>
AuthorDate: Thu May 19 22:07:44 2022 +0800
[fix][broker] fix MetadataStoreException$NotFoundException while doing topic lookup (#15633)
(cherry picked from commit 70551a6f6be05826b90295d3da5915613e53fa46)
---
.../pulsar/broker/loadbalance/ResourceUnit.java | 5 +++++
.../impl/ModularLoadManagerWrapper.java | 14 ++++++++++++--
.../loadbalance/impl/SimpleResourceUnit.java | 22 ++++++++++++++++++++--
.../pulsar/broker/namespace/NamespaceService.java | 16 +++++++++++-----
.../loadbalance/AdvertisedListenersTest.java | 18 +++++++++++-------
5 files changed, 59 insertions(+), 16 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ResourceUnit.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ResourceUnit.java
index 1afde4e3657..51becdb7f77 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ResourceUnit.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ResourceUnit.java
@@ -22,9 +22,14 @@ package org.apache.pulsar.broker.loadbalance;
ResourceUnit represents any machine/unit which has resources that broker can use to serve its service units
*/
public interface ResourceUnit extends Comparable<ResourceUnit> {
+
+ String PROPERTY_KEY_BROKER_ZNODE_NAME = "__advertised_addr";
+
String getResourceId();
ResourceDescription getAvailableResource();
boolean canFit(ResourceDescription resourceDescription);
+
+ Object getProperty(String key);
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerWrapper.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerWrapper.java
index 1ab33d0ab0f..d18cd616586 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerWrapper.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerWrapper.java
@@ -19,6 +19,7 @@
package org.apache.pulsar.broker.loadbalance.impl;
import java.util.List;
+import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
@@ -65,8 +66,12 @@ public class ModularLoadManagerWrapper implements LoadManager {
@Override
public Optional<ResourceUnit> getLeastLoaded(final ServiceUnitId serviceUnit) {
Optional<String> leastLoadedBroker = loadManager.selectBrokerForAssignment(serviceUnit);
- return leastLoadedBroker.map(s -> new SimpleResourceUnit(getBrokerWebServiceUrl(s),
- new PulsarResourceDescription()));
+ return leastLoadedBroker.map(s -> {
+ String webServiceUrl = getBrokerWebServiceUrl(s);
+ String brokerZnodeName = getBrokerZnodeName(s, webServiceUrl);
+ return new SimpleResourceUnit(webServiceUrl,
+ new PulsarResourceDescription(), Map.of(ResourceUnit.PROPERTY_KEY_BROKER_ZNODE_NAME, brokerZnodeName));
+ });
}
private String getBrokerWebServiceUrl(String broker) {
@@ -78,6 +83,11 @@ public class ModularLoadManagerWrapper implements LoadManager {
return String.format("http://%s", broker);
}
+ private String getBrokerZnodeName(String broker, String webServiceUrl) {
+ String scheme = webServiceUrl.substring(0, webServiceUrl.indexOf("://"));
+ return String.format("%s://%s", scheme, broker);
+ }
+
@Override
public List<Metrics> getLoadBalancingMetrics() {
return loadManager.getLoadBalancingMetrics();
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleResourceUnit.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleResourceUnit.java
index 62f9b3e94a5..863a75dff42 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleResourceUnit.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleResourceUnit.java
@@ -19,19 +19,32 @@
package org.apache.pulsar.broker.loadbalance.impl;
import com.google.common.base.MoreObjects;
+import java.util.Collections;
+import java.util.Map;
import org.apache.pulsar.broker.loadbalance.ResourceDescription;
import org.apache.pulsar.broker.loadbalance.ResourceUnit;
public class SimpleResourceUnit implements ResourceUnit {
- private String resourceId;
- private ResourceDescription resourceDescription;
+ private final String resourceId;
+ private final ResourceDescription resourceDescription;
+
+ private final Map<String, Object> properties;
public SimpleResourceUnit(String resourceId, ResourceDescription resourceDescription) {
this.resourceId = resourceId;
this.resourceDescription = resourceDescription;
+ this.properties = Collections.emptyMap();
+ }
+
+ public SimpleResourceUnit(String resourceId, ResourceDescription resourceDescription,
+ Map<String, Object> properties) {
+ this.resourceId = resourceId;
+ this.resourceDescription = resourceDescription;
+ this.properties = properties == null ? Collections.emptyMap() : properties;
}
+
@Override
public String getResourceId() {
// TODO Auto-generated method stub
@@ -50,6 +63,11 @@ public class SimpleResourceUnit implements ResourceUnit {
return this.resourceDescription.compareTo(resourceDescription) > 0;
}
+ @Override
+ public Object getProperty(String key) {
+ return properties.get(key);
+ }
+
@Override
public int compareTo(ResourceUnit o) {
return resourceId.compareTo(o.getResourceId());
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
index 98e65dc3e56..6b078107b6d 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
@@ -45,6 +45,7 @@ import java.util.stream.Collectors;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.commons.collections4.ListUtils;
import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
import org.apache.pulsar.broker.PulsarServerException;
import org.apache.pulsar.broker.PulsarService;
import org.apache.pulsar.broker.ServiceConfiguration;
@@ -455,6 +456,7 @@ public class NamespaceService implements AutoCloseable {
return;
}
String candidateBroker = null;
+ String candidateBrokerAdvertisedAddr = null;
LeaderElectionService les = pulsar.getLeaderElectionService();
if (les == null) {
@@ -515,7 +517,7 @@ public class NamespaceService implements AutoCloseable {
}
}
if (makeLoadManagerDecisionOnThisBroker) {
- Optional<String> availableBroker = getLeastLoadedFromLoadManager(bundle);
+ Optional<Pair<String, String>> availableBroker = getLeastLoadedFromLoadManager(bundle);
if (!availableBroker.isPresent()) {
LOG.warn("Load manager didn't return any available broker. "
+ "Returning empty result to lookup. NamespaceBundle[{}]",
@@ -523,7 +525,8 @@ public class NamespaceService implements AutoCloseable {
lookupFuture.complete(Optional.empty());
return;
}
- candidateBroker = availableBroker.get();
+ candidateBroker = availableBroker.get().getLeft();
+ candidateBrokerAdvertisedAddr = availableBroker.get().getRight();
authoritativeRedirect = true;
} else {
// forward to leader broker to make assignment
@@ -594,7 +597,8 @@ public class NamespaceService implements AutoCloseable {
}
// Now setting the redirect url
- createLookupResult(candidateBroker, authoritativeRedirect, options.getAdvertisedListenerName())
+ createLookupResult(candidateBrokerAdvertisedAddr == null ? candidateBroker
+ : candidateBrokerAdvertisedAddr, authoritativeRedirect, options.getAdvertisedListenerName())
.thenAccept(lookupResult -> lookupFuture.complete(Optional.of(lookupResult)))
.exceptionally(ex -> {
lookupFuture.completeExceptionally(ex);
@@ -689,7 +693,7 @@ public class NamespaceService implements AutoCloseable {
* @return
* @throws Exception
*/
- private Optional<String> getLeastLoadedFromLoadManager(ServiceUnitId serviceUnit) throws Exception {
+ private Optional<Pair<String, String>> getLeastLoadedFromLoadManager(ServiceUnitId serviceUnit) throws Exception {
Optional<ResourceUnit> leastLoadedBroker = loadManager.get().getLeastLoaded(serviceUnit);
if (!leastLoadedBroker.isPresent()) {
LOG.warn("No broker is available for {}", serviceUnit);
@@ -697,12 +701,14 @@ public class NamespaceService implements AutoCloseable {
}
String lookupAddress = leastLoadedBroker.get().getResourceId();
+ String advertisedAddr = (String) leastLoadedBroker.get()
+ .getProperty(ResourceUnit.PROPERTY_KEY_BROKER_ZNODE_NAME);
if (LOG.isDebugEnabled()) {
LOG.debug("{} : redirecting to the least loaded broker, lookup address={}",
pulsar.getSafeWebServiceAddress(),
lookupAddress);
}
- return Optional.of(lookupAddress);
+ return Optional.of(Pair.of(lookupAddress, advertisedAddr));
}
public CompletableFuture<Void> unloadNamespaceBundle(NamespaceBundle bundle) {
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/AdvertisedListenersTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/AdvertisedListenersTest.java
index 6ff49674e2e..489efa5755b 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/AdvertisedListenersTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/AdvertisedListenersTest.java
@@ -19,6 +19,8 @@
package org.apache.pulsar.broker.loadbalance;
import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+
import java.net.URI;
import java.util.Optional;
import lombok.Cleanup;
@@ -69,14 +71,14 @@ public class AdvertisedListenersTest extends MultiBrokerBaseTest {
int httpsPort = PortManager.nextFreePort();
// Use invalid domain name as identifier and instead make sure the advertised listeners work as intended
- this.conf.setAdvertisedAddress(advertisedAddress);
- this.conf.setAdvertisedListeners(
+ conf.setAdvertisedAddress(advertisedAddress);
+ conf.setAdvertisedListeners(
"public:pulsar://localhost:" + pulsarPort +
",public_http:http://localhost:" + httpPort +
",public_https:https://localhost:" + httpsPort);
- this.conf.setBrokerServicePort(Optional.of(pulsarPort));
- this.conf.setWebServicePort(Optional.of(httpPort));
- this.conf.setWebServicePortTls(Optional.of(httpsPort));
+ conf.setBrokerServicePort(Optional.of(pulsarPort));
+ conf.setWebServicePort(Optional.of(httpPort));
+ conf.setWebServicePortTls(Optional.of(httpsPort));
}
@Test
@@ -85,6 +87,7 @@ public class AdvertisedListenersTest extends MultiBrokerBaseTest {
new HttpGet(pulsar.getWebServiceAddress() + "/lookup/v2/topic/persistent/public/default/my-topic");
request.addHeader(HttpHeaders.CONTENT_TYPE, "application/json");
request.addHeader(HttpHeaders.ACCEPT, "application/json");
+ final String topic = "my-topic";
@Cleanup
CloseableHttpClient httpClient = HttpClients.createDefault();
@@ -104,14 +107,15 @@ public class AdvertisedListenersTest extends MultiBrokerBaseTest {
// Produce data
@Cleanup
Producer<String> p = pulsarClient.newProducer(Schema.STRING)
- .topic("my-topic")
+ .topic(topic)
.create();
p.send("hello");
// Verify we can get the correct HTTP redirect to the advertised listener
for (PulsarAdmin a : getAllAdmins()) {
- TopicStats s = a.topics().getStats("my-topic");
+ TopicStats s = a.topics().getStats(topic);
+ assertNotNull(a.lookups().lookupTopic(topic));
assertEquals(s.getPublishers().size(), 1);
}
}
[pulsar] 21/31: [fix][txn] Topic transaction buffer recover don't close reader when throw RuntimeException (#15361)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 810d707d8fcbcce8ce8e8e8b3a342114dff75ef5
Author: congbo <39...@users.noreply.github.com>
AuthorDate: Tue May 10 16:03:45 2022 +0800
[fix][txn] Topic transaction buffer recover don't close reader when throw RuntimeException (#15361)
Fixes: https://github.com/apache/pulsar/issues/14878
### Motivation
clear unuse reader in topicTransactionBufferSnapshot topic
When reader decode the Snapshot will throw RuntimeException not PulsarClientException
We should catch the Exception then close the reader and topic
```
"java.util.concurrent.CompletionException: com.google.common.util.concurrent.UncheckedExecutionException: org.apache.commons.lang3.SerializationException: Failed at fetching schema info for 0
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:704) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer.lambda$checkIfTBRecoverCompletely$3(TopicTransactionBuffer.java:232) ~[org.apache-pulsar-broker-2.9.2.jar:2.9.2]
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986) ~[?:?]
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer$1.recoverExceptionally(TopicTransactionBuffer.java:196) ~[org.apache-pulsar-broker-2.9.2.jar:2.9.2]
at org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer$TopicTransactionBufferRecover.lambda$null$1(TopicTransactionBuffer.java:647) ~[org.apache-pulsar-broker-2.9.2.jar:2.9.2]
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986) [?:?]
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970) [?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610) [?:?]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:722) [?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: org.apache.commons.lang3.SerializationException: Failed at fetching schema info for 0
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) ~[com.google.guava-guava-30.1-jre.jar:?]
at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.getSchemaReader(AbstractMultiVersionReader.java:83) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.read(AbstractMultiVersionReader.java:90) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.schema.AbstractStructSchema.decode(AbstractStructSchema.java:67) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.MessageImpl.decode(MessageImpl.java:484) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:462) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer$TopicTransactionBufferRecover.lambda$null$0(TopicTransactionBuffer.java:583) ~[org.apache-pulsar-broker-2.9.2.jar:2.9.2]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
... 8 more
Caused by: org.apache.commons.lang3.SerializationException: Failed at fetching schema info for 0
at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.getSchemaInfoByVersion(AbstractMultiVersionReader.java:129) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.schema.reader.MultiVersionAvroReader.loadReader(MultiVersionAvroReader.java:47) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader$1.load(AbstractMultiVersionReader.java:52) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader$1.load(AbstractMultiVersionReader.java:49) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) ~[com.google.guava-guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) ~[com.google.guava-guava-30.1-jre.jar:?]
at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.getSchemaReader(AbstractMultiVersionReader.java:83) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.read(AbstractMultiVersionReader.java:90) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.schema.AbstractStructSchema.decode(AbstractStructSchema.java:67) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.MessageImpl.decode(MessageImpl.java:484) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:462) ~[org.apache-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer$TopicTransactionBufferRecover.lambda$null$0(TopicTransactionBuffer.java:583) ~[org.apache-pulsar-broker-2.9.2.jar:2.9.2]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
... 8 more
```
### Modifications
catch Exception then close the topic and reader
(cherry picked from commit 0c58810d29838a161481f03c14990d0eb021a185)
---
.../transaction/buffer/impl/TopicTransactionBuffer.java | 8 ++++----
.../broker/transaction/TopicTransactionBufferRecoverTest.java | 11 +++++++++--
2 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
index 3cbc3f14ea5..a432b76f5fa 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/TopicTransactionBuffer.java
@@ -595,10 +595,10 @@ public class TopicTransactionBuffer extends TopicTransactionBufferState implemen
callBack.noNeedToRecover();
return;
}
- } catch (PulsarClientException pulsarClientException) {
- log.error("[{}]Transaction buffer recover fail when read "
- + "transactionBufferSnapshot!", topic.getName(), pulsarClientException);
- callBack.recoverExceptionally(pulsarClientException);
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ callBack.recoverExceptionally(ex);
closeReader(reader);
return;
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
index c3491738985..351fe124852 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
@@ -455,7 +455,7 @@ public class TopicTransactionBufferRecoverTest extends TransactionTestBase {
@Test(timeOut=30000)
- public void testTransactionBufferRecoverThrowPulsarClientException() throws Exception {
+ public void testTransactionBufferRecoverThrowException() throws Exception {
String topic = NAMESPACE1 + "/testTransactionBufferRecoverThrowPulsarClientException";
@Cleanup
Producer<byte[]> producer = pulsarClient
@@ -489,7 +489,14 @@ public class TopicTransactionBufferRecoverTest extends TransactionTestBase {
field.setAccessible(true);
TransactionBufferSnapshotService transactionBufferSnapshotServiceOriginal =
(TransactionBufferSnapshotService) field.get(getPulsarServiceList().get(0));
- // mock reader can't read snapshot fail
+ // mock reader can't read snapshot fail throw RuntimeException
+ doThrow(new RuntimeException("test")).when(reader).hasMoreEvents();
+ // check reader close topic
+ checkCloseTopic(pulsarClient, transactionBufferSnapshotServiceOriginal,
+ transactionBufferSnapshotService, originalTopic, field, producer);
+ doReturn(true).when(reader).hasMoreEvents();
+
+ // mock reader can't read snapshot fail throw PulsarClientException
doThrow(new PulsarClientException("test")).when(reader).hasMoreEvents();
// check reader close topic
checkCloseTopic(pulsarClient, transactionBufferSnapshotServiceOriginal,
[pulsar] 15/31: Revert "Fix: LockManagerTest.updateValue is flaky (#13911)" (#15235)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit c9ca16e95bb080e63dd93eba02a455d7723cd0e5
Author: Baodi Shi <wu...@icloud.com>
AuthorDate: Sat May 7 08:57:09 2022 +0800
Revert "Fix: LockManagerTest.updateValue is flaky (#13911)" (#15235)
(cherry picked from commit 306a0a11c9405abce8de9c09ed3aa6db5e22062e)
---
.../metadata/impl/LocalMemoryMetadataStore.java | 26 +++++++++-------------
1 file changed, 10 insertions(+), 16 deletions(-)
diff --git a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/LocalMemoryMetadataStore.java b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/LocalMemoryMetadataStore.java
index 54120732b0b..63efba8f724 100644
--- a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/LocalMemoryMetadataStore.java
+++ b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/LocalMemoryMetadataStore.java
@@ -161,24 +161,21 @@ public class LocalMemoryMetadataStore extends AbstractMetadataStore implements M
long now = System.currentTimeMillis();
- CompletableFuture<Stat> future = new CompletableFuture<>();
if (hasVersion && expectedVersion == -1) {
Value newValue = new Value(0, data, now, now, options.contains(CreateOption.Ephemeral));
Value existingValue = map.putIfAbsent(path, newValue);
if (existingValue != null) {
- execute(() -> future.completeExceptionally(new BadVersionException("")), future);
+ return FutureUtils.exception(new BadVersionException(""));
} else {
receivedNotification(new Notification(NotificationType.Created, path));
notifyParentChildrenChanged(path);
- String finalPath = path;
- execute(() -> future.complete(new Stat(finalPath, 0, now, now, newValue.isEphemeral(),
- true)), future);
+ return FutureUtils.value(new Stat(path, 0, now, now, newValue.isEphemeral(), true));
}
} else {
Value existingValue = map.get(path);
long existingVersion = existingValue != null ? existingValue.version : -1;
if (hasVersion && expectedVersion != existingVersion) {
- execute(() -> future.completeExceptionally(new BadVersionException("")), future);
+ return FutureUtils.exception(new BadVersionException(""));
} else {
long newVersion = existingValue != null ? existingValue.version + 1 : 0;
long createdTimestamp = existingValue != null ? existingValue.createdTimestamp : now;
@@ -192,13 +189,12 @@ public class LocalMemoryMetadataStore extends AbstractMetadataStore implements M
if (type == NotificationType.Created) {
notifyParentChildrenChanged(path);
}
- String finalPath = path;
- execute(() -> future.complete(new Stat(finalPath, newValue.version, newValue.createdTimestamp,
- newValue.modifiedTimestamp,
- false, true)), future);
+ return FutureUtils
+ .value(new Stat(path, newValue.version, newValue.createdTimestamp,
+ newValue.modifiedTimestamp,
+ false, true));
}
}
- return future;
}
}
@@ -208,20 +204,18 @@ public class LocalMemoryMetadataStore extends AbstractMetadataStore implements M
return FutureUtil.failedFuture(new MetadataStoreException.InvalidPathException(path));
}
synchronized (map) {
- CompletableFuture<Void> future = new CompletableFuture<>();
Value value = map.get(path);
if (value == null) {
- execute(() -> future.completeExceptionally(new NotFoundException("")), future);
+ return FutureUtils.exception(new NotFoundException(""));
} else if (optExpectedVersion.isPresent() && optExpectedVersion.get() != value.version) {
- execute(() -> future.completeExceptionally(new BadVersionException("")), future);
+ return FutureUtils.exception(new BadVersionException(""));
} else {
map.remove(path);
receivedNotification(new Notification(NotificationType.Deleted, path));
notifyParentChildrenChanged(path);
- execute(() -> future.complete(null), future);
+ return FutureUtils.value(null);
}
- return future;
}
}
}
[pulsar] 13/31: [fix][package-management] Fix the new path `/data` introduced regression (#15367)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 46d6a7f41b780000201f3d2a548f0e9d94f35ad4
Author: Yong Zhang <zh...@gmail.com>
AuthorDate: Fri May 6 10:04:03 2022 +0800
[fix][package-management] Fix the new path `/data` introduced regression (#15367)
---
Fixes #15362
*Motivation*
The PR https://github.com/apache/pulsar/pull/13218 supports saving
package data into filesystem. But it introduces a regression for the
old versions.
For example, we have a package function://public/default/package@v0.1,
it will save the meta to the path function/public/default/package/v0.1/meta,
and save the data to the path function/public/default/package/v0.1.
By default, we are using distributed log as the package storage, and
it supports saving data in a directory.
But some storage like filesystem doesn't have the similar ability, it
needs another path for saving data.
This API provides the ability to support saving the data in another place.
If you specify the data path as `/data`, the package will be saved into
function/public/default/package/v0.1/data.
*Modifications*
- make the data path configurable in the storage implementation.
(cherry picked from commit aa9aa1886079f6a244d07ee71ee129c44bccdb2a)
---
.../packages/management/core/PackagesStorage.java | 21 +++++++
.../core/impl/PackagesManagementImpl.java | 9 ++-
.../core/impl/PackagesManagementImplTest.java | 69 ++++++++++++++++++++--
.../filesystem/FileSystemPackagesStorage.java | 5 ++
4 files changed, 95 insertions(+), 9 deletions(-)
diff --git a/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesStorage.java b/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesStorage.java
index 9672cf86603..6cebe6c6370 100644
--- a/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesStorage.java
+++ b/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesStorage.java
@@ -83,4 +83,25 @@ public interface PackagesStorage {
* @return
*/
CompletableFuture<Void> closeAsync();
+
+ /**
+ * The extra path for saving package data.
+ *
+ * For example, we have a package function://public/default/package@v0.1,
+ * it will save the meta to the path function/public/default/package/v0.1/meta,
+ * and save the data to the path function/public/default/package/v0.1.
+ * By default, we are using distributed log as the package storage, and it supports
+ * saving data at a directory.
+ * But some storage like filesystem don't have the similar ability, it needs another path
+ * for saving the data.
+ * This api provides the ability to support saving the data in another place.
+ * If you specify the data path as `/data`, the package will saved into
+ * function/public/default/package/v0.1/data.
+ *
+ * @return
+ * the data path
+ */
+ default String dataPath() {
+ return "";
+ }
}
diff --git a/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java b/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java
index ca1adaa458d..792a1ecf106 100644
--- a/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java
+++ b/pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImpl.java
@@ -143,8 +143,7 @@ public class PackagesManagementImpl implements PackagesManagement {
public CompletableFuture<Void> delete(PackageName packageName) {
return CompletableFuture.allOf(
storage.deleteAsync(metadataPath(packageName)),
- storage.deleteAsync(packagePath(packageName)),
- storage.deleteAsync(packageName.toRestPath()));
+ storage.deleteAsync(packagePath(packageName)));
}
@Override
@@ -244,12 +243,12 @@ public class PackagesManagementImpl implements PackagesManagement {
return future;
}
- private String metadataPath(PackageName packageName) {
+ protected String metadataPath(PackageName packageName) {
return packageName.toRestPath() + "/meta";
}
- private String packagePath(PackageName packageName) {
- return packageName.toRestPath() + "/data";
+ protected String packagePath(PackageName packageName) {
+ return packageName.toRestPath() + storage.dataPath();
}
private String packageWithoutVersionPath(PackageName packageName) {
diff --git a/pulsar-package-management/core/src/test/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImplTest.java b/pulsar-package-management/core/src/test/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImplTest.java
index 6dc28352c00..aba86501372 100644
--- a/pulsar-package-management/core/src/test/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImplTest.java
+++ b/pulsar-package-management/core/src/test/java/org/apache/pulsar/packages/management/core/impl/PackagesManagementImplTest.java
@@ -21,7 +21,10 @@ package org.apache.pulsar.packages.management.core.impl;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
import java.util.List;
+import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import org.apache.pulsar.packages.management.core.MockedPackagesStorageProvider;
import org.apache.pulsar.packages.management.core.PackagesManagement;
@@ -32,15 +35,15 @@ import org.apache.pulsar.packages.management.core.common.PackageMetadataUtil;
import org.apache.pulsar.packages.management.core.common.PackageName;
import org.apache.pulsar.packages.management.core.exceptions.PackagesManagementException;
import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class PackagesManagementImplTest {
private static PackagesStorage storage;
private static PackagesManagement packagesManagement;
- @BeforeClass
+ @BeforeMethod
public static void setup() throws IOException {
PackagesStorageProvider storageProvider = PackagesStorageProvider.newProvider(MockedPackagesStorageProvider.class.getName());
DefaultPackagesStorageConfiguration packagesStorageConfiguration = new DefaultPackagesStorageConfiguration();
@@ -50,7 +53,7 @@ public class PackagesManagementImplTest {
packagesManagement.initialize(storage);
}
- @AfterClass(alwaysRun = true)
+ @AfterMethod(alwaysRun = true)
public static void teardown() throws ExecutionException, InterruptedException {
storage.closeAsync().get();
}
@@ -192,4 +195,62 @@ public class PackagesManagementImplTest {
Assert.fail("should not throw any exception");
}
}
+
+ @Test
+ public void testPackagePath() {
+ PackagesManagementImpl impl = (PackagesManagementImpl) packagesManagement;
+ PackageName pn = PackageName.get("function://public/default/test@v1");
+ String metaPath = impl.metadataPath(pn);
+ Assert.assertEquals(metaPath, "function/public/default/test/v1/meta");
+ String dataPath = impl.packagePath(pn);
+ Assert.assertEquals(dataPath, "function/public/default/test/v1");
+
+ impl.initialize(new PackagesStorage() {
+ @Override
+ public void initialize() {
+
+ }
+
+ @Override
+ public CompletableFuture<Void> writeAsync(String path, InputStream inputStream) {
+ return null;
+ }
+
+ @Override
+ public CompletableFuture<Void> readAsync(String path, OutputStream outputStream) {
+ return null;
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAsync(String path) {
+ return null;
+ }
+
+ @Override
+ public CompletableFuture<List<String>> listAsync(String path) {
+ return null;
+ }
+
+ @Override
+ public CompletableFuture<Boolean> existAsync(String path) {
+ return null;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return null;
+ }
+
+ @Override
+ public String dataPath() {
+ return "/tmp";
+ }
+ });
+
+
+ metaPath = impl.metadataPath(pn);
+ Assert.assertEquals(metaPath, "function/public/default/test/v1/meta");
+ dataPath = impl.packagePath(pn);
+ Assert.assertEquals(dataPath, "function/public/default/test/v1/tmp");
+ }
}
diff --git a/pulsar-package-management/filesystem-storage/src/main/java/org/apache/pulsar/packages/management/storage/filesystem/FileSystemPackagesStorage.java b/pulsar-package-management/filesystem-storage/src/main/java/org/apache/pulsar/packages/management/storage/filesystem/FileSystemPackagesStorage.java
index ff34c482f15..74d77fe3572 100644
--- a/pulsar-package-management/filesystem-storage/src/main/java/org/apache/pulsar/packages/management/storage/filesystem/FileSystemPackagesStorage.java
+++ b/pulsar-package-management/filesystem-storage/src/main/java/org/apache/pulsar/packages/management/storage/filesystem/FileSystemPackagesStorage.java
@@ -147,4 +147,9 @@ public class FileSystemPackagesStorage implements PackagesStorage {
public CompletableFuture<Void> closeAsync() {
return CompletableFuture.completedFuture(null);
}
+
+ @Override
+ public String dataPath() {
+ return "/data";
+ }
}
[pulsar] 17/31: Support handling single role and non-jwt-token in MultiRolesTokenAuthorizationProvider (#14857)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit d3c08dba9199d5d4f19993ba946f67f7b61f9b3a
Author: Zike Yang <zi...@apache.org>
AuthorDate: Sun May 8 23:08:45 2022 +0800
Support handling single role and non-jwt-token in MultiRolesTokenAuthorizationProvider (#14857)
### Motivation
Currently, `MultiRolesTokenAuthorizationProvider` doesn't support handling the single string type role. It will return the empty role in that case. This PR adds support for handling the string-type role. This PR also adds support for handling the non-jwt-token.
### Modifications
* Add support for handling the string-type role
* Add support for handling the non-jwt-token
### Verifying this change
This change is already covered by existing tests, such as *testMultiRolesAuthzWithSingleRole*.
(cherry picked from commit 8bf6785c0803d314465b2d9156df6ca5bbb3c644)
---
.../MultiRolesTokenAuthorizationProvider.java | 8 +-
.../MultiRolesTokenAuthorizationProviderTest.java | 101 +++++++++++++++++++++
2 files changed, 107 insertions(+), 2 deletions(-)
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java
index 3c5d3a25e15..e43e184587d 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java
@@ -58,7 +58,7 @@ public class MultiRolesTokenAuthorizationProvider extends PulsarAuthorizationPro
// The token's claim that corresponds to the "role" string
static final String CONF_TOKEN_AUTH_CLAIM = "tokenAuthClaim";
- private JwtParser parser;
+ private final JwtParser parser;
private String roleClaim;
public MultiRolesTokenAuthorizationProvider() {
@@ -107,11 +107,15 @@ public class MultiRolesTokenAuthorizationProvider extends PulsarAuthorizationPro
}
String[] splitToken = token.split("\\.");
+ if (splitToken.length < 2) {
+ log.warn("Unable to extract additional roles from JWT token");
+ return Collections.emptyList();
+ }
String unsignedToken = splitToken[0] + "." + splitToken[1] + ".";
Jwt<?, Claims> jwt = parser.parseClaimsJwt(unsignedToken);
try {
- Collections.singletonList(jwt.getBody().get(roleClaim, String.class));
+ return Collections.singletonList(jwt.getBody().get(roleClaim, String.class));
} catch (RequiredTypeException requiredTypeException) {
try {
List list = jwt.getBody().get(roleClaim, List.class);
diff --git a/pulsar-broker-common/src/test/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProviderTest.java b/pulsar-broker-common/src/test/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProviderTest.java
index edd0baa42ae..078e2aad07a 100644
--- a/pulsar-broker-common/src/test/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProviderTest.java
+++ b/pulsar-broker-common/src/test/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProviderTest.java
@@ -18,10 +18,14 @@
*/
package org.apache.pulsar.broker.authorization;
+import static org.mockito.Mockito.mock;
import io.jsonwebtoken.Jwts;
import io.jsonwebtoken.SignatureAlgorithm;
+import java.util.Properties;
+import org.apache.pulsar.broker.ServiceConfiguration;
import org.apache.pulsar.broker.authentication.AuthenticationDataSource;
import org.apache.pulsar.broker.authentication.utils.AuthTokenUtils;
+import org.apache.pulsar.broker.resources.PulsarResources;
import org.junit.Assert;
import org.testng.annotations.Test;
@@ -96,4 +100,101 @@ public class MultiRolesTokenAuthorizationProviderTest {
Assert.assertFalse(provider.authorize(ads, role -> CompletableFuture.completedFuture(false)).get());
}
+
+ @Test
+ public void testMultiRolesAuthzWithSingleRole() throws Exception {
+ SecretKey secretKey = AuthTokenUtils.createSecretKey(SignatureAlgorithm.HS256);
+ String testRole = "test-role";
+ String token = Jwts.builder().claim("sub", testRole).signWith(secretKey).compact();
+
+ MultiRolesTokenAuthorizationProvider provider = new MultiRolesTokenAuthorizationProvider();
+
+ AuthenticationDataSource ads = new AuthenticationDataSource() {
+ @Override
+ public boolean hasDataFromHttp() {
+ return true;
+ }
+
+ @Override
+ public String getHttpHeader(String name) {
+ if (name.equals("Authorization")) {
+ return "Bearer " + token;
+ } else {
+ throw new IllegalArgumentException("Wrong HTTP header");
+ }
+ }
+ };
+
+ Assert.assertTrue(provider.authorize(ads, role -> {
+ if (role.equals(testRole)) {
+ return CompletableFuture.completedFuture(true);
+ }
+ return CompletableFuture.completedFuture(false);
+ }).get());
+ }
+
+ @Test
+ public void testMultiRolesNotFailNonJWT() throws Exception {
+ String token = "a-non-jwt-token";
+
+ MultiRolesTokenAuthorizationProvider provider = new MultiRolesTokenAuthorizationProvider();
+
+ AuthenticationDataSource ads = new AuthenticationDataSource() {
+ @Override
+ public boolean hasDataFromHttp() {
+ return true;
+ }
+
+ @Override
+ public String getHttpHeader(String name) {
+ if (name.equals("Authorization")) {
+ return "Bearer " + token;
+ } else {
+ throw new IllegalArgumentException("Wrong HTTP header");
+ }
+ }
+ };
+
+ Assert.assertFalse(provider.authorize(ads, role -> CompletableFuture.completedFuture(false)).get());
+ }
+
+ @Test
+ public void testMultiRolesAuthzWithCustomRolesClaims() throws Exception {
+ SecretKey secretKey = AuthTokenUtils.createSecretKey(SignatureAlgorithm.HS256);
+ String testRole = "test-role";
+ String customRolesClaims = "role";
+ String token = Jwts.builder().claim(customRolesClaims, new String[]{testRole}).signWith(secretKey).compact();
+
+ Properties properties = new Properties();
+ properties.setProperty("tokenSettingPrefix", "prefix_");
+ properties.setProperty("prefix_tokenAuthClaim", customRolesClaims);
+ ServiceConfiguration conf = new ServiceConfiguration();
+ conf.setProperties(properties);
+
+ MultiRolesTokenAuthorizationProvider provider = new MultiRolesTokenAuthorizationProvider();
+ provider.initialize(conf, mock(PulsarResources.class));
+
+ AuthenticationDataSource ads = new AuthenticationDataSource() {
+ @Override
+ public boolean hasDataFromHttp() {
+ return true;
+ }
+
+ @Override
+ public String getHttpHeader(String name) {
+ if (name.equals("Authorization")) {
+ return "Bearer " + token;
+ } else {
+ throw new IllegalArgumentException("Wrong HTTP header");
+ }
+ }
+ };
+
+ Assert.assertTrue(provider.authorize(ads, role -> {
+ if (role.equals(testRole)) {
+ return CompletableFuture.completedFuture(true);
+ }
+ return CompletableFuture.completedFuture(false);
+ }).get());
+ }
}
[pulsar] 29/31: [cleanup][broker] Override close method to avoid caching exception (#15529)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e3add83871c96dfdcbbb6b225a9400417ba54842
Author: Jiwei Guo <te...@apache.org>
AuthorDate: Wed May 11 15:38:15 2022 +0800
[cleanup][broker] Override close method to avoid caching exception (#15529)
(cherry picked from commit 526979a87e3cea5c4c90667ce86aed52927fd631)
---
.../main/java/org/apache/pulsar/broker/service/AbstractTopic.java | 4 +---
.../java/org/apache/pulsar/broker/service/PrecisPublishLimiter.java | 2 +-
.../java/org/apache/pulsar/broker/service/PublishRateLimiter.java | 5 +++++
.../org/apache/pulsar/broker/service/PublishRateLimiterDisable.java | 2 +-
.../org/apache/pulsar/broker/service/PublishRateLimiterImpl.java | 2 +-
.../pulsar/broker/service/nonpersistent/NonPersistentTopic.java | 6 +-----
.../apache/pulsar/broker/service/persistent/PersistentTopic.java | 6 +-----
7 files changed, 11 insertions(+), 16 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index a4983be4bd8..04fb3ca952c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -987,9 +987,7 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener<TopicP
}
// attach the resource-group level rate limiters, if set
- String rgName = policies.resource_group_name != null
- ? policies.resource_group_name
- : null;
+ String rgName = policies.resource_group_name;
if (rgName != null) {
final ResourceGroup resourceGroup =
brokerService.getPulsar().getResourceGroupServiceManager().resourceGroupGet(rgName);
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PrecisPublishLimiter.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PrecisPublishLimiter.java
index e61597e2d13..67cc46d95fa 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PrecisPublishLimiter.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PrecisPublishLimiter.java
@@ -133,7 +133,7 @@ public class PrecisPublishLimiter implements PublishRateLimiter {
}
@Override
- public void close() throws Exception {
+ public void close() {
rateLimitFunction.apply();
replaceLimiters(null);
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiter.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiter.java
index 397887978b2..931f35cfa1b 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiter.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiter.java
@@ -71,4 +71,9 @@ public interface PublishRateLimiter extends AutoCloseable {
* @param bytes
*/
boolean tryAcquire(int numbers, long bytes);
+
+ /**
+ * Close the limiter.
+ */
+ void close();
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterDisable.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterDisable.java
index 81c4b82317f..72c8132128e 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterDisable.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterDisable.java
@@ -63,7 +63,7 @@ public class PublishRateLimiterDisable implements PublishRateLimiter {
}
@Override
- public void close() throws Exception {
+ public void close() {
// No-op
}
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterImpl.java
index 0e1200edc31..f1646684b82 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterImpl.java
@@ -110,7 +110,7 @@ public class PublishRateLimiterImpl implements PublishRateLimiter {
}
@Override
- public void close() throws Exception {
+ public void close() {
// no-op
}
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
index 2ca106e7efb..61c9779dbcf 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
@@ -493,11 +493,7 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
replicators.forEach((cluster, replicator) -> futures.add(replicator.disconnect()));
producers.values().forEach(producer -> futures.add(producer.disconnect()));
if (topicPublishRateLimiter != null) {
- try {
- topicPublishRateLimiter.close();
- } catch (Exception e) {
- log.warn("Error closing topicPublishRateLimiter for topic {}", topic, e);
- }
+ topicPublishRateLimiter.close();
}
subscriptions.forEach((s, sub) -> futures.add(sub.disconnect()));
if (this.resourceGroupPublishLimiter != null) {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 1ef90c35841..6e7efc0a90a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -1272,11 +1272,7 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
replicators.forEach((cluster, replicator) -> futures.add(replicator.disconnect()));
producers.values().forEach(producer -> futures.add(producer.disconnect()));
if (topicPublishRateLimiter != null) {
- try {
- topicPublishRateLimiter.close();
- } catch (Exception e) {
- log.warn("Error closing topicPublishRateLimiter for topic {}", topic, e);
- }
+ topicPublishRateLimiter.close();
}
subscriptions.forEach((s, sub) -> futures.add(sub.disconnect()));
if (this.resourceGroupPublishLimiter != null) {
[pulsar] 03/31: [enh][monitor]: add metrics for pulsar web service thread pool (#14742)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a6eb62286fdc4fb0e6f00d94305ae604790d5cab
Author: Tao Jiuming <95...@users.noreply.github.com>
AuthorDate: Tue Apr 19 21:26:18 2022 +0800
[enh][monitor]: add metrics for pulsar web service thread pool (#14742)
Fixes https://github.com/apache/pulsar/issues/14459
See the issue
1. Add WebExecutorStats to record web thread pool metrics
(cherry picked from commit 32d7a51936aac72a1b22d5ed1e41f1658a6c618c)
---
.../PrometheusMetricsGeneratorUtils.java | 4 +-
.../apache/pulsar/broker/web/WebExecutorStats.java | 100 +++++++++++++++++++++
.../org/apache/pulsar/broker/web/WebService.java | 3 +
.../apache/pulsar/broker/web/WebServiceTest.java | 42 +++++++++
4 files changed, 148 insertions(+), 1 deletion(-)
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java
index 0e298151971..ead3c332b2b 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java
@@ -65,7 +65,9 @@ public class PrometheusMetricsGeneratorUtils {
for (int i = 0; i < metricFamily.samples.size(); i++) {
Collector.MetricFamilySamples.Sample sample = metricFamily.samples.get(i);
stream.write(sample.name);
- stream.write("{cluster=\"").write(cluster).write('"');
+ if (!sample.labelNames.contains("cluster")) {
+ stream.write("{cluster=\"").write(cluster).write('"');
+ }
for (int j = 0; j < sample.labelNames.size(); j++) {
String labelValue = sample.labelValues.get(j);
if (labelValue != null) {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebExecutorStats.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebExecutorStats.java
new file mode 100644
index 00000000000..1c89318305b
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebExecutorStats.java
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.web;
+
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Gauge;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+class WebExecutorStats implements AutoCloseable {
+ private static final AtomicBoolean CLOSED = new AtomicBoolean(false);
+
+ private final Gauge maxThreads;
+ private final Gauge minThreads;
+ private final Gauge idleThreads;
+ private final Gauge activeThreads;
+ private final Gauge currentThreads;
+ private final WebExecutorThreadPool executor;
+
+ private static volatile WebExecutorStats instance;
+
+ static synchronized WebExecutorStats getStats(WebExecutorThreadPool executor) {
+ if (null == instance) {
+ instance = new WebExecutorStats(executor);
+ }
+
+ return instance;
+ }
+
+ private WebExecutorStats(WebExecutorThreadPool executor) {
+ this.executor = executor;
+
+ this.maxThreads = Gauge.build("pulsar_web_executor_max_threads", "-").create()
+ .setChild(new Gauge.Child() {
+ public double get() {
+ return WebExecutorStats.this.executor.getMaxThreads();
+ }
+ })
+ .register();
+
+ this.minThreads = Gauge.build("pulsar_web_executor_min_threads", "-").create()
+ .setChild(new Gauge.Child() {
+ public double get() {
+ return WebExecutorStats.this.executor.getMinThreads();
+ }
+ })
+ .register();
+
+ this.idleThreads = Gauge.build("pulsar_web_executor_idle_threads", "-").create()
+ .setChild(new Gauge.Child() {
+ public double get() {
+ return WebExecutorStats.this.executor.getIdleThreads();
+ }
+ })
+ .register();
+
+ this.activeThreads = Gauge.build("pulsar_web_executor_active_threads", "-").create()
+ .setChild(new Gauge.Child() {
+ public double get() {
+ return WebExecutorStats.this.executor.getThreads()
+ - WebExecutorStats.this.executor.getIdleThreads();
+ }
+ })
+ .register();
+
+ this.currentThreads = Gauge.build("pulsar_web_executor_current_threads", "-").create()
+ .setChild(new Gauge.Child() {
+ public double get() {
+ return WebExecutorStats.this.executor.getThreads();
+ }
+ })
+ .register();
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (CLOSED.compareAndSet(false, true)) {
+ CollectorRegistry.defaultRegistry.unregister(this.activeThreads);
+ CollectorRegistry.defaultRegistry.unregister(this.maxThreads);
+ CollectorRegistry.defaultRegistry.unregister(this.minThreads);
+ CollectorRegistry.defaultRegistry.unregister(this.idleThreads);
+ CollectorRegistry.defaultRegistry.unregister(this.currentThreads);
+ }
+ }
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
index 7e6b1636a5c..fc800880b83 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java
@@ -66,6 +66,7 @@ public class WebService implements AutoCloseable {
private final PulsarService pulsar;
private final Server server;
private final List<Handler> handlers;
+ private final WebExecutorStats executorStats;
private final WebExecutorThreadPool webServiceExecutor;
public final int maxConcurrentRequests;
@@ -79,6 +80,7 @@ public class WebService implements AutoCloseable {
this.webServiceExecutor = new WebExecutorThreadPool(
pulsar.getConfiguration().getNumHttpServerThreads(),
"pulsar-web");
+ this.executorStats = WebExecutorStats.getStats(webServiceExecutor);
this.server = new Server(webServiceExecutor);
this.maxConcurrentRequests = pulsar.getConfiguration().getMaxConcurrentHttpRequests();
List<ServerConnector> connectors = new ArrayList<>();
@@ -273,6 +275,7 @@ public class WebService implements AutoCloseable {
jettyStatisticsCollector = null;
}
webServiceExecutor.join();
+ this.executorStats.close();
log.info("Web service closed");
} catch (Exception e) {
throw new PulsarServerException(e);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
index ae611d7e4eb..0cd72f19a4d 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/web/WebServiceTest.java
@@ -23,10 +23,12 @@ import static org.mockito.Mockito.doReturn;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
+import com.google.common.collect.Multimap;
import com.google.common.collect.Sets;
import com.google.common.io.CharStreams;
import com.google.common.io.Closeables;
import io.netty.handler.ssl.util.InsecureTrustManagerFactory;
+import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
@@ -35,6 +37,7 @@ import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.SecureRandom;
import java.security.cert.Certificate;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
@@ -52,6 +55,8 @@ import org.apache.pulsar.broker.MockedBookKeeperClientFactory;
import org.apache.pulsar.broker.PulsarService;
import org.apache.pulsar.broker.ServiceConfiguration;
import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.stats.PrometheusMetricsTest;
+import org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator;
import org.apache.pulsar.client.admin.PulsarAdmin;
import org.apache.pulsar.client.admin.PulsarAdminBuilder;
import org.apache.pulsar.client.admin.PulsarAdminException.ConflictException;
@@ -88,6 +93,43 @@ public class WebServiceTest {
private static final String TLS_CLIENT_CERT_FILE_PATH = "./src/test/resources/certificate/client.crt";
private static final String TLS_CLIENT_KEY_FILE_PATH = "./src/test/resources/certificate/client.key";
+
+ @Test
+ public void testWebExecutorMetrics() throws Exception {
+ setupEnv(true, "1.0", true, false, false, false, -1, false);
+ ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+ PrometheusMetricsGenerator.generate(pulsar, false, false, false, statsOut);
+ String metricsStr = statsOut.toString();
+ Multimap<String, PrometheusMetricsTest.Metric> metrics = PrometheusMetricsTest.parseMetrics(metricsStr);
+
+ Collection<PrometheusMetricsTest.Metric> maxThreads = metrics.get("pulsar_web_executor_max_threads");
+ Collection<PrometheusMetricsTest.Metric> minThreads = metrics.get("pulsar_web_executor_min_threads");
+ Collection<PrometheusMetricsTest.Metric> activeThreads = metrics.get("pulsar_web_executor_active_threads");
+ Collection<PrometheusMetricsTest.Metric> idleThreads = metrics.get("pulsar_web_executor_idle_threads");
+ Collection<PrometheusMetricsTest.Metric> currentThreads = metrics.get("pulsar_web_executor_current_threads");
+
+ for (PrometheusMetricsTest.Metric metric : maxThreads) {
+ Assert.assertNotNull(metric.tags.get("cluster"));
+ Assert.assertTrue(metric.value > 0);
+ }
+ for (PrometheusMetricsTest.Metric metric : minThreads) {
+ Assert.assertNotNull(metric.tags.get("cluster"));
+ Assert.assertTrue(metric.value > 0);
+ }
+ for (PrometheusMetricsTest.Metric metric : activeThreads) {
+ Assert.assertNotNull(metric.tags.get("cluster"));
+ Assert.assertTrue(metric.value >= 0);
+ }
+ for (PrometheusMetricsTest.Metric metric : idleThreads) {
+ Assert.assertNotNull(metric.tags.get("cluster"));
+ Assert.assertTrue(metric.value >= 0);
+ }
+ for (PrometheusMetricsTest.Metric metric : currentThreads) {
+ Assert.assertNotNull(metric.tags.get("cluster"));
+ Assert.assertTrue(metric.value > 0);
+ }
+ }
+
/**
* Test that the {@WebService} class properly passes the allowUnversionedClients value. We do this by setting
* allowUnversionedClients to true, then making a request with no version, which should go through.
[pulsar] 19/31: [fix][broker] Fix MultiRolesTokenAuthorizationProvider `authorize` issue. (#15454)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 836ed1e25fcfecb86ea162f3b35b8978ed7e406a
Author: Jiwei Guo <te...@apache.org>
AuthorDate: Mon May 9 09:11:19 2022 +0800
[fix][broker] Fix MultiRolesTokenAuthorizationProvider `authorize` issue. (#15454)
(cherry picked from commit 19f61d53b88bb195fabb367be722694902c79d22)
---
.../MultiRolesTokenAuthorizationProvider.java | 27 +------------
.../org/apache/pulsar/common/util/FutureUtil.java | 44 ++++++++++++++++++++++
.../apache/pulsar/common/util/FutureUtilTest.java | 43 +++++++++++++++++++++
3 files changed, 88 insertions(+), 26 deletions(-)
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java
index e43e184587d..1c710094ce1 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/MultiRolesTokenAuthorizationProvider.java
@@ -28,7 +28,6 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ExecutionException;
import java.util.function.Function;
import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.broker.ServiceConfiguration;
@@ -138,31 +137,7 @@ public class MultiRolesTokenAuthorizationProvider extends PulsarAuthorizationPro
}
List<CompletableFuture<Boolean>> futures = new ArrayList<>(roles.size());
roles.forEach(r -> futures.add(authorizeFunc.apply(r)));
- return CompletableFuture.supplyAsync(() -> {
- do {
- try {
- List<CompletableFuture<Boolean>> doneFutures = new ArrayList<>();
- FutureUtil.waitForAny(futures).get();
- for (CompletableFuture<Boolean> future : futures) {
- if (!future.isDone()) {
- continue;
- }
- doneFutures.add(future);
- if (future.get()) {
- futures.forEach(f -> {
- if (!f.isDone()) {
- f.cancel(false);
- }
- });
- return true;
- }
- }
- futures.removeAll(doneFutures);
- } catch (InterruptedException | ExecutionException ignored) {
- }
- } while (!futures.isEmpty());
- return false;
- });
+ return FutureUtil.waitForAny(futures, ret -> (boolean) ret).thenApply(v -> v.isPresent());
}
/**
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
index a29ac8c2ee8..f51888a76df 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
@@ -28,7 +28,9 @@ import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
import java.util.function.Supplier;
+import java.util.stream.Collectors;
/**
* This class is aimed at simplifying work with {@code CompletableFuture}.
@@ -55,6 +57,48 @@ public class FutureUtil {
return CompletableFuture.anyOf(futures.toArray(new CompletableFuture[0]));
}
+ /**
+ * Return a future that represents the completion of any future that match the predicate in the provided Collection.
+ *
+ * @param futures futures to wait any
+ * @param tester if any future match the predicate
+ * @return a new CompletableFuture that is completed when any of the given CompletableFutures match the tester
+ */
+ public static CompletableFuture<Optional<Object>> waitForAny(Collection<? extends CompletableFuture<?>> futures,
+ Predicate<Object> tester) {
+ return waitForAny(futures).thenCompose(v -> {
+ if (tester.test(v)) {
+ futures.forEach(f -> {
+ if (!f.isDone()) {
+ f.cancel(true);
+ }
+ });
+ return CompletableFuture.completedFuture(Optional.of(v));
+ }
+ Collection<CompletableFuture<?>> doneFutures = futures.stream()
+ .filter(f -> f.isDone())
+ .collect(Collectors.toList());
+ futures.removeAll(doneFutures);
+ Optional<?> value = doneFutures.stream()
+ .filter(f -> !f.isCompletedExceptionally())
+ .map(CompletableFuture::join)
+ .filter(tester)
+ .findFirst();
+ if (!value.isPresent()) {
+ if (futures.size() == 0) {
+ return CompletableFuture.completedFuture(Optional.empty());
+ }
+ return waitForAny(futures, tester);
+ }
+ futures.forEach(f -> {
+ if (!f.isDone()) {
+ f.cancel(true);
+ }
+ });
+ return CompletableFuture.completedFuture(Optional.of(value.get()));
+ });
+ }
+
/**
* Return a future that represents the completion of the futures in the provided list.
diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java b/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java
index 41d822eec07..7929672fa33 100644
--- a/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java
+++ b/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java
@@ -23,7 +23,9 @@ import java.io.PrintWriter;
import java.io.StringWriter;
import java.time.Duration;
import java.util.List;
+import java.util.Optional;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
@@ -33,6 +35,7 @@ import org.assertj.core.util.Lists;
import org.awaitility.Awaitility;
import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
@@ -136,4 +139,44 @@ public class FutureUtilTest {
assertTrue(future3Exception.get(0) instanceof IllegalStateException);
});
}
+
+ @Test
+ public void testWaitForAny() {
+ CompletableFuture<String> f1 = new CompletableFuture<>();
+ CompletableFuture<String> f2 = new CompletableFuture<>();
+ CompletableFuture<String> f3 = new CompletableFuture<>();
+ CompletableFuture<String> f4 = new CompletableFuture<>();
+ f1.complete("1");
+ f2.complete("2");
+ f3.complete("3");
+ f4.complete("4");
+ CompletableFuture<Optional<Object>> ret = FutureUtil.waitForAny(Lists.newArrayList(f1, f2, f3, f4), p -> p.equals("3"));
+ assertEquals(ret.join().get(), "3");
+ // test not matched predicate result
+ CompletableFuture<String> f5 = new CompletableFuture<>();
+ CompletableFuture<String> f6 = new CompletableFuture<>();
+ f5.complete("5");
+ f6.complete("6");
+ ret = FutureUtil.waitForAny(Lists.newArrayList(f5, f6), p -> p.equals("3"));
+ assertFalse(ret.join().isPresent());
+ // test one complete, others are cancelled.
+ CompletableFuture<String> f55 = new CompletableFuture<>();
+ CompletableFuture<String> f66 = new CompletableFuture<>();
+ f55.complete("55");
+ ret = FutureUtil.waitForAny(Lists.newArrayList(f55, f66), p -> p.equals("55"));
+ assertTrue(ret.join().isPresent());
+ assertTrue(f66.isCancelled());
+ // test with exception
+ CompletableFuture<String> f7 = new CompletableFuture<>();
+ CompletableFuture<String> f8 = new CompletableFuture<>();
+ f8.completeExceptionally(new RuntimeException("f7 exception"));
+ f8.completeExceptionally(new RuntimeException("f8 exception"));
+ ret = FutureUtil.waitForAny(Lists.newArrayList(f7, f8), p -> p.equals("3"));
+ try {
+ ret.join();
+ fail("Should have failed");
+ } catch (CompletionException ex) {
+ assertTrue(ex.getCause() instanceof RuntimeException);
+ }
+ }
}
\ No newline at end of file
[pulsar] 24/31: Fix http produce msg redirect issue. (#15551)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 530fafccaec8e665cf51fca2689f7b7cf827aeef
Author: Jiwei Guo <te...@apache.org>
AuthorDate: Fri May 13 10:36:33 2022 +0800
Fix http produce msg redirect issue. (#15551)
Master Issue: #15546
### Motivation
When lookup the topic ownership using REST produce, the redirect URI is incorrect, because :
```
uri.getPath(false); //Get the path of the current request relative to the base URI as a string.
```
So the redirect URI does not contain the base path:
```
URI redirectURI = new URI(String.format("%s%s", redirectAddresses.get(0), uri.getPath(false)))
```
(cherry picked from commit 7f976da1b51cd868ec49b5ab43259fea4d48c8e9)
---
.../main/java/org/apache/pulsar/broker/rest/TopicsBase.java | 11 ++++++++---
.../test/java/org/apache/pulsar/broker/admin/TopicsTest.java | 8 ++++----
2 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest/TopicsBase.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest/TopicsBase.java
index 770d77794d5..86e8956d950 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest/TopicsBase.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest/TopicsBase.java
@@ -21,7 +21,7 @@ package org.apache.pulsar.broker.rest;
import io.netty.buffer.ByteBuf;
import java.io.IOException;
import java.net.URI;
-import java.net.URISyntaxException;
+import java.net.URL;
import java.nio.ByteBuffer;
import java.sql.Time;
import java.sql.Timestamp;
@@ -41,6 +41,7 @@ import java.util.stream.Collectors;
import javax.ws.rs.container.AsyncResponse;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
+import javax.ws.rs.core.UriBuilder;
import lombok.extern.slf4j.Slf4j;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericDatumReader;
@@ -378,10 +379,14 @@ public class TopicsBase extends PersistentTopicsBase {
log.debug("Redirect rest produce request for topic {} from {} to {}.",
topicName, pulsar().getWebServiceAddress(), redirectAddresses.get(0));
}
- URI redirectURI = new URI(String.format("%s%s", redirectAddresses.get(0), uri.getPath(false)));
+ URL redirectAddress = new URL(redirectAddresses.get(0));
+ URI redirectURI = UriBuilder.fromUri(uri.getRequestUri())
+ .host(redirectAddress.getHost())
+ .port(redirectAddress.getPort())
+ .build();
asyncResponse.resume(Response.temporaryRedirect(redirectURI).build());
future.complete(true);
- } catch (URISyntaxException | NullPointerException e) {
+ } catch (Exception e) {
if (log.isDebugEnabled()) {
log.error("Error in preparing redirect url with rest produce message request for topic {}: {}",
topicName, e.getMessage(), e);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicsTest.java
index 4edfbfd17db..b0273714c01 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicsTest.java
@@ -77,6 +77,7 @@ import javax.ws.rs.container.AsyncResponse;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;
import java.io.ByteArrayOutputStream;
+import java.net.URI;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
@@ -313,13 +314,13 @@ public class TopicsTest extends MockedPulsarServiceBaseTest {
@Test
public void testLookUpWithRedirect() throws Exception {
String topicName = "persistent://" + testTenant + "/" + testNamespace + "/" + testTopicName;
- String requestPath = "/admin/v3/topics/my-tenant/my-namespace/my-topic";
+ URI requestPath = URI.create(pulsar.getWebServiceAddress() + "/topics/my-tenant/my-namespace/my-topic");
//create topic on one broker
admin.topics().createNonPartitionedTopic(topicName);
PulsarService pulsar2 = startBroker(getDefaultConf());
doReturn(false).when(topics).isRequestHttps();
UriInfo uriInfo = mock(UriInfo.class);
- doReturn(requestPath).when(uriInfo).getPath(anyBoolean());
+ doReturn(requestPath).when(uriInfo).getRequestUri();
Whitebox.setInternalState(topics, "uri", uriInfo);
//do produce on another broker
topics.setPulsar(pulsar2);
@@ -336,8 +337,7 @@ public class TopicsTest extends MockedPulsarServiceBaseTest {
// Verify got redirect response
Assert.assertEquals(responseCaptor.getValue().getStatusInfo(), Response.Status.TEMPORARY_REDIRECT);
// Verify URI point to address of broker the topic was created on
- Assert.assertEquals(responseCaptor.getValue().getLocation().toString(),
- pulsar.getWebServiceAddress() + requestPath);
+ Assert.assertEquals(responseCaptor.getValue().getLocation().toString(), requestPath.toString());
}
@Test
[pulsar] 12/31: [improve][java-client] Add pending messages information while print the producer stats (#15440)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 5a9b56a9865399ed2577abb43940a53118a1d49f
Author: lipenghui <pe...@apache.org>
AuthorDate: Thu May 5 18:37:38 2022 +0800
[improve][java-client] Add pending messages information while print the producer stats (#15440)
(cherry picked from commit fbe650ce72462e97ca1ba8f9dcb41ab1b7ce47bd)
---
.../org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java
index 29b8cf28c31..1f1b5b2efe7 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerStatsRecorderImpl.java
@@ -182,7 +182,8 @@ public class ProducerStatsRecorderImpl implements ProducerStatsRecorder {
+ "BatchSize: med: {} - 95pct: {} - 99pct: {} - 99.9pct: {} - max: {} --- "
+ "MsgSize: med: {} bytes - 95pct: {} bytes - 99pct: {} bytes - 99.9pct: {} bytes "
+ "- max: {} bytes --- "
- + "Ack received rate: {} ack/s --- Failed messages: {}", producer.getTopic(),
+ + "Ack received rate: {} ack/s --- Failed messages: {} --- Pending messages: {}",
+ producer.getTopic(),
producer.getProducerName(), producer.getPendingQueueSize(),
THROUGHPUT_FORMAT.format(sendMsgsRate),
THROUGHPUT_FORMAT.format(sendBytesRate / 1024 / 1024 * 8),
@@ -195,7 +196,8 @@ public class ProducerStatsRecorderImpl implements ProducerStatsRecorder {
DEC.format(msgSizePctValues[0]), DEC.format(msgSizePctValues[2]),
DEC.format(msgSizePctValues[3]), DEC.format(msgSizePctValues[4]),
DEC.format(msgSizePctValues[5]),
- THROUGHPUT_FORMAT.format(currentNumAcksReceived / elapsed), currentNumSendFailedMsgs);
+ THROUGHPUT_FORMAT.format(currentNumAcksReceived / elapsed), currentNumSendFailedMsgs,
+ getPendingQueueSize());
}
}
[pulsar] 06/31: [Broker] Fix typo in enum name and handle closing of the channel properly since writeAndFlush is asynchronous (#15384)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit f829ca99068736fee5b621e7464553b8d6a11feb
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sat Apr 30 04:57:42 2022 +0300
[Broker] Fix typo in enum name and handle closing of the channel properly since writeAndFlush is asynchronous (#15384)
(cherry picked from commit cd3816aa351ba8a1f0e9876eefe019b7f0d282d8)
---
.../pulsar/broker/service/ConnectionController.java | 16 ++++++++--------
.../java/org/apache/pulsar/broker/service/ServerCnx.java | 15 ++++++++-------
2 files changed, 16 insertions(+), 15 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ConnectionController.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ConnectionController.java
index 51540e179be..65c3a6c4f2a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ConnectionController.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ConnectionController.java
@@ -36,7 +36,7 @@ public interface ConnectionController {
* @param remoteAddress
* @return
*/
- Sate increaseConnection(SocketAddress remoteAddress);
+ State increaseConnection(SocketAddress remoteAddress);
/**
* Decrease the number of connections counter.
@@ -44,7 +44,7 @@ public interface ConnectionController {
*/
void decreaseConnection(SocketAddress remoteAddress);
- enum Sate {
+ enum State {
OK, REACH_MAX_CONNECTION_PER_IP, REACH_MAX_CONNECTION;
}
@@ -68,13 +68,13 @@ public interface ConnectionController {
}
@Override
- public Sate increaseConnection(SocketAddress remoteAddress) {
+ public State increaseConnection(SocketAddress remoteAddress) {
if (!maxConnectionsLimitEnabled && !maxConnectionsLimitPerIpEnabled) {
- return Sate.OK;
+ return State.OK;
}
if (!(remoteAddress instanceof InetSocketAddress)
|| !isLegalIpAddress(((InetSocketAddress) remoteAddress).getHostString())) {
- return Sate.OK;
+ return State.OK;
}
lock.lock();
try {
@@ -88,20 +88,20 @@ public interface ConnectionController {
if (maxConnectionsLimitEnabled && totalConnectionNum > maxConnections) {
log.info("Reject connect request from {}, because reached the maximum number of connections {}",
remoteAddress, totalConnectionNum);
- return Sate.REACH_MAX_CONNECTION;
+ return State.REACH_MAX_CONNECTION;
}
if (maxConnectionsLimitPerIpEnabled && CONNECTIONS.get(ip).getValue() > maxConnectionPerIp) {
log.info("Reject connect request from {}, because reached the maximum number "
+ "of connections per Ip {}",
remoteAddress, CONNECTIONS.get(ip).getValue());
- return Sate.REACH_MAX_CONNECTION_PER_IP;
+ return State.REACH_MAX_CONNECTION_PER_IP;
}
} catch (Exception e) {
log.error("increase connection failed", e);
} finally {
lock.unlock();
}
- return Sate.OK;
+ return State.OK;
}
@Override
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
index 7fa6c9dde8a..10a4a2c412d 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
@@ -28,6 +28,7 @@ import static org.apache.pulsar.common.protocol.Commands.newLookupErrorResponse;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFutureListener;
import io.netty.channel.ChannelHandler;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelOption;
@@ -272,13 +273,13 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
- ConnectionController.Sate sate = connectionController.increaseConnection(remoteAddress);
- if (!sate.equals(ConnectionController.Sate.OK)) {
- ctx.channel().writeAndFlush(Commands.newError(-1, ServerError.NotAllowedError,
- sate.equals(ConnectionController.Sate.REACH_MAX_CONNECTION)
- ? "Reached the maximum number of connections"
- : "Reached the maximum number of connections on address" + remoteAddress));
- ctx.channel().close();
+ ConnectionController.State state = connectionController.increaseConnection(remoteAddress);
+ if (!state.equals(ConnectionController.State.OK)) {
+ ctx.writeAndFlush(Commands.newError(-1, ServerError.NotAllowedError,
+ state.equals(ConnectionController.State.REACH_MAX_CONNECTION)
+ ? "Reached the maximum number of connections"
+ : "Reached the maximum number of connections on address" + remoteAddress))
+ .addListener(ChannelFutureListener.CLOSE);
return;
}
log.info("New connection from {}", remoteAddress);
[pulsar] 30/31: Fix potential to add duplicated consumer (#15051)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 9dead563ad0218063461fb03d40618c5b09c9ab4
Author: fengyubiao <99...@qq.com>
AuthorDate: Wed May 18 18:41:08 2022 +0800
Fix potential to add duplicated consumer (#15051)
It's because of this issue https://github.com/apache/pulsar/issues/13787.
Then diving into the codes, I find that if the client tries to subscribe multiple times over a short period of time, it is possible to have more than one consumer at the same dispatcher. just like below:
```
for ( long requestId = 1; i < 5; i++ ){
ByteBuf request1 = Commands.newSubscribe(topic, subscription, consumerId, requestId , getSubType(),
priorityLevel, consumerName, isDurable, startMessageIdData, metadata, readCompacted,
conf.isReplicateSubscriptionState(),
InitialPosition.valueOf(subscriptionInitialPosition.getValue()),
startMessageRollbackDuration, si, createTopicIfDoesNotExist, conf.getKeySharedPolicy(),
// Use the current epoch to subscribe.
conf.getSubscriptionProperties(), CONSUMER_EPOCH.get(this));
cnx.sendRequestWithId(request1, requestId).thenRun(() -> {});
}
```
The root cause is below snippet:
https://github.com/apache/pulsar/blob/c2c05c49aff1ebc7b2b7a1d5bd547c33211e4479/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L994-L1021
If the consumer1 comes and is not done, then the same consumer2(it's the same with consumer1) comes, it may remove the prior consumer1(line 1015), but consumer1 add to subscription success in the end, Then the same cusumer3 comes, and it succeed, and will cause the same consumer to add duplicated.
The right way to remove consumer (line 1015) is when the `existingConsumerFuture` is completedExceptionally.
Even though the Java client couldn't occur the above behavior, other clients may not. So it's better to handle `subscribe` correctly on the broker side.
Modify the process execution sequence to improve stability
(cherry picked from commit 7bf495aca70798257d79d370f33c3870f31815a9)
---
.../apache/pulsar/broker/service/ServerCnx.java | 45 ++++---
.../pulsar/broker/service/ServerCnxTest.java | 147 +++++++++++++++++++++
.../TopicTransactionBufferRecoverTest.java | 2 +-
3 files changed, 175 insertions(+), 19 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
index 10a4a2c412d..0c102a1198d 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
@@ -992,32 +992,31 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
}
if (existingConsumerFuture != null) {
- if (existingConsumerFuture.isDone() && !existingConsumerFuture.isCompletedExceptionally()) {
- Consumer consumer = existingConsumerFuture.getNow(null);
- log.info("[{}] Consumer with the same id is already created:"
- + " consumerId={}, consumer={}",
- remoteAddress, consumerId, consumer);
- commandSender.sendSuccessResponse(requestId);
- return null;
- } else {
+ if (!existingConsumerFuture.isDone()){
// There was an early request to create a consumer with same consumerId. This can happen
// when
// client timeout is lower the broker timeouts. We need to wait until the previous
// consumer
// creation request either complete or fails.
log.warn("[{}][{}][{}] Consumer with id is already present on the connection,"
- + " consumerId={}", remoteAddress, topicName, subscriptionName, consumerId);
- ServerError error = null;
- if (!existingConsumerFuture.isDone()) {
- error = ServerError.ServiceNotReady;
- } else {
- error = getErrorCode(existingConsumerFuture);
- consumers.remove(consumerId, existingConsumerFuture);
- }
- commandSender.sendErrorResponse(requestId, error,
+ + " consumerId={}", remoteAddress, topicName, subscriptionName, consumerId);
+ commandSender.sendErrorResponse(requestId, ServerError.ServiceNotReady,
"Consumer is already present on the connection");
- return null;
+ } else if (existingConsumerFuture.isCompletedExceptionally()){
+ ServerError error = getErrorCodeWithErrorLog(existingConsumerFuture, true,
+ String.format("Consumer subscribe failure. remoteAddress: %s, subscription: %s",
+ remoteAddress, subscriptionName));
+ consumers.remove(consumerId, existingConsumerFuture);
+ commandSender.sendErrorResponse(requestId, error,
+ "Consumer that failed is already present on the connection");
+ } else {
+ Consumer consumer = existingConsumerFuture.getNow(null);
+ log.info("[{}] Consumer with the same id is already created:"
+ + " consumerId={}, consumer={}",
+ remoteAddress, consumerId, consumer);
+ commandSender.sendSuccessResponse(requestId);
}
+ return null;
}
boolean createTopicIfDoesNotExist = forceTopicCreation
@@ -2711,6 +2710,11 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
}
private <T> ServerError getErrorCode(CompletableFuture<T> future) {
+ return getErrorCodeWithErrorLog(future, false, null);
+ }
+
+ private <T> ServerError getErrorCodeWithErrorLog(CompletableFuture<T> future, boolean logIfError,
+ String errorMessageIfLog) {
ServerError error = ServerError.UnknownError;
try {
future.getNow(null);
@@ -2718,6 +2722,11 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
if (e.getCause() instanceof BrokerServiceException) {
error = BrokerServiceException.getClientErrorCode(e.getCause());
}
+ if (logIfError){
+ String finalErrorMessage = StringUtils.isNotBlank(errorMessageIfLog)
+ ? errorMessageIfLog : "Unknown Error";
+ log.error(finalErrorMessage, e);
+ }
}
return error;
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java
index 1772581baf5..5bef206a44d 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java
@@ -51,6 +51,8 @@ import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
import java.util.function.Supplier;
import org.apache.bookkeeper.common.util.OrderedExecutor;
import org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback;
@@ -107,10 +109,13 @@ import org.apache.pulsar.common.protocol.Commands;
import org.apache.pulsar.common.protocol.Commands.ChecksumType;
import org.apache.pulsar.common.protocol.PulsarHandler;
import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentLongHashMap;
import org.apache.pulsar.metadata.api.extended.MetadataStoreExtended;
import org.apache.pulsar.metadata.impl.ZKMetadataStore;
import org.apache.zookeeper.ZooKeeper;
import org.awaitility.Awaitility;
+import org.mockito.ArgumentCaptor;
+import org.mockito.MockedStatic;
import org.mockito.Mockito;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.stubbing.Answer;
@@ -1827,4 +1832,146 @@ public class ServerCnxTest {
channel.finish();
}
+
+ @Test
+ public void testNeverDelayConsumerFutureWhenNotFail() throws Exception{
+ // Mock ServerCnx.field: consumers
+ ConcurrentLongHashMap.Builder mapBuilder = Mockito.mock(ConcurrentLongHashMap.Builder.class);
+ Mockito.when(mapBuilder.expectedItems(Mockito.anyInt())).thenReturn(mapBuilder);
+ Mockito.when(mapBuilder.concurrencyLevel(Mockito.anyInt())).thenReturn(mapBuilder);
+ ConcurrentLongHashMap consumers = Mockito.mock(ConcurrentLongHashMap.class);
+ Mockito.when(mapBuilder.build()).thenReturn(consumers);
+ ArgumentCaptor<Long> ignoreArgumentCaptor = ArgumentCaptor.forClass(Long.class);
+ final ArgumentCaptor<CompletableFuture> deleteTimesMark = ArgumentCaptor.forClass(CompletableFuture.class);
+ Mockito.when(consumers.remove(ignoreArgumentCaptor.capture())).thenReturn(true);
+ Mockito.when(consumers.remove(ignoreArgumentCaptor.capture(), deleteTimesMark.capture())).thenReturn(true);
+ // case1: exists existingConsumerFuture, already complete or delay done after execute 'isDone()' many times
+ // case2: exists existingConsumerFuture, delay complete after execute 'isDone()' many times
+ // Why is the design so complicated, see: https://github.com/apache/pulsar/pull/15051
+ // Try a delay of 3 stages. The simulation is successful after repeated judgments.
+ for(AtomicInteger futureWillDoneAfterDelayTimes = new AtomicInteger(1);
+ futureWillDoneAfterDelayTimes.intValue() <= 3;
+ futureWillDoneAfterDelayTimes.incrementAndGet()){
+ final AtomicInteger futureCallTimes = new AtomicInteger();
+ final Consumer mockConsumer = Mockito.mock(Consumer.class);
+ CompletableFuture existingConsumerFuture = new CompletableFuture<Consumer>(){
+
+ private boolean complete;
+
+ // delay complete after execute 'isDone()' many times
+ @Override
+ public boolean isDone() {
+ if (complete) {
+ return true;
+ }
+ int executeIsDoneCommandTimes = futureCallTimes.incrementAndGet();
+ return executeIsDoneCommandTimes >= futureWillDoneAfterDelayTimes.intValue();
+ }
+
+ // if trig "getNow()", then complete
+ @Override
+ public Consumer get(){
+ complete = true;
+ return mockConsumer;
+ }
+
+ // if trig "get()", then complete
+ @Override
+ public Consumer get(long timeout, TimeUnit unit){
+ complete = true;
+ return mockConsumer;
+ }
+
+ // if trig "get()", then complete
+ @Override
+ public Consumer getNow(Consumer ifAbsent){
+ complete = true;
+ return mockConsumer;
+ }
+
+ // never fail
+ public boolean isCompletedExceptionally(){
+ return false;
+ }
+ };
+ Mockito.when(consumers.putIfAbsent(Mockito.anyLong(), Mockito.any())).thenReturn(existingConsumerFuture);
+ // do test: delay complete after execute 'isDone()' many times
+ // Why is the design so complicated, see: https://github.com/apache/pulsar/pull/15051
+ try (MockedStatic<ConcurrentLongHashMap> theMock = Mockito.mockStatic(ConcurrentLongHashMap.class)) {
+ // Inject consumers to ServerCnx
+ theMock.when(ConcurrentLongHashMap::newBuilder).thenReturn(mapBuilder);
+ // reset channels( serverChannel, clientChannel )
+ resetChannel();
+ setChannelConnected();
+ // auth check disable
+ doReturn(false).when(brokerService).isAuthenticationEnabled();
+ doReturn(false).when(brokerService).isAuthorizationEnabled();
+ // do subscribe
+ ByteBuf clientCommand = Commands.newSubscribe(successTopicName, //
+ successSubName, 1 /* consumer id */, 1 /* request id */, SubType.Exclusive, 0,
+ "test" /* consumer name */, 0 /* avoid reseting cursor */);
+ channel.writeInbound(clientCommand);
+ Object responseObj = getResponse();
+ Predicate<Object> responseAssert = obj -> {
+ if (responseObj instanceof CommandSuccess) {
+ return true;
+ }
+ if (responseObj instanceof CommandError) {
+ CommandError commandError = (CommandError) responseObj;
+ return ServerError.ServiceNotReady == commandError.getError();
+ }
+ return false;
+ };
+ // assert no consumer-delete event occur
+ assertFalse(deleteTimesMark.getAllValues().contains(existingConsumerFuture));
+ // assert without another error occur
+ assertTrue(responseAssert.test(responseAssert));
+ // Server will not close the connection
+ assertTrue(channel.isOpen());
+ channel.finish();
+ }
+ }
+ // case3: exists existingConsumerFuture, already complete and exception
+ CompletableFuture existingConsumerFuture = Mockito.mock(CompletableFuture.class);
+ Mockito.when(consumers.putIfAbsent(Mockito.anyLong(), Mockito.any())).thenReturn(existingConsumerFuture);
+ // make consumerFuture delay finish
+ Mockito.when(existingConsumerFuture.isDone()).thenReturn(true);
+ // when sync get return, future will return success value.
+ Mockito.when(existingConsumerFuture.get()).thenThrow(new NullPointerException());
+ Mockito.when(existingConsumerFuture.get(Mockito.anyLong(), Mockito.any())).
+ thenThrow(new NullPointerException());
+ Mockito.when(existingConsumerFuture.isCompletedExceptionally()).thenReturn(true);
+ Mockito.when(existingConsumerFuture.getNow(Mockito.any())).thenThrow(new NullPointerException());
+ try (MockedStatic<ConcurrentLongHashMap> theMock = Mockito.mockStatic(ConcurrentLongHashMap.class)) {
+ // Inject consumers to ServerCnx
+ theMock.when(ConcurrentLongHashMap::newBuilder).thenReturn(mapBuilder);
+ // reset channels( serverChannel, clientChannel )
+ resetChannel();
+ setChannelConnected();
+ // auth check disable
+ doReturn(false).when(brokerService).isAuthenticationEnabled();
+ doReturn(false).when(brokerService).isAuthorizationEnabled();
+ // do subscribe
+ ByteBuf clientCommand = Commands.newSubscribe(successTopicName, //
+ successSubName, 1 /* consumer id */, 1 /* request id */, SubType.Exclusive, 0,
+ "test" /* consumer name */, 0 /* avoid reseting cursor */);
+ channel.writeInbound(clientCommand);
+ Object responseObj = getResponse();
+ Predicate<Object> responseAssert = obj -> {
+ if (responseObj instanceof CommandError) {
+ CommandError commandError = (CommandError) responseObj;
+ return ServerError.ServiceNotReady != commandError.getError();
+ }
+ return false;
+ };
+ // assert error response
+ assertTrue(responseAssert.test(responseAssert));
+ // assert consumer-delete event occur
+ assertEquals(1L,
+ deleteTimesMark.getAllValues().stream().filter(f -> f == existingConsumerFuture).count());
+ // Server will not close the connection
+ assertTrue(channel.isOpen());
+ channel.finish();
+ }
+ }
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
index 351fe124852..9f6d4eb121b 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java
@@ -493,7 +493,7 @@ public class TopicTransactionBufferRecoverTest extends TransactionTestBase {
doThrow(new RuntimeException("test")).when(reader).hasMoreEvents();
// check reader close topic
checkCloseTopic(pulsarClient, transactionBufferSnapshotServiceOriginal,
- transactionBufferSnapshotService, originalTopic, field, producer);
+ transactionBufferSnapshotService, originalTopic, field);
doReturn(true).when(reader).hasMoreEvents();
// mock reader can't read snapshot fail throw PulsarClientException
[pulsar] 27/31: [improve][common] Use `Collection` to instead of `List` parameter type (#15329)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 08219aef8840cb00369ea8d7aada42754d510f4d
Author: Qiang Zhao <74...@users.noreply.github.com>
AuthorDate: Tue Apr 26 22:11:35 2022 +0800
[improve][common] Use `Collection` to instead of `List` parameter type (#15329)
### Motivation
We can use `Collection` instead of `List` parameter type in `FutureUtil` for better compatibility.
For example when we need to use the values of `Map`:
```java
FutureUtil.waitForAll(map.values());
```
### Modifications
- Use `Collection` instead of `List` parameter type.
(cherry picked from commit 0c694cfdc9b11f915f1da86260ad3655c2e99a35)
---
.../java/org/apache/pulsar/common/util/FutureUtil.java | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
index f51888a76df..e5c2caeb7d0 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
@@ -19,7 +19,7 @@
package org.apache.pulsar.common.util;
import java.time.Duration;
-import java.util.List;
+import java.util.Collection;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
@@ -38,22 +38,22 @@ import java.util.stream.Collectors;
public class FutureUtil {
/**
- * Return a future that represents the completion of the futures in the provided list.
+ * Return a future that represents the completion of the futures in the provided Collection.
*
* @param futures futures to wait for
* @return a new CompletableFuture that is completed when all of the given CompletableFutures complete
*/
- public static CompletableFuture<Void> waitForAll(List<? extends CompletableFuture<?>> futures) {
+ public static CompletableFuture<Void> waitForAll(Collection<? extends CompletableFuture<?>> futures) {
return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));
}
/**
- * Return a future that represents the completion of any future in the provided list.
+ * Return a future that represents the completion of any future in the provided Collection.
*
* @param futures futures to wait any
* @return a new CompletableFuture that is completed when any of the given CompletableFutures complete
*/
- public static CompletableFuture<Object> waitForAny(List<? extends CompletableFuture<?>> futures) {
+ public static CompletableFuture<Object> waitForAny(Collection<? extends CompletableFuture<?>> futures) {
return CompletableFuture.anyOf(futures.toArray(new CompletableFuture[0]));
}
@@ -101,14 +101,15 @@ public class FutureUtil {
/**
- * Return a future that represents the completion of the futures in the provided list.
+ * Return a future that represents the completion of the futures in the provided Collection.
* The future will support {@link CompletableFuture#cancel(boolean)}. It will cancel
* all unfinished futures when the future gets cancelled.
*
* @param futures futures to wait for
* @return a new CompletableFuture that is completed when all of the given CompletableFutures complete
*/
- public static CompletableFuture<Void> waitForAllAndSupportCancel(List<? extends CompletableFuture<?>> futures) {
+ public static CompletableFuture<Void> waitForAllAndSupportCancel(
+ Collection<? extends CompletableFuture<?>> futures) {
CompletableFuture[] futuresArray = futures.toArray(new CompletableFuture[0]);
CompletableFuture<Void> combinedFuture = CompletableFuture.allOf(futuresArray);
whenCancelledOrTimedOut(combinedFuture, () -> {
[pulsar] 14/31: [Improve][doc] Add config of IO and acceptor threads in proxy (#15340)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 32f123c4beba1906fdd398c149063c8ee1387e30
Author: Kai Wang <kw...@streamnative.io>
AuthorDate: Fri May 6 21:40:51 2022 +0800
[Improve][doc] Add config of IO and acceptor threads in proxy (#15340)
* Add config of IO and acceptor threads in proxy
* Update doc
* Update site2/docs/reference-configuration.md
Co-authored-by: Anonymitaet <50...@users.noreply.github.com>
* Update site2/docs/reference-configuration.md
Co-authored-by: Anonymitaet <50...@users.noreply.github.com>
(cherry picked from commit da3f017240662fe9498dcc3d0f8513c02a740bb8)
---
conf/proxy.conf | 6 ++++++
.../java/org/apache/pulsar/proxy/server/ProxyConfiguration.java | 4 ++--
site2/docs/reference-configuration.md | 2 ++
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/conf/proxy.conf b/conf/proxy.conf
index cdaf0ff732f..9f83278d7a0 100644
--- a/conf/proxy.conf
+++ b/conf/proxy.conf
@@ -71,6 +71,12 @@ webServicePort=8080
# Port to use to server HTTPS request
webServicePortTls=
+# Number of threads used for Netty IO. Default is set to `2 * Runtime.getRuntime().availableProcessors()`
+numIOThreads=
+
+# Number of threads used for Netty Acceptor. Default is set to `1`
+numAcceptorThreads=
+
### --- TLS config variables --- ###
## Note that some of the above TLS configs also apply to the KeyStore TLS configuration.
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
index 2c8204bc231..5a412dff65c 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java
@@ -615,14 +615,14 @@ public class ProxyConfiguration implements PulsarConfiguration {
@FieldContext(
category = CATEGORY_SERVER,
- doc = "Number of threads to use for Netty IO."
+ doc = "Number of threads used for Netty IO."
+ " Default is set to `2 * Runtime.getRuntime().availableProcessors()`"
)
private int numIOThreads = 2 * Runtime.getRuntime().availableProcessors();
@FieldContext(
category = CATEGORY_SERVER,
- doc = "Number of threads to use for Netty Acceptor."
+ doc = "Number of threads used for Netty Acceptor."
+ " Default is set to `1`"
)
private int numAcceptorThreads = 1;
diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md
index b1ac376fd8b..4ca94c5a65f 100644
--- a/site2/docs/reference-configuration.md
+++ b/site2/docs/reference-configuration.md
@@ -800,6 +800,8 @@ The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) can be config
|tokenAudienceClaim| The token audience "claim" name, e.g. "aud". It is used to get the audience from token. If it is not set, the audience is not verified. ||
| tokenAudience | The token audience stands for this broker. The field `tokenAudienceClaim` of a valid token need contains this parameter.| |
|haProxyProtocolEnabled | Enable or disable the [HAProxy](http://www.haproxy.org/) protocol. |false|
+| numIOThreads | Number of threads used for Netty IO. | 2 * Runtime.getRuntime().availableProcessors() |
+| numAcceptorThreads | Number of threads used for Netty Acceptor. | 1 |
#### Deprecated parameters of Pulsar proxy
The following parameters have been deprecated in the `conf/proxy.conf` file.
[pulsar] 02/31: Use tlsCertRefreshCheckDurationSec instead of 0 for refresh value (#15075)
Posted by pe...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 1952a9cd87696b3540404bd50f48b1ae2ed040e3
Author: Michael Marshall <mm...@apache.org>
AuthorDate: Fri Apr 8 00:44:15 2022 -0500
Use tlsCertRefreshCheckDurationSec instead of 0 for refresh value (#15075)
(cherry picked from commit e398d7e412c21e47c7e5d48225088f12874ebd29)
---
.../src/main/java/org/apache/pulsar/common/util/SecurityUtility.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
index 4560113ff67..e9266589ffc 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
@@ -548,7 +548,7 @@ public class SecurityUtility {
SslContextFactory sslCtxFactory = null;
if (autoRefresh) {
sslCtxFactory = new SslContextFactoryWithAutoRefresh(tlsAllowInsecureConnection, tlsTrustCertsFilePath,
- tlsCertificateFilePath, tlsKeyFilePath, tlsRequireTrustedClientCertOnConnect, 0);
+ tlsCertificateFilePath, tlsKeyFilePath, tlsRequireTrustedClientCertOnConnect, certRefreshInSec);
} else {
sslCtxFactory = new SslContextFactory();
SSLContext sslCtx = createSslContext(tlsAllowInsecureConnection, tlsTrustCertsFilePath,