You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Dávid Paksy (Jira)" <ji...@apache.org> on 2026/05/13 07:35:00 UTC
[jira] [Updated] (ZOOKEEPER-5047) Make PrometheusMetricsProvider KeyStore type detection consistent
[ https://issues.apache.org/jira/browse/ZOOKEEPER-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dávid Paksy updated ZOOKEEPER-5047:
-----------------------------------
Description:
When we have a BCFKS KeyStore and we do NOT explicitly set KeyStore type, PrometheusMetricsProvider fails to start:
{noformat}
2026-05-11 05:20:49,512 ERROR org.apache.zookeeper.server.ZooKeeperServerMain: Unexpected exception, exiting abnormally
java.io.IOException: Cannot boot MetricsProvider org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:131)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: org.apache.zookeeper.metrics.MetricsProviderLifeCycleException: Failed to start Prometheus Jetty server
at org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:246)
at org.apache.zookeeper.metrics.impl.MetricsProviderBootstrap.startMetricsProvider(MetricsProviderBootstrap.java:45)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:127)
... 4 more
Caused by: java.io.IOException: DerValue.getBigIntegerInternal, not expected 48
at java.base/sun.security.util.DerValue.getBigIntegerInternal(DerValue.java:633)
at java.base/sun.security.util.DerValue.getIntegerInternal(DerValue.java:594)
at java.base/sun.security.util.DerValue.getInteger(DerValue.java:590)
at java.base/sun.security.util.DerInputStream.getInteger(DerInputStream.java:126)
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2014)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.apache.zookeeper.server.admin.UnifiedConnectionFactory.doStart(UnifiedConnectionFactory.java:60)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:237)
... 6 more
{noformat}
PrometheusMetricsProvider can’t load the configured KeyStore because it tries to load it as PKCS12 while the keystore is BCFKS - *NOK*.
Explicitly setting KeyStore type to BCFKS resolves the issue:
{noformat}
metricsProvider.ssl.keyStore.type=BCFKS
metricsProvider.ssl.trustStore.type=BCFKS
{noformat}
However we don't need to set KeyStore type explicitly for the other keystores in ZooKeeper, e.g.:
- ssl.keyStore.type
- ssl.quorum.keyStore.type
JettyAdminServer also seem to detect BCFKS correctly because it uses X509Util.loadKeyStore() and X509Util.loadTrustStore().
The reason is that PrometheusMetricsProvider has hard coded type for KeyStore, TrustStore as PKCS12 and no other detection:
https://github.com/apache/zookeeper/blob/master/zookeeper-metrics-providers/zookeeper-prometheus-metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java#L147
Propose that PrometheusMetricsProvider should be consistent with the rest of the codebase with keytore type detection.
was:
When we have a BCFKS KeyStore and we do NOT explicitly set KeyStore type, PrometheusMetricsProvider fails to start:
{noformat}
2026-05-11 05:20:49,512 ERROR org.apache.zookeeper.server.ZooKeeperServerMain: Unexpected exception, exiting abnormally
java.io.IOException: Cannot boot MetricsProvider org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:131)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: org.apache.zookeeper.metrics.MetricsProviderLifeCycleException: Failed to start Prometheus Jetty server
at org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:246)
at org.apache.zookeeper.metrics.impl.MetricsProviderBootstrap.startMetricsProvider(MetricsProviderBootstrap.java:45)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:127)
... 4 more
Caused by: java.io.IOException: DerValue.getBigIntegerInternal, not expected 48
at java.base/sun.security.util.DerValue.getBigIntegerInternal(DerValue.java:633)
at java.base/sun.security.util.DerValue.getIntegerInternal(DerValue.java:594)
at java.base/sun.security.util.DerValue.getInteger(DerValue.java:590)
at java.base/sun.security.util.DerInputStream.getInteger(DerInputStream.java:126)
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2014)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.apache.zookeeper.server.admin.UnifiedConnectionFactory.doStart(UnifiedConnectionFactory.java:60)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:237)
... 6 more
{noformat}
PrometheusMetricsProvider can’t load the configured KeyStore because it tries to load it as PKCS12 while the keystore is BCFKS - *NOK*.
Explicitly setting KeyStore type to BCFKS resolves the issue:
{noformat}
metricsProvider.ssl.keyStore.type=BCFKS
metricsProvider.ssl.trustStore.type=BCFKS
{noformat}
However we don't need to set KeyStore type explicitly for the other keystores in ZooKeeper, e.g.:
- ssl.keyStore.type
- ssl.quorum.keyStore.type
JettyAdminServer also seem to detect BCFKS correctly because it uses X509Util.loadKeyStore() and X509Util.loadTrustStore().
The reason is that PrometheusMetricsProvider has hard coded type for KeyStore, TrustStore as PKCS12 and no other detecion:
https://github.com/apache/zookeeper/blob/master/zookeeper-metrics-providers/zookeeper-prometheus-metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java#L147
> Make PrometheusMetricsProvider KeyStore type detection consistent
> -----------------------------------------------------------------
>
> Key: ZOOKEEPER-5047
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5047
> Project: ZooKeeper
> Issue Type: Task
> Components: metric system
> Reporter: Dávid Paksy
> Priority: Major
>
> When we have a BCFKS KeyStore and we do NOT explicitly set KeyStore type, PrometheusMetricsProvider fails to start:
> {noformat}
> 2026-05-11 05:20:49,512 ERROR org.apache.zookeeper.server.ZooKeeperServerMain: Unexpected exception, exiting abnormally
> java.io.IOException: Cannot boot MetricsProvider org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
> at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:131)
> at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113)
> at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
> Caused by: org.apache.zookeeper.metrics.MetricsProviderLifeCycleException: Failed to start Prometheus Jetty server
> at org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:246)
> at org.apache.zookeeper.metrics.impl.MetricsProviderBootstrap.startMetricsProvider(MetricsProviderBootstrap.java:45)
> at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:127)
> ... 4 more
> Caused by: java.io.IOException: DerValue.getBigIntegerInternal, not expected 48
> at java.base/sun.security.util.DerValue.getBigIntegerInternal(DerValue.java:633)
> at java.base/sun.security.util.DerValue.getIntegerInternal(DerValue.java:594)
> at java.base/sun.security.util.DerValue.getInteger(DerValue.java:590)
> at java.base/sun.security.util.DerInputStream.getInteger(DerInputStream.java:126)
> at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2014)
> at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
> at java.base/java.security.KeyStore.load(KeyStore.java:1473)
> at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
> at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
> at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
> at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.apache.zookeeper.server.admin.UnifiedConnectionFactory.doStart(UnifiedConnectionFactory.java:60)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
> at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
> at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.eclipse.jetty.server.Server.doStart(Server.java:401)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:237)
> ... 6 more
> {noformat}
> PrometheusMetricsProvider can’t load the configured KeyStore because it tries to load it as PKCS12 while the keystore is BCFKS - *NOK*.
> Explicitly setting KeyStore type to BCFKS resolves the issue:
> {noformat}
> metricsProvider.ssl.keyStore.type=BCFKS
> metricsProvider.ssl.trustStore.type=BCFKS
> {noformat}
> However we don't need to set KeyStore type explicitly for the other keystores in ZooKeeper, e.g.:
> - ssl.keyStore.type
> - ssl.quorum.keyStore.type
> JettyAdminServer also seem to detect BCFKS correctly because it uses X509Util.loadKeyStore() and X509Util.loadTrustStore().
> The reason is that PrometheusMetricsProvider has hard coded type for KeyStore, TrustStore as PKCS12 and no other detection:
> https://github.com/apache/zookeeper/blob/master/zookeeper-metrics-providers/zookeeper-prometheus-metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java#L147
> Propose that PrometheusMetricsProvider should be consistent with the rest of the codebase with keytore type detection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)