You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/01 23:54:24 UTC
[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
avijayanhwx opened a new pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751
…enabled on it.
## What changes were proposed in this pull request?
Prometheus does not support targets which have Kerberos SPNEGO based authentication. Hence, on a secure cluster, if we have prometheus endpoint enabled, it makes sense to skip the authentication filter for it.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3321
## How was this patch tested?
Manually tested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#discussion_r402548932
##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/BaseHttpServer.java
##########
@@ -116,7 +120,15 @@ public BaseHttpServer(Configuration conf, String name) throws IOException {
prometheusMetricsSink = new PrometheusMetricsSink();
httpServer.getWebAppContext().getServletContext()
.setAttribute(PROMETHEUS_SINK, prometheusMetricsSink);
- httpServer.addInternalServlet("prometheus", "/prom", PrometheusServlet.class);
+ HddsPrometheusConfig prometheusConfig =
+ ((OzoneConfiguration)conf).getObject(HddsPrometheusConfig.class);
+ if (!prometheusConfig.isSpnegoAuthEnabled()) {
Review comment:
```suggestion
if (prometheusConfig.isSpnegoAuthEnabled()) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#discussion_r402549306
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/HddsPrometheusConfig.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.conf;
+
+/**
+ * The configuration class for the Prometheus endpoint.
+ */
+@ConfigGroup(prefix = "hdds.prometheus.")
+public class HddsPrometheusConfig {
+ @Config(key = "endpoint.spnego.auth.enabled",
+ type = ConfigType.BOOLEAN,
+ defaultValue = "false",
+ tags = { ConfigTag.SECURITY, ConfigTag.MANAGEMENT },
+ description = "This Kerberos principal is used by the Recon service."
Review comment:
Wrong description?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#discussion_r402550880
##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/BaseHttpServer.java
##########
@@ -116,7 +120,15 @@ public BaseHttpServer(Configuration conf, String name) throws IOException {
prometheusMetricsSink = new PrometheusMetricsSink();
httpServer.getWebAppContext().getServletContext()
.setAttribute(PROMETHEUS_SINK, prometheusMetricsSink);
- httpServer.addInternalServlet("prometheus", "/prom", PrometheusServlet.class);
+ HddsPrometheusConfig prometheusConfig =
+ ((OzoneConfiguration)conf).getObject(HddsPrometheusConfig.class);
+ if (!prometheusConfig.isSpnegoAuthEnabled()) {
Review comment:
Thanks! Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-610549379
It seems that the recommendation from Prometheus is to implement basic authentication via reverse proxy. But I'm open to explore other mechanism like simple and token in addition to SPNEGO. This way, we could remove a lot of hadoop-common dependencies.
https://prometheus.io/docs/guides/basic-auth/
"If you'd like to enforce basic auth for those connections, we recommend using Prometheus in conjunction with a reverse proxy and applying authentication at the proxy layer. "
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-607806945
One alternative approach is using token based or basic auth based authentication instead of SPNEGO. They seems to be supported by prometheus.
Not sure about the risk of this patch, as (in case of separated storage and execution) the webport also can be guarded with ip based firewall rules (they are not required for the users).
But would be great to hear the opinion of a security expert...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-608094749
Working on fixing CI failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-608390621
> Given the fact that prometheus does not support SPNEGO yet, there are three options:
I think there are more options. Like securing the Prometheus endpoint but using different authentication scheme not SPNEGO. For example basic auth or token based authentication. Both are supported by Prometheus.
It should be easy (especially with token based authentication): Just use a configuration value and if set, check the content of the Bearer token authorization header in `PrometheusServer`.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-607805685
I am fine with this change but interested about how is it used in HDFS....
To publish prometheus endpoint without authentication may expose some information which might not be available for the users... Is the `/jmx` endpoint of HDFS is exposed for everybody?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek edited a comment on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
elek edited a comment on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-610920020
> Prometheus does not directly support basic authentication (aka "basic auth") for connections to the Prometheus expression browser and HTTP API
It seems to be a different question: about how to authenticate when access the Prometheus server itself (API and UI). It can be an interesting question when Recon will read data from Prometheus via Prometheus API, but that's a different story, IMHO.
As far as I understood the current PR is about the security of the ozone `/prom` endpoints (as we assume that Prometheus is provided by the environment and is secured).
Based on the Promethes documentation it seems to be possible with a few lines of code as Promethes supports Bearer tokens:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
```
# Sets the `Authorization` header on every scrape request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <secret> ]
```
It seems to be enough to modify the `PrometheusServlet`:
```java
public static final String SECURITY_TOKEN = "PROMETHEUS_SECURITY_TOKEN";
public static final String BEARER = "Bearer";
public PrometheusMetricsSink getPrometheusSink() {
return
(PrometheusMetricsSink) getServletContext().getAttribute(
BaseHttpServer.PROMETHEUS_SINK);
}
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String securityToken =
(String) getServletContext().getAttribute(SECURITY_TOKEN);
if (securityToken != null) {
String authorizationHeader = req.getHeader("Authorization");
if (authorizationHeader == null
|| !authorizationHeader.startsWith(BEARER)
|| !securityToken.equals(authorizationHeader.substring(BEARER.length() + 1))) {
resp.setStatus(HttpServletResponse.SC_FORBIDDEN);
return;
}
}
```
And we should set the configuration in the `BaseHttpServer.java`:
```java
if (prometheusSupport) {
prometheusMetricsSink = new PrometheusMetricsSink();
httpServer.getWebAppContext().getServletContext().setAttribute(PROMETHEUS_SINK, prometheusMetricsSink);
httpServer.getWebAppContext().getServletContext().setAttribute(PrometheusServlet.SECURITY_TOKEN,
conf.get("hdds.prometheus.endpoint.token"));
httpServer.addServlet("prometheus", "/prom", PrometheusServlet.class);
}
```
And it works well with the following configuration:
```
scrape_configs:
- job_name: ozone
bearer_token: <putyourtokenhere>
metrics_path: /prom
static_configs:
- targets:
- "127.0.0.1:9876"
```
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-607980977
@elek historically, hadoop web endpoint apply authentication filter only on the public ones and not the internal ones. Later on, we harden the endpoint protection by enforcing authentication filter on an All OR NONE basis.
Given the fact that prometheus does not support SPNEGO yet, there are three options:
1. disable /prom endpoint completely when http authentication is configured.
2. leave /prom endpoint as is and wait for prometheus to support SPNEGO.
3. skip authentication filter for /prom endpoint which is the approach taken in this PR so that prometheus server can access the endpoint without SPNEGO.
Both 1,2 will make /prom endpoint useless in production (secured) environment. With security concern on 3, we could have a configuration switch to disable this by default and revert it when SPNEGO is supported by prometheus.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao edited a comment on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
xiaoyuyao edited a comment on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-612984665
Thanks @elek for the pointer. Adding token/password support for prometheus endpoint sounds good to me. With token support, I think it makes more sense to skip SPNEGO for the prometheus endpoint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek edited a comment on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
elek edited a comment on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-608390621
> Given the fact that prometheus does not support SPNEGO yet, there are three options:
I think there are more options. Like securing the Prometheus endpoint but using different authentication scheme not SPNEGO. For example basic auth or token based authentication. Both are supported by Prometheus.
It should be easy (especially with token based authentication): Just use a configuration value and if set, check the content of the Bearer token authorization header in `PrometheusServlet`.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-612984665
Thanks @elek for the pointer. Adding token/password support for prometheus endpoint sounds good to me. With token support, I think it makes more sense to skip SPNEGO for the prometheus endpoing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-607969254
In HDFS, there seem to be a few examples of using the "internal servlet"
` httpServer.addInternalServlet("fsck", "/fsck", FsckServlet.class,
true);
httpServer.addInternalServlet("imagetransfer", ImageServlet.PATH_SPEC,`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
Posted by GitBox <gi...@apache.org>.
elek commented on issue #751: HDDS-3321. Prometheus endpoint should not have Authentication filter …
URL: https://github.com/apache/hadoop-ozone/pull/751#issuecomment-610920020
> Prometheus does not directly support basic authentication (aka "basic auth") for connections to the Prometheus expression browser and HTTP API
It seems to be a different question: about how to authenticate the Prometheus server itself (API and UI). It can be an interesting question when Recon will read data from Prometheus via Prometheus API, but that's a different story, IMHO.
As far as I understood the current PR is about the security of the ozone `/prom` endpoints (as we assume that Prometheus is provided by the environment and is secured).
Based on the Promethes documentation it seems to be possible with a few lines of code as Promethes supports Bearer tokens:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
```
# Sets the `Authorization` header on every scrape request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <secret> ]
```
It seems to be enough to modify the `PrometheusServlet`:
```java
public static final String SECURITY_TOKEN = "PROMETHEUS_SECURITY_TOKEN";
public static final String BEARER = "Bearer";
public PrometheusMetricsSink getPrometheusSink() {
return
(PrometheusMetricsSink) getServletContext().getAttribute(
BaseHttpServer.PROMETHEUS_SINK);
}
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String securityToken =
(String) getServletContext().getAttribute(SECURITY_TOKEN);
if (securityToken != null) {
String authorizationHeader = req.getHeader("Authorization");
if (authorizationHeader == null
|| !authorizationHeader.startsWith(BEARER)
|| !securityToken.equals(authorizationHeader.substring(BEARER.length() + 1))) {
resp.setStatus(HttpServletResponse.SC_FORBIDDEN);
return;
}
}
```
And we should set the configuration in the `BaseHttpServer.java`:
```java
if (prometheusSupport) {
prometheusMetricsSink = new PrometheusMetricsSink();
httpServer.getWebAppContext().getServletContext().setAttribute(PROMETHEUS_SINK, prometheusMetricsSink);
httpServer.getWebAppContext().getServletContext().setAttribute(PrometheusServlet.SECURITY_TOKEN,
conf.get("hdds.prometheus.endpoint.token"));
httpServer.addServlet("prometheus", "/prom", PrometheusServlet.class);
}
```
And it works well with the following configuration:
```
scrape_configs:
- job_name: ozone
bearer_token: <putyourtokenhere>
metrics_path: /prom
static_configs:
- targets:
- "127.0.0.1:9876"
```
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org