You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/01/30 14:45:05 UTC
[GitHub] [hadoop-ozone] elek opened a new pull request #508: HDDS-2950.
Upgrade jetty to the latest 9.4 release
elek opened a new pull request #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508
## What changes were proposed in this pull request?
The jetty which is used by web interfaces of Ozone is from the September of 2018.
Since then many bug and security fixes added to the Jetty project. I would suggest to use the latest jetty (from January of 2020).
As `HttpServer2` (hadoop 3.2 class) has a strong dependency on the older version of Jetty (it uses SessionManager which is removed), it seems to be easier to clone HttpServer2 and move it to the Ozone project.
It provides us the flexibility:
* To upgrade jetty independent from the Hadoop releases
* To remove unused features and make our HTTP stack more reliable (eg. Yarn, WebHDFS part of the code)
* To customize our HTTP server usage (eg. SPNEGO should be ignore for S3 REST)
* To support ozone style configuration and maintain all our configuration est
* To add additional insights/metrics to our own HttpServer
* To simplify the current server initialization (current BaseHttpServer class of hdds/ozone is a wrapper around the Hadoop class, but we can combine the two functionalities)
## What is included in this patch
* `HttpServer2` and lightweight dependencies are cloned from Hadoop **3.2**
* HADOOP-16152 (Jetty upgrade since 3.2 release)
* Checkstyle / findbugs (to follow our coding standards)
* As we have >5 http related classes I moved them to a dedicated package in the *framework* project.
* **The main logic has not been modified by this patch ** This is mainly code organization and code import.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2950
## How was this patch tested?
There are related unit tests, but also checked with docker-compose based cluster + checking the ui pages from browser.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl edited a comment on issue #508: HDDS-2950.
Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
smengcl edited a comment on issue #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#issuecomment-582625425
> Thank you very much @smengcl
>
> As a background: as of now we use hadoop 3.2. The most safest import is to use exactly the same `HttpServer2.java` what we have in the latest hadoop 3.2 release.
>
> I tried to use trunk HttpServer2 from the Hadoop **trunk** but found multiple changes related the authorization broke S3 gateway.
>
> After that I decided to keep the same `HttpServer2` which is used now (with Jetty patch included).
>
> Therefore (as a rule of thumb) I prefer to import here just minimal changes and import newer patches in separated jira (where we can carefully test s3g).
>
> I checked the suggested patches (thanks for the suggestions):
>
> * HADOOP-16727 It's a small null check, can be good to add it as soon as possible. I added it to the patch.
> * HADOOP-16398 Prometheus support is backported from Ozone to Hadoop. We don't need the patch as we already have the original solution. (Later we can simplify our initialization to make it more similar to the other default servlets but it's a bigger refactor.)
> * HADOOP-16718 I tried to understand why this is required and didn't find any information. If you have something, let me know. (Might be required for webhdfs which shouldn't be supported in our side.) For me it seems to be optional, and s3g and other ozone components work well based on the tests. Unless we have a strong reason I would prefer to keep the default Jetty behavior (and use one less config).
Thanks for digging into this. I'm good with using same `HttpServer2.java` as Hadoop 3.2.
The two new commits lgtm.
As for HADOOP-16718, I believe the scope is larger than WebHDFS. Web UIs with HTTPS using jetty may all be impacted under certain cases. In short, this can cause connection failure in some circumstances:
> if the server's JKS file has a private/public key/cert pairing that is valid but it also has another trustedCertEntry certificate that has the hostname in subjectAltName extension, the trusted cert gets picked.
> This triggers an internal failure to determine a common cipher to use and the server will return the following exception to the client:
> fatal error: 40: no cipher suites in common
For details, please take a look at the link I sent over to you on Slack.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #508: HDDS-2950. Upgrade
jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
elek commented on issue #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#issuecomment-581323524
Thank you very much @smengcl
As a background: as of now we use hadoop 3.2. The most safest import is to use exactly the same `HttpServer2.java` what we have in the latest hadoop 3.2 release.
I tried to use trunk HttpServer2 from the Hadoop **trunk** but found multiple changes related the authorization broke S3 gateway.
After that I decided to keep the same `HttpServer2` which is used now (with Jetty patch included).
Therefore (as a rule of thumb) I prefer to import here just minimal changes and import newer patches in separated jira (where we can carefully test s3g).
I checked the suggested patches (thanks for the suggestions):
* HADOOP-16727 It's a small null check, can be good to add it as soon as possible. I added it to the patch.
* HADOOP-16398 Prometheus support is backported from Ozone to Hadoop. We don't need the patch as we already have the original solution. (Later we can simplify our initialization to make it more similar to the other default servlets but it's a bigger refactor.)
* HADOOP-16718 I tried to understand why this is required and didn't find any information. If you have something, let me know. (Might be required for webhdfs which shouldn't be supported in our side.) For me it seems to be optional, and s3g and other ozone components work well based on the tests. Unless we have a strong reason I would prefer to keep the default Jetty behavior (and use one less config).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on issue #508: HDDS-2950.
Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on issue #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#issuecomment-583618499
Thanks @elek for the update. The latest change LGTM, +1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #508: HDDS-2950.
Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
xiaoyuyao merged pull request #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #508:
HDDS-2950. Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#discussion_r373180559
##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java
##########
@@ -0,0 +1,1701 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.server.http;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.io.PrintStream;
+import java.net.BindException;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletRequestWrapper;
+import javax.servlet.http.HttpServletResponse;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.sun.jersey.spi.container.servlet.ServletContainer;
+import org.apache.hadoop.HadoopIllegalArgumentException;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.ConfServlet;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configuration.IntegerRanges;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.jmx.JMXJsonServlet;
+import org.apache.hadoop.log.LogLevel;
+import org.apache.hadoop.security.AuthenticationFilterInitializer;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.security.ssl.SSLFactory;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.util.StringUtils;
+import org.eclipse.jetty.http.HttpVersion;
+import org.eclipse.jetty.server.ConnectionFactory;
+import org.eclipse.jetty.server.Connector;
+import org.eclipse.jetty.server.Handler;
+import org.eclipse.jetty.server.HttpConfiguration;
+import org.eclipse.jetty.server.HttpConnectionFactory;
+import org.eclipse.jetty.server.RequestLog;
+import org.eclipse.jetty.server.SecureRequestCustomizer;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.server.SslConnectionFactory;
+import org.eclipse.jetty.server.handler.ContextHandlerCollection;
+import org.eclipse.jetty.server.handler.HandlerCollection;
+import org.eclipse.jetty.server.handler.RequestLogHandler;
+import org.eclipse.jetty.server.session.SessionHandler;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.FilterHolder;
+import org.eclipse.jetty.servlet.FilterMapping;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.servlet.ServletMapping;
+import org.eclipse.jetty.util.ArrayUtil;
+import org.eclipse.jetty.util.MultiException;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.eclipse.jetty.webapp.WebAppContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Create a Jetty embedded server to answer http requests. The primary goal is
+ * to serve up status information for the server. There are three contexts:
+ * "/logs/" -> points to the log directory "/static/" -> points to common static
+ * files (src/webapps/static) "/" -> the jsp server code from
+ * (src/webapps/<name>)
+ *
+ * This class is a fork of the old HttpServer. HttpServer exists for
+ * compatibility reasons. See HBASE-10336 for more details.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+public final class HttpServer2 implements FilterContainer {
+ public static final Logger LOG = LoggerFactory.getLogger(HttpServer2.class);
+
+ private static final String HTTP_SCHEME = "http";
+ public static final String HTTPS_SCHEME = "https";
+
+ public static final String HTTP_MAX_REQUEST_HEADER_SIZE_KEY =
+ "hadoop.http.max.request.header.size";
+ public static final int HTTP_MAX_REQUEST_HEADER_SIZE_DEFAULT = 65536;
+ public static final String HTTP_MAX_RESPONSE_HEADER_SIZE_KEY =
+ "hadoop.http.max.response.header.size";
+ public static final int HTTP_MAX_RESPONSE_HEADER_SIZE_DEFAULT = 65536;
+
+ public static final String HTTP_SOCKET_BACKLOG_SIZE_KEY =
+ "hadoop.http.socket.backlog.size";
+ public static final int HTTP_SOCKET_BACKLOG_SIZE_DEFAULT = 128;
+ public static final String HTTP_MAX_THREADS_KEY = "hadoop.http.max.threads";
+ public static final String HTTP_ACCEPTOR_COUNT_KEY =
+ "hadoop.http.acceptor.count";
+ // -1 to use default behavior of setting count based on CPU core count
+ public static final int HTTP_ACCEPTOR_COUNT_DEFAULT = -1;
+ public static final String HTTP_SELECTOR_COUNT_KEY =
+ "hadoop.http.selector.count";
+ // -1 to use default behavior of setting count based on CPU core count
+ public static final int HTTP_SELECTOR_COUNT_DEFAULT = -1;
+ // idle timeout in milliseconds
+ public static final String HTTP_IDLE_TIMEOUT_MS_KEY =
+ "hadoop.http.idle_timeout.ms";
+ public static final int HTTP_IDLE_TIMEOUT_MS_DEFAULT = 10000;
+ public static final String HTTP_TEMP_DIR_KEY = "hadoop.http.temp.dir";
+
+ public static final String FILTER_INITIALIZER_PROPERTY
+ = "ozone.http.filter.initializers";
+
+ // The ServletContext attribute where the daemon Configuration
+ // gets stored.
+ public static final String CONF_CONTEXT_ATTRIBUTE = "hadoop.conf";
+ public static final String ADMINS_ACL = "admins.acl";
+ public static final String SPNEGO_FILTER = "SpnegoFilter";
+ public static final String NO_CACHE_FILTER = "NoCacheFilter";
+
+ public static final String BIND_ADDRESS = "bind.address";
+
+ private final AccessControlList adminsAcl;
+
+ private final Server webServer;
+
+ private final HandlerCollection handlers;
+
+ private final List<ServerConnector> listeners = Lists.newArrayList();
+
+ private final WebAppContext webAppContext;
+ private final boolean findPort;
+ private final IntegerRanges portRanges;
+ private final Map<ServletContextHandler, Boolean> defaultContexts =
+ new HashMap<>();
+ private final List<String> filterNames = new ArrayList<>();
+ static final String STATE_DESCRIPTION_ALIVE = " - alive";
+ static final String STATE_DESCRIPTION_NOT_LIVE = " - not live";
+ private final SignerSecretProvider secretProvider;
+ private XFrameOption xFrameOption;
+ private boolean xFrameOptionIsEnabled;
+ public static final String HTTP_HEADER_PREFIX = "hadoop.http.header.";
+ private static final String HTTP_HEADER_REGEX =
+ "hadoop\\.http\\.header\\.([a-zA-Z\\-_]+)";
+ static final String X_XSS_PROTECTION =
+ "X-XSS-Protection:1; mode=block";
+ static final String X_CONTENT_TYPE_OPTIONS =
+ "X-Content-Type-Options:nosniff";
+ private static final String X_FRAME_OPTIONS = "X-FRAME-OPTIONS";
+ private static final Pattern PATTERN_HTTP_HEADER_REGEX =
+ Pattern.compile(HTTP_HEADER_REGEX);
+ /**
+ * Class to construct instances of HTTP server with specific options.
+ */
+ public static class Builder {
+ private ArrayList<URI> endpoints = Lists.newArrayList();
+ private String name;
+ private Configuration conf;
+ private Configuration sslConf;
+ private String[] pathSpecs;
+ private AccessControlList adminsAcl;
+ private boolean securityEnabled = false;
+ private String usernameConfKey;
+ private String keytabConfKey;
+ private boolean needsClientAuth;
+ private String trustStore;
+ private String trustStorePassword;
+ private String trustStoreType;
+
+ private String keyStore;
+ private String keyStorePassword;
+ private String keyStoreType;
+
+ // The -keypass option in keytool
+ private String keyPassword;
+
+ private boolean findPort;
+ private IntegerRanges portRanges = null;
+
+ private String hostName;
+ private boolean disallowFallbackToRandomSignerSecretProvider;
+ private String authFilterConfigurationPrefix =
+ "hadoop.http.authentication.";
+ private String excludeCiphers;
+
+ private boolean xFrameEnabled;
+ private XFrameOption xFrameOption = XFrameOption.SAMEORIGIN;
+
+ public Builder setName(String serverName) {
+ this.name = serverName;
+ return this;
+ }
+
+ /**
+ * Add an endpoint that the HTTP server should listen to.
+ *
+ * @param endpoint
+ * the endpoint of that the HTTP server should listen to. The
+ * scheme specifies the protocol (i.e. HTTP / HTTPS), the host
+ * specifies the binding address, and the port specifies the
+ * listening port. Unspecified or zero port means that the server
+ * can listen to any port.
+ */
+ public Builder addEndpoint(URI endpoint) {
+ endpoints.add(endpoint);
+ return this;
+ }
+
+ /**
+ * Set the hostname of the http server. The host name is used to resolve the
+ * _HOST field in Kerberos principals. The hostname of the first listener
+ * will be used if the name is unspecified.
+ */
+ public Builder hostName(String host) {
+ this.hostName = host;
+ return this;
+ }
+
+ public Builder trustStore(String location, String password, String type) {
+ this.trustStore = location;
+ this.trustStorePassword = password;
+ this.trustStoreType = type;
+ return this;
+ }
+
+ public Builder keyStore(String location, String password, String type) {
+ this.keyStore = location;
+ this.keyStorePassword = password;
+ this.keyStoreType = type;
+ return this;
+ }
+
+ public Builder keyPassword(String password) {
+ this.keyPassword = password;
+ return this;
+ }
+
+ /**
+ * Specify whether the server should authorize the client in SSL
+ * connections.
+ */
+ public Builder needsClientAuth(boolean value) {
+ this.needsClientAuth = value;
+ return this;
+ }
+
+ public Builder setFindPort(boolean portFind) {
+ this.findPort = portFind;
+ return this;
+ }
+
+ public Builder setPortRanges(IntegerRanges ranges) {
+ this.portRanges = ranges;
+ return this;
+ }
+
+ public Builder setConf(Configuration configuration) {
+ this.conf = configuration;
+ return this;
+ }
+
+ /**
+ * Specify the SSL configuration to load. This API provides an alternative
+ * to keyStore/keyPassword/trustStore.
+ */
+ public Builder setSSLConf(Configuration sslCnf) {
+ this.sslConf = sslCnf;
+ return this;
+ }
+
+ public Builder setPathSpec(String[] pathSpec) {
+ this.pathSpecs = pathSpec.clone();
Review comment:
I see `.clone()` is added here. For safe practice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on issue #508: HDDS-2950. Upgrade
jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
arp7 commented on issue #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#issuecomment-580315814
@smengcl can you take a look at this pull request? This is forking the `HttpWebServer` from Hadoop and applying your changes to port to 9.4.26.
I think long term it makes sense to fork the Hadoop HttpWebServer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on issue #508: HDDS-2950. Upgrade
jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
smengcl commented on issue #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#issuecomment-582625425
> Thank you very much @smengcl
>
> As a background: as of now we use hadoop 3.2. The most safest import is to use exactly the same `HttpServer2.java` what we have in the latest hadoop 3.2 release.
>
> I tried to use trunk HttpServer2 from the Hadoop **trunk** but found multiple changes related the authorization broke S3 gateway.
>
> After that I decided to keep the same `HttpServer2` which is used now (with Jetty patch included).
>
> Therefore (as a rule of thumb) I prefer to import here just minimal changes and import newer patches in separated jira (where we can carefully test s3g).
>
> I checked the suggested patches (thanks for the suggestions):
>
> * HADOOP-16727 It's a small null check, can be good to add it as soon as possible. I added it to the patch.
> * HADOOP-16398 Prometheus support is backported from Ozone to Hadoop. We don't need the patch as we already have the original solution. (Later we can simplify our initialization to make it more similar to the other default servlets but it's a bigger refactor.)
> * HADOOP-16718 I tried to understand why this is required and didn't find any information. If you have something, let me know. (Might be required for webhdfs which shouldn't be supported in our side.) For me it seems to be optional, and s3g and other ozone components work well based on the tests. Unless we have a strong reason I would prefer to keep the default Jetty behavior (and use one less config).
Thanks for digging into this. I'm good with using same `HttpServer2.java` as Hadoop 3.2.
The two new commits lgtm.
As for HADOOP-16718, I believe the scope is larger than WebHDFS. Web UIs with HTTPS using jetty may all be impacted under certain cases. In short, this can cause connection failure if:
> ... the server's JKS file has a private/public key/cert pairing that is valid but it also has another trustedCertEntry certificate that has the hostname in subjectAltName extension, the trusted cert gets picked.
> This triggers an internal failure to determine a common cipher to use and the server will return the following exception to the client:
> fatal error: 40: no cipher suites in common
For details, please take a look at the link I sent over to you on Slack.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #508:
HDDS-2950. Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#discussion_r373995760
##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java
##########
@@ -0,0 +1,1701 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.server.http;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.io.PrintStream;
+import java.net.BindException;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletRequestWrapper;
+import javax.servlet.http.HttpServletResponse;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.sun.jersey.spi.container.servlet.ServletContainer;
+import org.apache.hadoop.HadoopIllegalArgumentException;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.ConfServlet;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configuration.IntegerRanges;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.jmx.JMXJsonServlet;
+import org.apache.hadoop.log.LogLevel;
+import org.apache.hadoop.security.AuthenticationFilterInitializer;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.security.ssl.SSLFactory;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.util.StringUtils;
+import org.eclipse.jetty.http.HttpVersion;
+import org.eclipse.jetty.server.ConnectionFactory;
+import org.eclipse.jetty.server.Connector;
+import org.eclipse.jetty.server.Handler;
+import org.eclipse.jetty.server.HttpConfiguration;
+import org.eclipse.jetty.server.HttpConnectionFactory;
+import org.eclipse.jetty.server.RequestLog;
+import org.eclipse.jetty.server.SecureRequestCustomizer;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.server.SslConnectionFactory;
+import org.eclipse.jetty.server.handler.ContextHandlerCollection;
+import org.eclipse.jetty.server.handler.HandlerCollection;
+import org.eclipse.jetty.server.handler.RequestLogHandler;
+import org.eclipse.jetty.server.session.SessionHandler;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.FilterHolder;
+import org.eclipse.jetty.servlet.FilterMapping;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.servlet.ServletMapping;
+import org.eclipse.jetty.util.ArrayUtil;
+import org.eclipse.jetty.util.MultiException;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.eclipse.jetty.webapp.WebAppContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Create a Jetty embedded server to answer http requests. The primary goal is
+ * to serve up status information for the server. There are three contexts:
+ * "/logs/" -> points to the log directory "/static/" -> points to common static
+ * files (src/webapps/static) "/" -> the jsp server code from
+ * (src/webapps/<name>)
+ *
+ * This class is a fork of the old HttpServer. HttpServer exists for
+ * compatibility reasons. See HBASE-10336 for more details.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+public final class HttpServer2 implements FilterContainer {
+ public static final Logger LOG = LoggerFactory.getLogger(HttpServer2.class);
+
+ private static final String HTTP_SCHEME = "http";
+ public static final String HTTPS_SCHEME = "https";
Review comment:
sure, done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on issue #508: HDDS-2950. Upgrade
jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
smengcl commented on issue #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#issuecomment-580416261
> @smengcl can you take a look at this pull request? This is forking the `HttpWebServer` from Hadoop and applying your changes to port to 9.4.26.
>
> I think long term it makes sense to fork the Hadoop HttpWebServer.
Sure. Looking at this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #508:
HDDS-2950. Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#discussion_r373177942
##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java
##########
@@ -0,0 +1,1701 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.server.http;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.io.PrintStream;
+import java.net.BindException;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletRequestWrapper;
+import javax.servlet.http.HttpServletResponse;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.sun.jersey.spi.container.servlet.ServletContainer;
+import org.apache.hadoop.HadoopIllegalArgumentException;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.ConfServlet;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configuration.IntegerRanges;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.jmx.JMXJsonServlet;
+import org.apache.hadoop.log.LogLevel;
+import org.apache.hadoop.security.AuthenticationFilterInitializer;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.security.ssl.SSLFactory;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.util.StringUtils;
+import org.eclipse.jetty.http.HttpVersion;
+import org.eclipse.jetty.server.ConnectionFactory;
+import org.eclipse.jetty.server.Connector;
+import org.eclipse.jetty.server.Handler;
+import org.eclipse.jetty.server.HttpConfiguration;
+import org.eclipse.jetty.server.HttpConnectionFactory;
+import org.eclipse.jetty.server.RequestLog;
+import org.eclipse.jetty.server.SecureRequestCustomizer;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.server.SslConnectionFactory;
+import org.eclipse.jetty.server.handler.ContextHandlerCollection;
+import org.eclipse.jetty.server.handler.HandlerCollection;
+import org.eclipse.jetty.server.handler.RequestLogHandler;
+import org.eclipse.jetty.server.session.SessionHandler;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.FilterHolder;
+import org.eclipse.jetty.servlet.FilterMapping;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.servlet.ServletMapping;
+import org.eclipse.jetty.util.ArrayUtil;
+import org.eclipse.jetty.util.MultiException;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.eclipse.jetty.webapp.WebAppContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Create a Jetty embedded server to answer http requests. The primary goal is
+ * to serve up status information for the server. There are three contexts:
+ * "/logs/" -> points to the log directory "/static/" -> points to common static
+ * files (src/webapps/static) "/" -> the jsp server code from
+ * (src/webapps/<name>)
+ *
+ * This class is a fork of the old HttpServer. HttpServer exists for
+ * compatibility reasons. See HBASE-10336 for more details.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+public final class HttpServer2 implements FilterContainer {
+ public static final Logger LOG = LoggerFactory.getLogger(HttpServer2.class);
+
+ private static final String HTTP_SCHEME = "http";
+ public static final String HTTPS_SCHEME = "https";
Review comment:
This can be changed to `private` as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #508:
HDDS-2950. Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#discussion_r373180067
##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java
##########
@@ -0,0 +1,1701 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.server.http;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.io.PrintStream;
+import java.net.BindException;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletRequestWrapper;
+import javax.servlet.http.HttpServletResponse;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.sun.jersey.spi.container.servlet.ServletContainer;
+import org.apache.hadoop.HadoopIllegalArgumentException;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.ConfServlet;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configuration.IntegerRanges;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.jmx.JMXJsonServlet;
+import org.apache.hadoop.log.LogLevel;
+import org.apache.hadoop.security.AuthenticationFilterInitializer;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.security.ssl.SSLFactory;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.util.StringUtils;
+import org.eclipse.jetty.http.HttpVersion;
+import org.eclipse.jetty.server.ConnectionFactory;
+import org.eclipse.jetty.server.Connector;
+import org.eclipse.jetty.server.Handler;
+import org.eclipse.jetty.server.HttpConfiguration;
+import org.eclipse.jetty.server.HttpConnectionFactory;
+import org.eclipse.jetty.server.RequestLog;
+import org.eclipse.jetty.server.SecureRequestCustomizer;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.server.SslConnectionFactory;
+import org.eclipse.jetty.server.handler.ContextHandlerCollection;
+import org.eclipse.jetty.server.handler.HandlerCollection;
+import org.eclipse.jetty.server.handler.RequestLogHandler;
+import org.eclipse.jetty.server.session.SessionHandler;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.FilterHolder;
+import org.eclipse.jetty.servlet.FilterMapping;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.servlet.ServletMapping;
+import org.eclipse.jetty.util.ArrayUtil;
+import org.eclipse.jetty.util.MultiException;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.eclipse.jetty.webapp.WebAppContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Create a Jetty embedded server to answer http requests. The primary goal is
+ * to serve up status information for the server. There are three contexts:
+ * "/logs/" -> points to the log directory "/static/" -> points to common static
+ * files (src/webapps/static) "/" -> the jsp server code from
+ * (src/webapps/<name>)
+ *
+ * This class is a fork of the old HttpServer. HttpServer exists for
+ * compatibility reasons. See HBASE-10336 for more details.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+public final class HttpServer2 implements FilterContainer {
+ public static final Logger LOG = LoggerFactory.getLogger(HttpServer2.class);
+
+ private static final String HTTP_SCHEME = "http";
+ public static final String HTTPS_SCHEME = "https";
+
+ public static final String HTTP_MAX_REQUEST_HEADER_SIZE_KEY =
+ "hadoop.http.max.request.header.size";
+ public static final int HTTP_MAX_REQUEST_HEADER_SIZE_DEFAULT = 65536;
+ public static final String HTTP_MAX_RESPONSE_HEADER_SIZE_KEY =
+ "hadoop.http.max.response.header.size";
+ public static final int HTTP_MAX_RESPONSE_HEADER_SIZE_DEFAULT = 65536;
+
+ public static final String HTTP_SOCKET_BACKLOG_SIZE_KEY =
+ "hadoop.http.socket.backlog.size";
+ public static final int HTTP_SOCKET_BACKLOG_SIZE_DEFAULT = 128;
+ public static final String HTTP_MAX_THREADS_KEY = "hadoop.http.max.threads";
+ public static final String HTTP_ACCEPTOR_COUNT_KEY =
+ "hadoop.http.acceptor.count";
+ // -1 to use default behavior of setting count based on CPU core count
+ public static final int HTTP_ACCEPTOR_COUNT_DEFAULT = -1;
+ public static final String HTTP_SELECTOR_COUNT_KEY =
+ "hadoop.http.selector.count";
+ // -1 to use default behavior of setting count based on CPU core count
+ public static final int HTTP_SELECTOR_COUNT_DEFAULT = -1;
+ // idle timeout in milliseconds
+ public static final String HTTP_IDLE_TIMEOUT_MS_KEY =
+ "hadoop.http.idle_timeout.ms";
+ public static final int HTTP_IDLE_TIMEOUT_MS_DEFAULT = 10000;
+ public static final String HTTP_TEMP_DIR_KEY = "hadoop.http.temp.dir";
+
+ public static final String FILTER_INITIALIZER_PROPERTY
+ = "ozone.http.filter.initializers";
+
Review comment:
`hadoop.http.sni.host.check.enabled` is added in [HADOOP-16718](https://issues.apache.org/jira/browse/HADOOP-16718) in trunk. We might want it here as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #508:
HDDS-2950. Upgrade jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#discussion_r373996091
##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java
##########
@@ -0,0 +1,1701 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.server.http;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.io.PrintStream;
+import java.net.BindException;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletRequestWrapper;
+import javax.servlet.http.HttpServletResponse;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.sun.jersey.spi.container.servlet.ServletContainer;
+import org.apache.hadoop.HadoopIllegalArgumentException;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.ConfServlet;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configuration.IntegerRanges;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.jmx.JMXJsonServlet;
+import org.apache.hadoop.log.LogLevel;
+import org.apache.hadoop.security.AuthenticationFilterInitializer;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.security.ssl.SSLFactory;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.util.StringUtils;
+import org.eclipse.jetty.http.HttpVersion;
+import org.eclipse.jetty.server.ConnectionFactory;
+import org.eclipse.jetty.server.Connector;
+import org.eclipse.jetty.server.Handler;
+import org.eclipse.jetty.server.HttpConfiguration;
+import org.eclipse.jetty.server.HttpConnectionFactory;
+import org.eclipse.jetty.server.RequestLog;
+import org.eclipse.jetty.server.SecureRequestCustomizer;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.server.SslConnectionFactory;
+import org.eclipse.jetty.server.handler.ContextHandlerCollection;
+import org.eclipse.jetty.server.handler.HandlerCollection;
+import org.eclipse.jetty.server.handler.RequestLogHandler;
+import org.eclipse.jetty.server.session.SessionHandler;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.FilterHolder;
+import org.eclipse.jetty.servlet.FilterMapping;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.servlet.ServletMapping;
+import org.eclipse.jetty.util.ArrayUtil;
+import org.eclipse.jetty.util.MultiException;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.eclipse.jetty.webapp.WebAppContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Create a Jetty embedded server to answer http requests. The primary goal is
+ * to serve up status information for the server. There are three contexts:
+ * "/logs/" -> points to the log directory "/static/" -> points to common static
+ * files (src/webapps/static) "/" -> the jsp server code from
+ * (src/webapps/<name>)
+ *
+ * This class is a fork of the old HttpServer. HttpServer exists for
+ * compatibility reasons. See HBASE-10336 for more details.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+public final class HttpServer2 implements FilterContainer {
+ public static final Logger LOG = LoggerFactory.getLogger(HttpServer2.class);
+
+ private static final String HTTP_SCHEME = "http";
+ public static final String HTTPS_SCHEME = "https";
+
+ public static final String HTTP_MAX_REQUEST_HEADER_SIZE_KEY =
+ "hadoop.http.max.request.header.size";
+ public static final int HTTP_MAX_REQUEST_HEADER_SIZE_DEFAULT = 65536;
+ public static final String HTTP_MAX_RESPONSE_HEADER_SIZE_KEY =
+ "hadoop.http.max.response.header.size";
+ public static final int HTTP_MAX_RESPONSE_HEADER_SIZE_DEFAULT = 65536;
+
+ public static final String HTTP_SOCKET_BACKLOG_SIZE_KEY =
+ "hadoop.http.socket.backlog.size";
+ public static final int HTTP_SOCKET_BACKLOG_SIZE_DEFAULT = 128;
+ public static final String HTTP_MAX_THREADS_KEY = "hadoop.http.max.threads";
+ public static final String HTTP_ACCEPTOR_COUNT_KEY =
+ "hadoop.http.acceptor.count";
+ // -1 to use default behavior of setting count based on CPU core count
+ public static final int HTTP_ACCEPTOR_COUNT_DEFAULT = -1;
+ public static final String HTTP_SELECTOR_COUNT_KEY =
+ "hadoop.http.selector.count";
+ // -1 to use default behavior of setting count based on CPU core count
+ public static final int HTTP_SELECTOR_COUNT_DEFAULT = -1;
+ // idle timeout in milliseconds
+ public static final String HTTP_IDLE_TIMEOUT_MS_KEY =
+ "hadoop.http.idle_timeout.ms";
+ public static final int HTTP_IDLE_TIMEOUT_MS_DEFAULT = 10000;
+ public static final String HTTP_TEMP_DIR_KEY = "hadoop.http.temp.dir";
+
+ public static final String FILTER_INITIALIZER_PROPERTY
+ = "ozone.http.filter.initializers";
+
+ // The ServletContext attribute where the daemon Configuration
+ // gets stored.
+ public static final String CONF_CONTEXT_ATTRIBUTE = "hadoop.conf";
+ public static final String ADMINS_ACL = "admins.acl";
+ public static final String SPNEGO_FILTER = "SpnegoFilter";
+ public static final String NO_CACHE_FILTER = "NoCacheFilter";
+
+ public static final String BIND_ADDRESS = "bind.address";
+
+ private final AccessControlList adminsAcl;
+
+ private final Server webServer;
+
+ private final HandlerCollection handlers;
+
+ private final List<ServerConnector> listeners = Lists.newArrayList();
+
+ private final WebAppContext webAppContext;
+ private final boolean findPort;
+ private final IntegerRanges portRanges;
+ private final Map<ServletContextHandler, Boolean> defaultContexts =
+ new HashMap<>();
+ private final List<String> filterNames = new ArrayList<>();
+ static final String STATE_DESCRIPTION_ALIVE = " - alive";
+ static final String STATE_DESCRIPTION_NOT_LIVE = " - not live";
+ private final SignerSecretProvider secretProvider;
+ private XFrameOption xFrameOption;
+ private boolean xFrameOptionIsEnabled;
+ public static final String HTTP_HEADER_PREFIX = "hadoop.http.header.";
+ private static final String HTTP_HEADER_REGEX =
+ "hadoop\\.http\\.header\\.([a-zA-Z\\-_]+)";
+ static final String X_XSS_PROTECTION =
+ "X-XSS-Protection:1; mode=block";
+ static final String X_CONTENT_TYPE_OPTIONS =
+ "X-Content-Type-Options:nosniff";
+ private static final String X_FRAME_OPTIONS = "X-FRAME-OPTIONS";
+ private static final Pattern PATTERN_HTTP_HEADER_REGEX =
+ Pattern.compile(HTTP_HEADER_REGEX);
+ /**
+ * Class to construct instances of HTTP server with specific options.
+ */
+ public static class Builder {
+ private ArrayList<URI> endpoints = Lists.newArrayList();
+ private String name;
+ private Configuration conf;
+ private Configuration sslConf;
+ private String[] pathSpecs;
+ private AccessControlList adminsAcl;
+ private boolean securityEnabled = false;
+ private String usernameConfKey;
+ private String keytabConfKey;
+ private boolean needsClientAuth;
+ private String trustStore;
+ private String trustStorePassword;
+ private String trustStoreType;
+
+ private String keyStore;
+ private String keyStorePassword;
+ private String keyStoreType;
+
+ // The -keypass option in keytool
+ private String keyPassword;
+
+ private boolean findPort;
+ private IntegerRanges portRanges = null;
+
+ private String hostName;
+ private boolean disallowFallbackToRandomSignerSecretProvider;
+ private String authFilterConfigurationPrefix =
+ "hadoop.http.authentication.";
+ private String excludeCiphers;
+
+ private boolean xFrameEnabled;
+ private XFrameOption xFrameOption = XFrameOption.SAMEORIGIN;
+
+ public Builder setName(String serverName) {
+ this.name = serverName;
+ return this;
+ }
+
+ /**
+ * Add an endpoint that the HTTP server should listen to.
+ *
+ * @param endpoint
+ * the endpoint of that the HTTP server should listen to. The
+ * scheme specifies the protocol (i.e. HTTP / HTTPS), the host
+ * specifies the binding address, and the port specifies the
+ * listening port. Unspecified or zero port means that the server
+ * can listen to any port.
+ */
+ public Builder addEndpoint(URI endpoint) {
+ endpoints.add(endpoint);
+ return this;
+ }
+
+ /**
+ * Set the hostname of the http server. The host name is used to resolve the
+ * _HOST field in Kerberos principals. The hostname of the first listener
+ * will be used if the name is unspecified.
+ */
+ public Builder hostName(String host) {
+ this.hostName = host;
+ return this;
+ }
+
+ public Builder trustStore(String location, String password, String type) {
+ this.trustStore = location;
+ this.trustStorePassword = password;
+ this.trustStoreType = type;
+ return this;
+ }
+
+ public Builder keyStore(String location, String password, String type) {
+ this.keyStore = location;
+ this.keyStorePassword = password;
+ this.keyStoreType = type;
+ return this;
+ }
+
+ public Builder keyPassword(String password) {
+ this.keyPassword = password;
+ return this;
+ }
+
+ /**
+ * Specify whether the server should authorize the client in SSL
+ * connections.
+ */
+ public Builder needsClientAuth(boolean value) {
+ this.needsClientAuth = value;
+ return this;
+ }
+
+ public Builder setFindPort(boolean portFind) {
+ this.findPort = portFind;
+ return this;
+ }
+
+ public Builder setPortRanges(IntegerRanges ranges) {
+ this.portRanges = ranges;
+ return this;
+ }
+
+ public Builder setConf(Configuration configuration) {
+ this.conf = configuration;
+ return this;
+ }
+
+ /**
+ * Specify the SSL configuration to load. This API provides an alternative
+ * to keyStore/keyPassword/trustStore.
+ */
+ public Builder setSSLConf(Configuration sslCnf) {
+ this.sslConf = sslCnf;
+ return this;
+ }
+
+ public Builder setPathSpec(String[] pathSpec) {
+ this.pathSpecs = pathSpec.clone();
Review comment:
Yes, it was a findbugs violation (internal representation is leaked...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on issue #508: HDDS-2950. Upgrade
jetty to the latest 9.4 release
Posted by GitBox <gi...@apache.org>.
smengcl commented on issue #508: HDDS-2950. Upgrade jetty to the latest 9.4 release
URL: https://github.com/apache/hadoop-ozone/pull/508#issuecomment-583169434
I am good with backporting HADOOP-16718 in another jira. So far lgtm+1
Thanks @elek @arp7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org