You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by GitBox <gi...@apache.org> on 2022/03/23 07:18:56 UTC
[GitHub] [james-project] glennosss opened a new pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
glennosss opened a new pull request #929:
URL: https://github.com/apache/james-project/pull/929
Several fixes and improvements to make starting and managing netty servers easier - specifically:
1. Created 'EventLoopGroupManager' interface - see interface javadoc for more information - allows much better control over starting/sharing EventLoopGroups - https://github.com/glennosss/james-project/blob/netty4-combo/protocols/netty/src/main/java/org/apache/james/protocols/netty/EventLoopGroupManager.java
2. Updated 'Encryption' to support old static methods, but added additional static methods which add Netty SslContext support - see 'Encryption' javadoc for more information - https://github.com/glennosss/james-project/blob/netty4-combo/protocols/api/src/main/java/org/apache/james/protocols/api/Encryption.java
3. See 'ProtocolServer' javadoc - now supports async bind/unbind, control over how long unbind takes, and added unbindNow() to help with instantly stopping server for unit tests - https://github.com/glennosss/james-project/blob/netty4-combo/protocols/api/src/main/java/org/apache/james/protocols/api/ProtocolServer.java
4. Updated 'NettyServer', 'AbstractAsyncServer' to support proper construction using Factory/Builder approach - making it much easier to method chain and build a server.
5. Updated 'AbstractConfigurableAsyncServer' to support Factory/Builder approach - also 'bossThreads' and 'keepalive' parameters are now configurable.
FYI Also includes a fix for TCP_NODELAY still wasn't being set properly (`configureBootstrap()` was being run after socket was bound).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss commented on a change in pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss commented on a change in pull request #929:
URL: https://github.com/apache/james-project/pull/929#discussion_r832970016
##########
File path: protocols/netty/src/main/java/org/apache/james/protocols/netty/AbstractAsyncServer.java
##########
@@ -93,7 +93,7 @@ public synchronized void bind() throws Exception {
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.channel(NioServerSocketChannel.class);
- bossGroup = new NioEventLoopGroup(NamedThreadFactory.withName(jmxName + "-boss"));
+ bossGroup = new NioEventLoopGroup(2, NamedThreadFactory.withName(jmxName + "-boss"));
Review comment:
FYI These types of hard coded thread pool sizes are a big part of what this pull request is trying to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss commented on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss commented on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076007192
This pull request replaces https://github.com/chibenwa/james-project/pull/31
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076148022
Here is what I came up with regarding encryption: https://github.com/apache/james-project/pull/930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss commented on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss commented on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076971694
I'm having to move onto other projects. So good luck with the project guys. I'll close all my pull requests. But feel free to take whatever parts you did like if you want and do whatever you want with them. Later all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss edited a comment on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss edited a comment on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076069682
A shared 'EventLoopGroupManager' is also an important part of one doing being able to support a NIO SMTP client: https://issues.apache.org/jira/browse/JAMES-964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss commented on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss commented on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076066557
@chibenwa What's involved in getting this pull request approved and merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss closed pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss closed pull request #929:
URL: https://github.com/apache/james-project/pull/929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss commented on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss commented on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076069682
A shared 'EventLoopGroupManager' is also an important part of one doing being able to support a NIO client: https://issues.apache.org/jira/browse/JAMES-964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076095044
This work is not up to date with master. Could you rebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a change in pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
chibenwa commented on a change in pull request #929:
URL: https://github.com/apache/james-project/pull/929#discussion_r833003520
##########
File path: protocols/netty/src/main/java/org/apache/james/protocols/netty/LineDelimiterBasedChannelHandlerFactory.java
##########
@@ -32,7 +32,7 @@ public LineDelimiterBasedChannelHandlerFactory(int maxLineLength) {
@Override
public ChannelHandler create(ChannelPipeline pipeline) {
- return new LineBasedFrameDecoder(maxLineLength, false, !FAIL_FAST);
+ return new LineBasedFrameDecoder(maxLineLength, false, FAIL_FAST);
Review comment:
Unrelated change, and the constant name is no longer appropriate.
##########
File path: protocols/netty/src/main/java/org/apache/james/protocols/netty/EventLoopGroupManagerDefault.java
##########
@@ -0,0 +1,128 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+package org.apache.james.protocols.netty;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.james.util.concurrent.NamedThreadFactory;
+
+import io.netty.bootstrap.ServerBootstrap;
+import io.netty.channel.DefaultEventLoopGroup;
+import io.netty.channel.EventLoopGroup;
+import io.netty.channel.nio.NioEventLoopGroup;
+import io.netty.util.concurrent.EventExecutorGroup;
+import io.netty.util.concurrent.PromiseCombiner;
+
+public final class EventLoopGroupManagerDefault implements EventLoopGroupManager {
+
+ private static final int PROCESSORS = Runtime.getRuntime().availableProcessors();
+
+ public static final int DEFAULT_BOSS_THREADS = Math.min(2, PROCESSORS);
+ public static final int DEFAULT_WORKER_THREADS = PROCESSORS * 2;
+ public static final int DEFAULT_EXECUTOR_GROUP_THREADS = PROCESSORS * 2; // TODO potentially recommend setting this to 0 so reuses worker thread pool.
Review comment:
We can't as we have blocking logic that should not be executed on the IO eventloop...
##########
File path: protocols/api/pom.xml
##########
@@ -46,6 +46,10 @@
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-handler</artifactId>
+ </dependency>
Review comment:
I think this dependency was there on purpose, not to hard code the use of netty in the middle of protocols-api
##########
File path: protocols/api/src/main/java/org/apache/james/protocols/api/Encryption.java
##########
@@ -19,26 +19,41 @@
package org.apache.james.protocols.api;
+import java.util.List;
+
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLEngine;
import org.apache.commons.lang3.ArrayUtils;
+import io.netty.buffer.ByteBufAllocator;
+import io.netty.handler.ssl.JdkSslContext;
+import io.netty.handler.ssl.SslContext;
+import io.netty.handler.ssl.SslContextBuilder;
+
/**
* This class should be used to setup encrypted protocol handling
+ *
+ * It's recommended to use Netty {@link SslContext} in preference to the JDK's {@link SSLContext}.
+ * Netty {@link SslContext} is much easier to configure, use and supports more features.
+ *
+ * Netty {@link SslContext} can be configured to use Google's BoringSSL.
+ * BoringSSL is a clone of OpenSSL with all redundant/old ciphers removed and is compiled into to all the Chrome/Chromium browsers.
+ * Simply include the 'io.netty:netty-tcnative-boringssl-static' dependency as discussed here: https://netty.io/wiki/forked-tomcat-native.html
+ * You should regularly update the version of this library to keep in-sync with the latest version used by Chrome/Chromium browsers.
+ *
+ * Use {@link SslContextBuilder} to create an instance of {@link SslContext}.
*/
-public final class Encryption {
+public abstract class Encryption {
Review comment:
I worked on a clean abstraction, turning Encryption into an interface, moving it to protocols-netty, and providing Netty implementation of it.
Though I failed at using Netty native bindings... netty-ctnative was built against openssl-1.0 while I have openssl-1.1... Feeling sad here!
I'll PR this generalisation of OpenSSL separately...
##########
File path: protocols/api/src/main/java/org/apache/james/protocols/api/ProtocolServer.java
##########
@@ -46,26 +46,26 @@
* Start the server - blocks until server has started.
*/
default void bind() {
- bindAsync().syncUninterruptibly();
+ sync_bind(bindAsync());
Review comment:
syncBind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] glennosss edited a comment on pull request #929: JAMES-3715 Several Netty 4 fixes and improvements to make starting and managing netty servers easier
Posted by GitBox <gi...@apache.org>.
glennosss edited a comment on pull request #929:
URL: https://github.com/apache/james-project/pull/929#issuecomment-1076069682
A shared 'EventLoopGroupManager' is also an important part of one day being able to support a NIO SMTP client: https://issues.apache.org/jira/browse/JAMES-964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org