You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by "arturobernalg (via GitHub)" <gi...@apache.org> on 2023/04/01 18:50:42 UTC
[GitHub] [httpcomponents-client] arturobernalg opened a new pull request, #430: Add support for proxy failover
arturobernalg opened a new pull request, #430:
URL: https://github.com/apache/httpcomponents-client/pull/430
This pull request adds support for proxy failover in the Apache HttpClient. The current implementation of the ProxySelectorRoutePlanner selects a single proxy from the list returned by the java.net.ProxySelector API. If the client can't open a socket to the proxy, other proxies in the list are ignored and the request fails. This is not optimal in network configurations where proxies can go up and down for maintenance.
To address this, I have implemented a new FailoverProxySelector class that selects a list of proxies for the given URI by delegating to the next available ProxySelector in the list, until a non-empty proxy list is returned. Once a non-empty proxy list is returned, it becomes the active ProxySelector and is used for subsequent calls to select(). The active ProxySelector is remembered and used for future requests until it fails, at which point the next ProxySelector is used.
This implementation is more efficient than the previous solution, which created a disposable socket for every request. It also provides better support for network configurations where proxies can go up and down for maintenance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] arturobernalg commented on a diff in pull request #430: Add support for proxy failover
Posted by "arturobernalg (via GitHub)" <gi...@apache.org>.
arturobernalg commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1162044106
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/DistributedProxySelector.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A ProxySelector implementation that selects proxies using a distributed strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to distribute proxy selection among different
+ * ProxySelectors.
+ * <p>
+ * This implementation uses a ThreadLocal to store the currently chosen selector
+ * for each thread, ensuring thread-safety. Additionally, it uses a cache to store
+ * previously selected proxies for each URI, improving performance by avoiding
+ * redundant proxy selection calls.
+ *
+ * @since 5.3
+ */
+public class DistributedProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DistributedProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private final ThreadLocal<Integer> currentIndex;
Review Comment:
@ok2c
I have now incorporated this change into the DistributedProxySelector class, which has made the logic more readable and ensures thread safety.
The updated implementation now utilizes ThreadLocal<ProxySelector> for the currentSelector, allowing each thread to have its own instance of the ProxySelector. Additionally, I have updated the connectFailed method to remove the currentSelector from the ThreadLocal variable after handling the connection failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] ok2c commented on a diff in pull request #430: Add support for proxy failover
Posted by "ok2c (via GitHub)" <gi...@apache.org>.
ok2c commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1160485123
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/FailoverProxySelector.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A ProxySelector implementation that selects proxies using a failover strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to failover to a different proxy if the current
+ * one fails.
+ *
+ * @since 5.3
+ */
+public class FailoverProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FailoverProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private final AtomicInteger currentIndex;
+
+
+ /**
+ * Constructs a FailoverProxySelector from the given list of ProxySelectors.
+ *
+ * @param selectors the list of ProxySelectors to use.
+ * @throws IllegalArgumentException if the list is null or empty.
+ */
+ public FailoverProxySelector(final List<ProxySelector> selectors) {
+ if (selectors == null || selectors.isEmpty()) {
+ throw new IllegalArgumentException("At least one ProxySelector is required");
+ }
+ this.selectors = new ArrayList<>(selectors);
+ currentIndex = new AtomicInteger(0);
+
+ }
+
+ /**
+ * Selects a list of proxies for the given URI by delegating to the next
+ * available ProxySelector in the list, until a non-empty proxy list is
+ * returned. Once a non-empty proxy list is returned, it becomes the active
+ * ProxySelector and is used for subsequent calls to select().
+ *
+ * @param uri the URI to select a proxy for.
+ * @return a list of proxies for the given URI.
+ */
+ @Override
+ public List<Proxy> select(final URI uri) {
+ List<Proxy> result = Collections.emptyList();
+ for (int i = 0; i < selectors.size(); i++) {
+ final ProxySelector selector = selectors.get((i + currentIndex.get()) % selectors.size());
+ try {
+ result = selector.select(uri);
+ if (!result.isEmpty()) {
+ currentIndex.set((i + currentIndex.get()) % selectors.size());
+ break;
+ }
+ } catch (final Exception e) {
+ // ignore and try the next selector
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Exception caught while selecting proxy for URI {}: {}", uri, e.getMessage());
+
+ }
+ }
+ }
+ return result;
+ }
+
+
+ /**
+ * Notifies the active ProxySelector that a connection to the given URI
+ * using the given proxy has failed.
+ *
+ * @param uri the URI that failed to connect.
+ * @param sa the SocketAddress of the proxy that failed to connect.
+ * @param ioe the IOException that resulted from the failed connection.
+ */
+ @Override
+ public void connectFailed(final URI uri, final SocketAddress sa, final IOException ioe) {
+ final ProxySelector selector = selectors.get(currentIndex.get());
Review Comment:
@arturobernalg This approach can produce unexpected results in the selector is being used concurrently by multiple threads. The current index can easily change from another thread while the `#select` method is still being called. As much as I hate `ThreadLocal`s I suppose this implementation must maintain a separate state per each thread of execution.
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/FailoverProxySelector.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A ProxySelector implementation that selects proxies using a failover strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to failover to a different proxy if the current
+ * one fails.
+ *
+ * @since 5.3
+ */
+public class FailoverProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FailoverProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private final AtomicInteger currentIndex;
+
+
+ /**
+ * Constructs a FailoverProxySelector from the given list of ProxySelectors.
+ *
+ * @param selectors the list of ProxySelectors to use.
+ * @throws IllegalArgumentException if the list is null or empty.
+ */
+ public FailoverProxySelector(final List<ProxySelector> selectors) {
+ if (selectors == null || selectors.isEmpty()) {
+ throw new IllegalArgumentException("At least one ProxySelector is required");
+ }
+ this.selectors = new ArrayList<>(selectors);
+ currentIndex = new AtomicInteger(0);
+
+ }
+
+ /**
+ * Selects a list of proxies for the given URI by delegating to the next
+ * available ProxySelector in the list, until a non-empty proxy list is
+ * returned. Once a non-empty proxy list is returned, it becomes the active
+ * ProxySelector and is used for subsequent calls to select().
+ *
+ * @param uri the URI to select a proxy for.
+ * @return a list of proxies for the given URI.
+ */
+ @Override
+ public List<Proxy> select(final URI uri) {
Review Comment:
@arturobernalg What is exactly the intent of this implementation? Pick a backup selector in case the primary selector fails to produce a result or distribute the load randomly across multiple selectors? I am not quite sure I understand the logic of this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] arturobernalg commented on pull request #430: Add support for proxy failover
Posted by "arturobernalg (via GitHub)" <gi...@apache.org>.
arturobernalg commented on PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#issuecomment-1494666602
> @arturobernalg Could you please also update `HttpClientBuilder` and make sure both `HttpClientBuilder` and `HttpAsyncClientBuilder` behave consistently?
@ok2c I update the HttpClientBuilder and unified the logic.
TY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] ok2c commented on a diff in pull request #430: Add support for proxy failover
Posted by "ok2c (via GitHub)" <gi...@apache.org>.
ok2c commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1162559293
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/DistributedProxySelector.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.apache.hc.core5.annotation.Contract;
+import org.apache.hc.core5.annotation.ThreadingBehavior;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A DistributedProxySelector is a custom {@link ProxySelector} implementation that
+ * delegates proxy selection to a list of underlying ProxySelectors in a
+ * distributed manner. It ensures that proxy selection is load-balanced
+ * across the available ProxySelectors, and provides thread safety by
+ * maintaining separate states for each thread.
+ *
+ * <p>The DistributedProxySelector class maintains a list of ProxySelectors,
+ * a {@link ThreadLocal} variable for the current {@link ProxySelector}, and an {@link AtomicInteger}
+ * to keep track of the shared index across all threads. When the select()
+ * method is called, it delegates the proxy selection to the current
+ * ProxySelector or the next available one in the list if the current one
+ * returns an empty proxy list. Any exceptions that occur during proxy
+ * selection are caught and ignored, and the next ProxySelector is tried.
+ *
+ * <p>The connectFailed() method notifies the active {@link ProxySelector} of a
+ * connection failure, allowing the underlying ProxySelector to handle
+ * connection failures according to its own logic.
+ *
+ * @since 5.3
+ */
+@Contract(threading = ThreadingBehavior.SAFE)
+public class DistributedProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DistributedProxySelector.class);
+
+ /**
+ * A list of {@link ProxySelector} instances to be used by the DistributedProxySelector
+ * for selecting proxies.
+ */
+ private final List<ProxySelector> selectors;
+
+ /**
+ * A {@link ThreadLocal} variable holding the current {@link ProxySelector} for each thread,
+ * ensuring thread safety when accessing the current {@link ProxySelector}.
+ */
+ private final ThreadLocal<ProxySelector> currentSelector;
+
+ /**
+ * An {@link AtomicInteger} representing the shared index across all threads for
+ * maintaining the current position in the list of ProxySelectors, ensuring
+ * proper distribution of {@link ProxySelector} usage.
+ */
+ private final AtomicInteger sharedIndex;
+
+
+ /**
+ * Constructs a DistributedProxySelector with the given list of {@link ProxySelector}.
+ * The constructor initializes the currentSelector as a {@link ThreadLocal}, and
+ * the sharedIndex as an {@link AtomicInteger}.
+ *
+ * @param selectors the list of ProxySelectors to use.
+ * @throws IllegalArgumentException if the list is null or empty.
+ */
+ public DistributedProxySelector(final List<ProxySelector> selectors) {
+ if (selectors == null || selectors.isEmpty()) {
+ throw new IllegalArgumentException("At least one ProxySelector is required");
+ }
+ this.selectors = new ArrayList<>(selectors);
+ this.currentSelector = new ThreadLocal<>();
+ this.sharedIndex = new AtomicInteger();
+ }
+
+ /**
+ * Selects a list of proxies for the given {@link URI} by delegating to the current
+ * {@link ProxySelector} or the next available {@link ProxySelector} in the list if the current
+ * one returns an empty proxy list. If an {@link Exception} occurs, it will be caught
+ * and ignored, and the next {@link ProxySelector} will be tried.
+ *
+ * @param uri the {@link URI} to select a proxy for.
+ * @return a list of proxies for the given {@link URI}.
+ */
+ @Override
+ public List<Proxy> select(final URI uri) {
+ List<Proxy> result = Collections.emptyList();
+ ProxySelector selector = currentSelector.get();
+
+ for (int i = 0; i < selectors.size(); i++) {
+ if (selector == null) {
+ selector = nextSelector();
+ currentSelector.set(selector);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Current selector is null, selecting next proxy selector for URI {}: {}", uri, selector);
+ }
+ }
+ try {
+ result = selector.select(uri);
Review Comment:
@arturobernalg Almost there. Please store the current selector only for the duration of the `result = selector.select(uri)` call and unset it immediately after (or in case of a failure in `#connectFailed`). The main idea is to use thread local only as short as possible and unset it as soon as possible to avoid thread local leaks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] ok2c commented on a diff in pull request #430: Add support for proxy failover
Posted by "ok2c (via GitHub)" <gi...@apache.org>.
ok2c commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1161242313
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/FailoverProxySelector.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A ProxySelector implementation that selects proxies using a failover strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to failover to a different proxy if the current
+ * one fails.
+ *
+ * @since 5.3
+ */
+public class FailoverProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FailoverProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private final AtomicInteger currentIndex;
+
+
+ /**
+ * Constructs a FailoverProxySelector from the given list of ProxySelectors.
+ *
+ * @param selectors the list of ProxySelectors to use.
+ * @throws IllegalArgumentException if the list is null or empty.
+ */
+ public FailoverProxySelector(final List<ProxySelector> selectors) {
+ if (selectors == null || selectors.isEmpty()) {
+ throw new IllegalArgumentException("At least one ProxySelector is required");
+ }
+ this.selectors = new ArrayList<>(selectors);
+ currentIndex = new AtomicInteger(0);
+
+ }
+
+ /**
+ * Selects a list of proxies for the given URI by delegating to the next
+ * available ProxySelector in the list, until a non-empty proxy list is
+ * returned. Once a non-empty proxy list is returned, it becomes the active
+ * ProxySelector and is used for subsequent calls to select().
+ *
+ * @param uri the URI to select a proxy for.
+ * @return a list of proxies for the given URI.
+ */
+ @Override
+ public List<Proxy> select(final URI uri) {
Review Comment:
@arturobernalg In this case `FailoverProxySelector` may be not the best choice of the name. I personally think `spraying` or distributing` would likely be a better term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] arturobernalg commented on a diff in pull request #430: Add support for proxy failover
Posted by "arturobernalg (via GitHub)" <gi...@apache.org>.
arturobernalg commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1160748793
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/FailoverProxySelector.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A ProxySelector implementation that selects proxies using a failover strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to failover to a different proxy if the current
+ * one fails.
+ *
+ * @since 5.3
+ */
+public class FailoverProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FailoverProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private final AtomicInteger currentIndex;
+
+
+ /**
+ * Constructs a FailoverProxySelector from the given list of ProxySelectors.
+ *
+ * @param selectors the list of ProxySelectors to use.
+ * @throws IllegalArgumentException if the list is null or empty.
+ */
+ public FailoverProxySelector(final List<ProxySelector> selectors) {
+ if (selectors == null || selectors.isEmpty()) {
+ throw new IllegalArgumentException("At least one ProxySelector is required");
+ }
+ this.selectors = new ArrayList<>(selectors);
+ currentIndex = new AtomicInteger(0);
+
+ }
+
+ /**
+ * Selects a list of proxies for the given URI by delegating to the next
+ * available ProxySelector in the list, until a non-empty proxy list is
+ * returned. Once a non-empty proxy list is returned, it becomes the active
+ * ProxySelector and is used for subsequent calls to select().
+ *
+ * @param uri the URI to select a proxy for.
+ * @return a list of proxies for the given URI.
+ */
+ @Override
+ public List<Proxy> select(final URI uri) {
Review Comment:
Is to select a list of proxies for the given URI by delegating to the next available ProxySelector in the list, until a non-empty proxy list is returned. It does not necessarily pick a backup selector in case the primary selector fails to produce a result, but rather rotates through the list of selectors until a non-empty proxy list is obtained
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] ok2c merged pull request #430: Add support for proxy failover
Posted by "ok2c (via GitHub)" <gi...@apache.org>.
ok2c merged PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] ok2c commented on a diff in pull request #430: Add support for proxy failover
Posted by "ok2c (via GitHub)" <gi...@apache.org>.
ok2c commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1156705196
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/FailoverProxySelector.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * A ProxySelector implementation that selects proxies using a failover strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to failover to a different proxy if the current
+ * one fails.
+ *
+ * @since 5.3
+ */
+public class FailoverProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FailoverProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private int currentIndex = 0;
Review Comment:
@arturobernalg I think `currentIndex` should be volatile or represented as an `AtomicInteger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] arturobernalg commented on a diff in pull request #430: Add support for proxy failover
Posted by "arturobernalg (via GitHub)" <gi...@apache.org>.
arturobernalg commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1157517970
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/FailoverProxySelector.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * A ProxySelector implementation that selects proxies using a failover strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to failover to a different proxy if the current
+ * one fails.
+ *
+ * @since 5.3
+ */
+public class FailoverProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FailoverProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private int currentIndex = 0;
Review Comment:
Yes, you are correct. If multiple threads access the currentIndex variable in the FailoverProxySelector class, then it should be made thread-safe.
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] ok2c commented on a diff in pull request #430: Add support for proxy failover
Posted by "ok2c (via GitHub)" <gi...@apache.org>.
ok2c commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1161242712
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/FailoverProxySelector.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A ProxySelector implementation that selects proxies using a failover strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to failover to a different proxy if the current
+ * one fails.
+ *
+ * @since 5.3
+ */
+public class FailoverProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FailoverProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private final AtomicInteger currentIndex;
+
+
+ /**
+ * Constructs a FailoverProxySelector from the given list of ProxySelectors.
+ *
+ * @param selectors the list of ProxySelectors to use.
+ * @throws IllegalArgumentException if the list is null or empty.
+ */
+ public FailoverProxySelector(final List<ProxySelector> selectors) {
+ if (selectors == null || selectors.isEmpty()) {
+ throw new IllegalArgumentException("At least one ProxySelector is required");
+ }
+ this.selectors = new ArrayList<>(selectors);
+ currentIndex = new AtomicInteger(0);
+
+ }
+
+ /**
+ * Selects a list of proxies for the given URI by delegating to the next
+ * available ProxySelector in the list, until a non-empty proxy list is
+ * returned. Once a non-empty proxy list is returned, it becomes the active
+ * ProxySelector and is used for subsequent calls to select().
+ *
+ * @param uri the URI to select a proxy for.
+ * @return a list of proxies for the given URI.
+ */
+ @Override
+ public List<Proxy> select(final URI uri) {
+ List<Proxy> result = Collections.emptyList();
+ for (int i = 0; i < selectors.size(); i++) {
+ final ProxySelector selector = selectors.get((i + currentIndex.get()) % selectors.size());
+ try {
+ result = selector.select(uri);
+ if (!result.isEmpty()) {
+ currentIndex.set((i + currentIndex.get()) % selectors.size());
+ break;
+ }
+ } catch (final Exception e) {
+ // ignore and try the next selector
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Exception caught while selecting proxy for URI {}: {}", uri, e.getMessage());
+
+ }
+ }
+ }
+ return result;
+ }
+
+
+ /**
+ * Notifies the active ProxySelector that a connection to the given URI
+ * using the given proxy has failed.
+ *
+ * @param uri the URI that failed to connect.
+ * @param sa the SocketAddress of the proxy that failed to connect.
+ * @param ioe the IOException that resulted from the failed connection.
+ */
+ @Override
+ public void connectFailed(final URI uri, final SocketAddress sa, final IOException ioe) {
+ final ProxySelector selector = selectors.get(currentIndex.get());
Review Comment:
@arturobernalg Better, but we can make it better still.
1. Use `ThreadLocal` to store the currently chosen selector
2. Unset the `ThreadLocal` once the `#select` call has been completed successfully or unsuccessfully.
3. Use `AtomicInteger` to store the shared current index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] ok2c commented on a diff in pull request #430: Add support for proxy failover
Posted by "ok2c (via GitHub)" <gi...@apache.org>.
ok2c commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1161699458
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/DistributedProxySelector.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A ProxySelector implementation that selects proxies using a distributed strategy.
+ * <p>
+ * This implementation maintains a list of other ProxySelectors, and selects a proxy
+ * by delegating to the next available ProxySelector in the list, until a non-empty
+ * proxy list is returned. If a ProxySelector fails with an exception, the next
+ * ProxySelector in the list is tried. Once a non-empty proxy list is returned by
+ * a ProxySelector, it becomes the active ProxySelector and is used for subsequent
+ * calls to select(). If a connection fails, connectFailed() is called on the
+ * active ProxySelector.
+ * <p>
+ * This implementation is useful in environments where multiple proxy servers may be
+ * used, and where it is desirable to distribute proxy selection among different
+ * ProxySelectors.
+ * <p>
+ * This implementation uses a ThreadLocal to store the currently chosen selector
+ * for each thread, ensuring thread-safety. Additionally, it uses a cache to store
+ * previously selected proxies for each URI, improving performance by avoiding
+ * redundant proxy selection calls.
+ *
+ * @since 5.3
+ */
+public class DistributedProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DistributedProxySelector.class);
+
+ private final List<ProxySelector> selectors;
+ private final ThreadLocal<Integer> currentIndex;
Review Comment:
@arturobernalg
1. You can stick `ProxySelector` into `ThreadLocal`. It will make the logic more readable.
2. Do not forget to remove it in `#connectFailed`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[GitHub] [httpcomponents-client] arturobernalg commented on a diff in pull request #430: Add support for proxy failover
Posted by "arturobernalg (via GitHub)" <gi...@apache.org>.
arturobernalg commented on code in PR #430:
URL: https://github.com/apache/httpcomponents-client/pull/430#discussion_r1163336628
##########
httpclient5/src/main/java/org/apache/hc/client5/http/impl/routing/DistributedProxySelector.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation. For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+package org.apache.hc.client5.http.impl.routing;
+
+import org.apache.hc.core5.annotation.Contract;
+import org.apache.hc.core5.annotation.ThreadingBehavior;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * A DistributedProxySelector is a custom {@link ProxySelector} implementation that
+ * delegates proxy selection to a list of underlying ProxySelectors in a
+ * distributed manner. It ensures that proxy selection is load-balanced
+ * across the available ProxySelectors, and provides thread safety by
+ * maintaining separate states for each thread.
+ *
+ * <p>The DistributedProxySelector class maintains a list of ProxySelectors,
+ * a {@link ThreadLocal} variable for the current {@link ProxySelector}, and an {@link AtomicInteger}
+ * to keep track of the shared index across all threads. When the select()
+ * method is called, it delegates the proxy selection to the current
+ * ProxySelector or the next available one in the list if the current one
+ * returns an empty proxy list. Any exceptions that occur during proxy
+ * selection are caught and ignored, and the next ProxySelector is tried.
+ *
+ * <p>The connectFailed() method notifies the active {@link ProxySelector} of a
+ * connection failure, allowing the underlying ProxySelector to handle
+ * connection failures according to its own logic.
+ *
+ * @since 5.3
+ */
+@Contract(threading = ThreadingBehavior.SAFE)
+public class DistributedProxySelector extends ProxySelector {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DistributedProxySelector.class);
+
+ /**
+ * A list of {@link ProxySelector} instances to be used by the DistributedProxySelector
+ * for selecting proxies.
+ */
+ private final List<ProxySelector> selectors;
+
+ /**
+ * A {@link ThreadLocal} variable holding the current {@link ProxySelector} for each thread,
+ * ensuring thread safety when accessing the current {@link ProxySelector}.
+ */
+ private final ThreadLocal<ProxySelector> currentSelector;
+
+ /**
+ * An {@link AtomicInteger} representing the shared index across all threads for
+ * maintaining the current position in the list of ProxySelectors, ensuring
+ * proper distribution of {@link ProxySelector} usage.
+ */
+ private final AtomicInteger sharedIndex;
+
+
+ /**
+ * Constructs a DistributedProxySelector with the given list of {@link ProxySelector}.
+ * The constructor initializes the currentSelector as a {@link ThreadLocal}, and
+ * the sharedIndex as an {@link AtomicInteger}.
+ *
+ * @param selectors the list of ProxySelectors to use.
+ * @throws IllegalArgumentException if the list is null or empty.
+ */
+ public DistributedProxySelector(final List<ProxySelector> selectors) {
+ if (selectors == null || selectors.isEmpty()) {
+ throw new IllegalArgumentException("At least one ProxySelector is required");
+ }
+ this.selectors = new ArrayList<>(selectors);
+ this.currentSelector = new ThreadLocal<>();
+ this.sharedIndex = new AtomicInteger();
+ }
+
+ /**
+ * Selects a list of proxies for the given {@link URI} by delegating to the current
+ * {@link ProxySelector} or the next available {@link ProxySelector} in the list if the current
+ * one returns an empty proxy list. If an {@link Exception} occurs, it will be caught
+ * and ignored, and the next {@link ProxySelector} will be tried.
+ *
+ * @param uri the {@link URI} to select a proxy for.
+ * @return a list of proxies for the given {@link URI}.
+ */
+ @Override
+ public List<Proxy> select(final URI uri) {
+ List<Proxy> result = Collections.emptyList();
+ ProxySelector selector = currentSelector.get();
+
+ for (int i = 0; i < selectors.size(); i++) {
+ if (selector == null) {
+ selector = nextSelector();
+ currentSelector.set(selector);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Current selector is null, selecting next proxy selector for URI {}: {}", uri, selector);
+ }
+ }
+ try {
+ result = selector.select(uri);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org