You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/11/08 20:09:37 UTC
[GitHub] [druid] maytasm opened a new pull request #11891: Support routing data through an HTTP proxy
maytasm opened a new pull request #11891:
URL: https://github.com/apache/druid/pull/11891
Support routing data through an HTTP proxy
### Description
This adds the ability for the HttpClient to connect through an HTTP proxy. We
augment the channel factory to check if it is supposed to be proxied and, if so,
we connect to the proxy host first, issue a CONNECT command through to the final
recipient host and *then* give the channel to the normal http client for usage.
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #11891: Support routing data through an HTTP proxy
Posted by GitBox <gi...@apache.org>.
maytasm commented on a change in pull request #11891:
URL: https://github.com/apache/druid/pull/11891#discussion_r745140581
##########
File path: docs/development/modules.md
##########
@@ -356,6 +356,24 @@ druid.coordinator.cleanupMetadata.duty.killSupervisors.retainDuration=PT0M
druid.coordinator.cleanupMetadata.period=PT10S
```
+### Routing data through a HTTP proxy for your extension
+
+It is possible to adds the ability for the HttpClient of your extension to connect through a HTTP proxy.
+Internally, Druid will augment the channel factory to check if it is supposed to be proxied and, if so, connect to the proxy host first,
+issue a CONNECT command through to the final recipient host and then give the channel to the normal http client for usage.
+
+To enable this feature and allow for proxy in your extension's HTTP client,
+first, add `HttpClientProxyConfig` as a `@JsonProperty` to the HTTP config class of your extension.
+Next, in the extension's module class, add `HttpProxyConfig` to `HttpClientConfig`. For example,
+```
+final HttpClientConfig.Builder builder = HttpClientConfig
+ .builder()
+ .withNumConnections(1)
+ .withReadTimeout(config.getReadTimeout().toStandardDuration())
+ .withHttpProxyConfig(config.getProxyConfig());
+```
+(note that the `config` variable in the example above is the HTTP config class of your extension)
Review comment:
Done
##########
File path: docs/development/modules.md
##########
@@ -356,6 +356,24 @@ druid.coordinator.cleanupMetadata.duty.killSupervisors.retainDuration=PT0M
druid.coordinator.cleanupMetadata.period=PT10S
```
+### Routing data through a HTTP proxy for your extension
+
+It is possible to adds the ability for the HttpClient of your extension to connect through a HTTP proxy.
+Internally, Druid will augment the channel factory to check if it is supposed to be proxied and, if so, connect to the proxy host first,
+issue a CONNECT command through to the final recipient host and then give the channel to the normal http client for usage.
+
+To enable this feature and allow for proxy in your extension's HTTP client,
+first, add `HttpClientProxyConfig` as a `@JsonProperty` to the HTTP config class of your extension.
+Next, in the extension's module class, add `HttpProxyConfig` to `HttpClientConfig`. For example,
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #11891: Support routing data through an HTTP proxy
Posted by GitBox <gi...@apache.org>.
maytasm commented on a change in pull request #11891:
URL: https://github.com/apache/druid/pull/11891#discussion_r745139048
##########
File path: docs/development/modules.md
##########
@@ -356,6 +356,24 @@ druid.coordinator.cleanupMetadata.duty.killSupervisors.retainDuration=PT0M
druid.coordinator.cleanupMetadata.period=PT10S
```
+### Routing data through a HTTP proxy for your extension
+
+It is possible to adds the ability for the HttpClient of your extension to connect through a HTTP proxy.
+Internally, Druid will augment the channel factory to check if it is supposed to be proxied and, if so, connect to the proxy host first,
+issue a CONNECT command through to the final recipient host and then give the channel to the normal http client for usage.
Review comment:
Reworded as suggestion and removed implementation details
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm merged pull request #11891: Support routing data through an HTTP proxy
Posted by GitBox <gi...@apache.org>.
maytasm merged pull request #11891:
URL: https://github.com/apache/druid/pull/11891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a change in pull request #11891: Support routing data through an HTTP proxy
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on a change in pull request #11891:
URL: https://github.com/apache/druid/pull/11891#discussion_r745120013
##########
File path: docs/development/modules.md
##########
@@ -356,6 +356,24 @@ druid.coordinator.cleanupMetadata.duty.killSupervisors.retainDuration=PT0M
druid.coordinator.cleanupMetadata.period=PT10S
```
+### Routing data through a HTTP proxy for your extension
+
+It is possible to adds the ability for the HttpClient of your extension to connect through a HTTP proxy.
+Internally, Druid will augment the channel factory to check if it is supposed to be proxied and, if so, connect to the proxy host first,
+issue a CONNECT command through to the final recipient host and then give the channel to the normal http client for usage.
Review comment:
```suggestion
You can add the ability for the `HttpClient` of your extension to connect through an HTTP proxy.
Internally, Druid augments the channel factory to check if it should proxy connections. If so the client connects to the proxy host first to issue a subsequent `CONNECT` command through to the final recipient host. After the connection is established, the channel passes back to the original HTTP client for usage.
```
I'm not sure what "normal" means in the connection here. It seems like the way that Druid handles it by passing the channel back and forth may be an implementation detail. I am wondering if I am enabling proxying why I need to know that?
##########
File path: docs/development/modules.md
##########
@@ -356,6 +356,24 @@ druid.coordinator.cleanupMetadata.duty.killSupervisors.retainDuration=PT0M
druid.coordinator.cleanupMetadata.period=PT10S
```
+### Routing data through a HTTP proxy for your extension
+
+It is possible to adds the ability for the HttpClient of your extension to connect through a HTTP proxy.
+Internally, Druid will augment the channel factory to check if it is supposed to be proxied and, if so, connect to the proxy host first,
+issue a CONNECT command through to the final recipient host and then give the channel to the normal http client for usage.
+
+To enable this feature and allow for proxy in your extension's HTTP client,
+first, add `HttpClientProxyConfig` as a `@JsonProperty` to the HTTP config class of your extension.
+Next, in the extension's module class, add `HttpProxyConfig` to `HttpClientConfig`. For example,
Review comment:
```suggestion
To enable proxy connection for your extension's HTTP client:
1. Add `HttpClientProxyConfig` as a `@JsonProperty` to the HTTP config class of your extension.
2. In the extension's module class, add `HttpProxyConfig` to `HttpClientConfig`. For example, where `HttpClientConfig` is the HTTP config class for your extension:
```
(Not sure if I interpreted this correctly, but include the information from the note on line 375 in the for example statement here)
##########
File path: docs/development/modules.md
##########
@@ -356,6 +356,24 @@ druid.coordinator.cleanupMetadata.duty.killSupervisors.retainDuration=PT0M
druid.coordinator.cleanupMetadata.period=PT10S
```
+### Routing data through a HTTP proxy for your extension
+
+It is possible to adds the ability for the HttpClient of your extension to connect through a HTTP proxy.
+Internally, Druid will augment the channel factory to check if it is supposed to be proxied and, if so, connect to the proxy host first,
+issue a CONNECT command through to the final recipient host and then give the channel to the normal http client for usage.
+
+To enable this feature and allow for proxy in your extension's HTTP client,
+first, add `HttpClientProxyConfig` as a `@JsonProperty` to the HTTP config class of your extension.
+Next, in the extension's module class, add `HttpProxyConfig` to `HttpClientConfig`. For example,
+```
+final HttpClientConfig.Builder builder = HttpClientConfig
+ .builder()
+ .withNumConnections(1)
+ .withReadTimeout(config.getReadTimeout().toStandardDuration())
+ .withHttpProxyConfig(config.getProxyConfig());
+```
+(note that the `config` variable in the example above is the HTTP config class of your extension)
Review comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org