You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by jugi92 <gi...@git.apache.org> on 2018/08/10 09:59:51 UTC
[GitHub] nifi pull request #2943: NIFI-5282 - Proxy support gcp
GitHub user jugi92 opened a pull request:
https://github.com/apache/nifi/pull/2943
NIFI-5282 - Proxy support gcp
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jugi92/nifi proxy_support_gcp
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2943.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2943
----
commit 76b82e23a16a3ca0b2556d5d3f54140f446c0d9d
Author: Andy LoPresto <al...@...>
Date: 2018-07-07T05:07:46Z
NIFI-5370 removed custom hostname verifier implementation from OkHttpReplicationClient (default handles wildcard certs).
This closes #2869.
Signed-off-by: Mark Payne <ma...@hotmail.com>
commit 333146b3fecef50cc16751cdf2fe7bcddcc7e03d
Author: Mark Bean <ma...@...>
Date: 2018-07-05T19:35:15Z
NIFI-5368 controller services validated prior to enabling; referenced controller services must be enabled for referencing component to be valid (mock framework)
This closes #2873.
Signed-off-by: Mark Payne <ma...@hotmail.com>
commit 54bb511e579a00535c976159fa2903385af74f77
Author: Mark Payne <ma...@...>
Date: 2018-07-09T15:50:06Z
NIFI-5377: Addressed issue of infinite recursion when enabling/disabling controller services if there is a recursive loop (i.e., Service A references Service B references Service A). This closes #2847
Signed-off-by: Matt Gilman <ma...@gmail.com>
commit 260bc29e1014a2fd21c3775e27f434756e13e0ee
Author: Bryan Bende <bb...@...>
Date: 2018-06-25T14:36:55Z
NIFI-5316 Fixed array handling for Avro that comes from Parquet's Avro reader
Signed-off-by: zenfenan <ze...@apache.org>
commit e09ab9b69a3bd21c09118dd2a2c9b8bd4e091bae
Author: Mark Payne <ma...@...>
Date: 2018-06-29T13:25:57Z
NIFI-5361: When submitting many processors to start, calculate the 'timeout timestamp' immediately before calling @OnScheduled method, after the task has been scheduled to run, instead of before the task has a chance to run.
This closes #2831
commit cfdb0de8f83c3e7d732cac6a5d18b2f1631d20e5
Author: Mark Payne <ma...@...>
Date: 2018-07-02T14:00:38Z
NIFI-5362: When a processor is terminated and has no more active threads, ensure that we set this.hasActiveThreads = false
Signed-off-by: Pierre Villard <pi...@gmail.com>
This closes #2832.
commit 7b28b914cd6ca4c5f0d566a8726ca02ebd38f3c2
Author: Peter Toth <pt...@...>
Date: 2018-06-07T10:13:21Z
NIFI-5278: fixes JSON escaping of code
Change-Id: I2cb0e6c658d4a0f2aad9c4aab9201a3334ee54df
NIFI-5278: adds Apache Commons Text to NOTICE
Change-Id: I8185239b0a888c16159b18f13d6682ba350cc766
NIFI-5278: adds tests
Change-Id: I9286ac71bc7399e5bdc1e6602609b5e8829db27e
NIFI-5278: fixes review findings
Change-Id: I292c93dae877cf1cd146f3897b7e132b6afac801
Signed-off-by: Matthew Burgess <ma...@apache.org>
This closes #2768
commit f4b2aae48ae32d2d8c7dbe89d342a749bdc32575
Author: Matthew Burgess <ma...@...>
Date: 2018-06-28T19:02:58Z
NIFI-5349: Fixed EL handling in Initial Max Value props for DB Fetch processors
Signed-off-by: Pierre Villard <pi...@gmail.com>
This closes #2822.
commit d326edb25765c02e66fb16f4b52c47c3bc444f00
Author: thenatog <th...@...>
Date: 2018-06-25T17:23:28Z
NIFI-5258 - Changed the way the servlets are created for the documentation webapp.
Removed some unnecessary code.
Fixed imports.
This closes #2812.
Signed-off-by: Andy LoPresto <al...@apache.org>
commit b46033be306b2ce66c3316b2f4fec92b941307ac
Author: thenatog <th...@...>
Date: 2018-07-03T18:21:03Z
NIFI-5374 - Added ExceptionFilter which catches RequestRejectedException thrown in the nifi-api Jersey code. These exceptions were not caught by the Jetty error-page configuration because they're thrown before the endpoint/Jetty routing is hit.
Added integration test for checking the ExceptionFilter catches malicious string exceptions.
Made minor changes to PR 2840 for code style.
This closes #2840.
Co-authored-by: Andy LoPresto <al...@apache.org>
Signed-off-by: Andy LoPresto <al...@apache.org>
commit 4f75a0b46cd98ad7642ad0cde98e0504881b7751
Author: Mark Bean <ma...@...>
Date: 2018-07-06T12:29:00Z
NIFI-5377 prevent infinite loop if a controller service circular reference exists
commit a618ea55975094087023d02e27a9bd859f671702
Author: Andy LoPresto <al...@...>
Date: 2018-07-12T06:32:23Z
NIFI-5415 Renamed ListenSyslogGroovyTest to ITListenSyslogGroovy.
commit e959630c22c9a52ec717141f6cf9f018830a38bf
Author: Andy LoPresto <al...@...>
Date: 2018-07-12T08:19:19Z
NIFI-5414 Fixed checkstyle error due to unused import.
commit 8bbef92c589d4854ceba50f7e7d47f37bc5c3abc
Author: Andy LoPresto <al...@...>
Date: 2018-07-12T17:35:08Z
NIFI-5414 Manually set pom versions to 1.7.1-SNAPSHOT to allow mvn release plugin to sign release.
commit 241e5411a24435ca2c39ab7bea1c5eb9ed195cb3
Author: Andy LoPresto <al...@...>
Date: 2018-07-12T17:55:33Z
NIFI-5414-RC1 prepare release nifi-1.7.1-RC1
commit 023e0cd3c512f278fb2bd3135843b7c26231fb46
Author: jugi92 <ju...@...>
Date: 2018-08-10T09:48:58Z
added http proxy support with authentication for GCP processors
added proxy support for Google Credential Service
----
---
[GitHub] nifi pull request #2943: NIFI-5282 - Proxy support gcp
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2943#discussion_r214253576
--- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/CredentialPropertyDescriptors.java ---
@@ -88,4 +88,22 @@ private CredentialPropertyDescriptors() {}
.description("The raw JSON containing a Service Account keyfile.")
.sensitive(true)
.build();
+
+ public static final PropertyDescriptor PROXY_HOST = new PropertyDescriptor
+ .Builder().name("gcp-proxy-host")
+ .displayName("Proxy host")
+ .description("IP or hostname of the proxy to be used")
+ .required(false)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor PROXY_PORT = new PropertyDescriptor
+ .Builder().name("gcp-proxy-port")
+ .displayName("Proxy port")
+ .description("Proxy port number")
+ .required(false)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .addValidator(StandardValidators.INTEGER_VALIDATOR)
+ .build();
--- End diff --
I will add ProxyConfigurationService support at GCPCredentialControllerService. But by doing so, I will omit these controller service level properties to configuration and implementation simpler, user will set proxy related config only via ProxyConfigurationService.
---
[GitHub] nifi pull request #2943: NIFI-5282 - Proxy support gcp
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2943
---
[GitHub] nifi issue #2943: NIFI-5282 - Proxy support gcp
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/2943
Hello @jugi92 , thank you very much for your contribution. I stumble upon the same needs and wanted to merged this urgently. Currently, GCP processors do not work if they run on a server which doesn't have direct internet connection and requires forward proxy to go out. Because GCPCredentialControllerService doesn't support proxy configuration.
I am going to create my own PR based on your commit to fix the conflict. I will review your change at the same time and make change if necessary. I hope this approach helps the PR to get merged quickly. Will update you once the new PR is ready. Thanks!
---
[GitHub] nifi pull request #2943: NIFI-5282 - Proxy support gcp
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2943#discussion_r214257349
--- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java ---
@@ -74,6 +76,22 @@
.addValidator(StandardValidators.INTEGER_VALIDATOR)
.build();
+ public static final PropertyDescriptor HTTP_PROXY_USERNAME = new PropertyDescriptor.Builder()
+ .name("Http Proxy Username")
+ .description("Http Proxy Username")
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+ .required(false)
+ .build();
+
+ public static final PropertyDescriptor HTTP_PROXY_PASSWORD = new PropertyDescriptor.Builder()
+ .name("Http Proxy Password")
--- End diff --
Name and displayName should be used.
---
[GitHub] nifi pull request #2943: NIFI-5282 - Proxy support gcp
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2943#discussion_r214253235
--- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/strategies/AbstractServiceAccountCredentialsStrategy.java ---
@@ -36,7 +43,24 @@ public AbstractServiceAccountCredentialsStrategy(String name, PropertyDescriptor
@Override
public GoogleCredentials getGoogleCredentials(Map<PropertyDescriptor, String> properties) throws IOException {
- return GoogleCredentials.fromStream(getServiceAccountJson(properties));
+ final String proxyHost = properties.get(CredentialPropertyDescriptors.PROXY_HOST);
+ final String proxyPortString = properties.get(CredentialPropertyDescriptors.PROXY_PORT);
+ final Integer proxyPort = (proxyPortString != null && proxyPortString.matches("-?\\d+")) ?
+ Integer.parseInt(proxyPortString) : 0;
+
+ if (!StringUtils.isBlank(proxyHost) && proxyPort > 0) {
+ return GoogleCredentials.fromStream(getServiceAccountJson(properties),
+ new HttpTransportFactory() {
+ @Override
+ public HttpTransport create() {
+ return new NetHttpTransport.Builder()
+ .setProxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
+ .build();
+ }
+ });
+ } else {
+ return GoogleCredentials.fromStream(getServiceAccountJson(properties));
+ }
--- End diff --
I'd implement these proxy related code at higher abstraction layer. Will update it at the new PR.
---
[GitHub] nifi pull request #2943: NIFI-5282 - Proxy support gcp
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2943#discussion_r214324506
--- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/AbstractGCSProcessor.java ---
@@ -88,6 +92,15 @@ protected StorageOptions getServiceOptions(ProcessContext context, GoogleCredent
storageOptionsBuilder.setTransportOptions(HttpTransportOptions.newBuilder().setHttpTransportFactory(new HttpTransportFactory() {
@Override
public HttpTransport create() {
+ if (!StringUtils.isBlank(proxyUser) && !StringUtils.isBlank(proxyPassword)) {
+ Authenticator authenticator = new Authenticator() {
+ public PasswordAuthentication getPasswordAuthentication() {
+ return (new PasswordAuthentication(proxyUser,
+ proxyPassword.toCharArray()));
+ }
+ };
+ Authenticator.setDefault(authenticator);
--- End diff --
We'd like to avoid setting default authenticator as it affect system wide. ApacheHttpTransport can be used to add proxy authentication support. I will update it in the new PR.
---
[GitHub] nifi pull request #2943: NIFI-5282 - Proxy support gcp
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2943#discussion_r214257335
--- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java ---
@@ -74,6 +76,22 @@
.addValidator(StandardValidators.INTEGER_VALIDATOR)
.build();
+ public static final PropertyDescriptor HTTP_PROXY_USERNAME = new PropertyDescriptor.Builder()
+ .name("Http Proxy Username")
--- End diff --
Name and displayName should be used.
---