You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/09/06 12:42:09 UTC
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #897: MINIFICPP-XXXX Implement and test proxy handling in InvokeHTTP
lordgamez opened a new pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484263666
##########
File path: extensions/http-curl/processors/InvokeHTTP.cpp
##########
@@ -246,6 +246,20 @@ void InvokeHTTP::onSchedule(const std::shared_ptr<core::ProcessContext> &context
if (context->getProperty(DisablePeerVerification.getName(), disablePeerVerification)) {
utils::StringUtils::StringToBool(disablePeerVerification, disable_peer_verification_);
}
+
+ std::string proxy_value;
+ if (context->getProperty(ProxyHost.getName(), proxy_value) && !proxy_value.empty()) {
+ proxy_.host = proxy_value;
+ }
+ if (context->getProperty(ProxyPort.getName(), proxy_value) && !proxy_value.empty()) {
+ proxy_.port = std::stoi(proxy_value);
Review comment:
`getProperty` is a templated method, that depending on the second argument's type carries out checks and conversions from `std::string`, so feel free to pass `proxy.port` as the second argument here, moreover I believe `getProperty` adheres to the contract "the out argument is not modified on failure, missing value", so we don't even need the "if"s here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484451982
##########
File path: docker/test/integration/test_http.py
##########
@@ -34,3 +33,26 @@ def test_invoke_listen():
cluster.deploy_flow(invoke_flow, name='minifi-invoke')
assert cluster.check_output()
+
+def test_invoke_listen_with_proxy():
+ """
+ Verify sending through a proxy using InvokeHTTP to a receiver using ListenHTTP.
+ """
+ invoke_flow = (GetFile('/tmp/input')
+ >> LogAttribute()
+ >> InvokeHTTP('http://minifi-listen:8080/contentListener',
+ method='POST',
+ proxy_host='http-proxy',
+ proxy_port='3128',
+ proxy_username='admin',
+ proxy_password='test101'))
+
+ listen_flow = ListenHTTP(8080) >> LogAttribute() >> PutFile('/tmp/output')
+
+ with DockerTestCluster(SingleFileOutputValidator('test')) as cluster:
+ cluster.put_test_data('test')
+ cluster.deploy_flow(None, engine='http-proxy')
+ cluster.deploy_flow(listen_flow, name='minifi-listen')
+ cluster.deploy_flow(invoke_flow, name='minifi-invoke')
+
+ assert cluster.check_output()
Review comment:
Added check in [5047001](https://github.com/apache/nifi-minifi-cpp/pull/897/commits/504700148228b571ee69a2d58322c235d5f71428)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484364547
##########
File path: libminifi/include/utils/ByteArrayCallback.h
##########
@@ -57,6 +57,7 @@ class ByteInputCallBack : public InputStreamCallback {
}
virtual void seek(size_t pos) {
+ ptr = &vec[pos];
Review comment:
Curl proxy authentication requires 2 steps (getting the available authentication methods then authenticate with the chosen one) and after the first step it needs a data rewind which requires implementing the seek function. You are right, it broke the getBuffer that's what partly caused the test failures, it is fixed in [fd3a7c6](https://github.com/apache/nifi-minifi-cpp/pull/897/commits/fd3a7c653202c7cdd460153175d3deb13b7651ce)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484365864
##########
File path: libminifi/include/utils/HTTPClient.h
##########
@@ -300,6 +319,9 @@ class BaseHTTPClient {
virtual void setUploadCallback(HTTPUploadCallback *callbackObj) {
}
+ virtual void setSeekFunction(HTTPUploadCallback *callbackObj) {
+ }
Review comment:
Done in [fd3a7c6](https://github.com/apache/nifi-minifi-cpp/pull/897/commits/fd3a7c653202c7cdd460153175d3deb13b7651ce)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484279314
##########
File path: libminifi/include/utils/HTTPClient.h
##########
@@ -300,6 +319,9 @@ class BaseHTTPClient {
virtual void setUploadCallback(HTTPUploadCallback *callbackObj) {
}
+ virtual void setSeekFunction(HTTPUploadCallback *callbackObj) {
+ }
Review comment:
it seems like `HTTPClient` is the only class inheriting from it, and noone directly instantiates a `BaseHTTPClient`, could we make these noop virtual methods pure virtual?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484364547
##########
File path: libminifi/include/utils/ByteArrayCallback.h
##########
@@ -57,6 +57,7 @@ class ByteInputCallBack : public InputStreamCallback {
}
virtual void seek(size_t pos) {
+ ptr = &vec[pos];
Review comment:
Curl proxy authentication requires 2 steps (getting the available authentication methods then authenticate with the chosen one) and after the first step it needs a data rewind which requires implementing the seek function. You are right, it broke the getBuffer that's what partly caused the test failures, it is fixed in [e0e31c5](https://github.com/apache/nifi-minifi-cpp/pull/897/commits/e0e31c5e96e0ad29c54ca09230a845588664b864)
##########
File path: extensions/http-curl/processors/InvokeHTTP.h
##########
@@ -155,8 +155,10 @@ class InvokeHTTP : public core::Processor {
bool use_chunked_encoding_{false};
// penalize on no retry
bool penalize_no_retry_{false};
- // disable peer verification ( makes susceptible for MITM attacks )
- bool disable_peer_verification_{false};
+ bool disable_peer_verification_{false}; // ( makes susceptible for MITM attacks )
+ bool follow_redirects_{true};
Review comment:
Fixed in [e0e31c5](https://github.com/apache/nifi-minifi-cpp/pull/897/commits/e0e31c5e96e0ad29c54ca09230a845588664b864)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484405587
##########
File path: libminifi/include/utils/ByteArrayCallback.h
##########
@@ -65,7 +66,7 @@ class ByteInputCallBack : public InputStreamCallback {
}
virtual char *getBuffer(size_t pos) {
- return ptr + pos;
+ return reinterpret_cast<char*>(&vec[0]) + pos;
Review comment:
Please use `vec.data()` to avoid out-of-bounds indexing when `vec` is empty, or index with pos.
```suggestion
return &vec[pos];
```
or
```suggestion
return vec.data() + pos;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484279314
##########
File path: libminifi/include/utils/HTTPClient.h
##########
@@ -300,6 +319,9 @@ class BaseHTTPClient {
virtual void setUploadCallback(HTTPUploadCallback *callbackObj) {
}
+ virtual void setSeekFunction(HTTPUploadCallback *callbackObj) {
+ }
Review comment:
it seems like `HTTPClient` is the only class inheriting from it, and noone directly instantiates a BaseHTTPClient, could we make these noop virtual methods pure virtual?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484451198
##########
File path: libminifi/include/utils/ByteArrayCallback.h
##########
@@ -65,7 +66,7 @@ class ByteInputCallBack : public InputStreamCallback {
}
virtual char *getBuffer(size_t pos) {
- return ptr + pos;
+ return reinterpret_cast<char*>(&vec[0]) + pos;
Review comment:
Done in [839ed47](https://github.com/apache/nifi-minifi-cpp/pull/897/commits/839ed47d2c3618a1fb1443f0929bfc8908051a87)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
szaszm closed pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
hunyadi-dev commented on pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#issuecomment-688205425
Please check the CI errors, they seem related to the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484365267
##########
File path: docker/test/integration/test_http.py
##########
@@ -34,3 +33,26 @@ def test_invoke_listen():
cluster.deploy_flow(invoke_flow, name='minifi-invoke')
assert cluster.check_output()
+
+def test_invoke_listen_with_proxy():
+ """
+ Verify sending through a proxy using InvokeHTTP to a receiver using ListenHTTP.
+ """
+ invoke_flow = (GetFile('/tmp/input')
+ >> LogAttribute()
+ >> InvokeHTTP('http://minifi-listen:8080/contentListener',
+ method='POST',
+ proxy_host='http-proxy',
+ proxy_port='3128',
+ proxy_username='admin',
+ proxy_password='test101'))
+
+ listen_flow = ListenHTTP(8080) >> LogAttribute() >> PutFile('/tmp/output')
+
+ with DockerTestCluster(SingleFileOutputValidator('test')) as cluster:
+ cluster.put_test_data('test')
+ cluster.deploy_flow(None, engine='http-proxy')
+ cluster.deploy_flow(listen_flow, name='minifi-listen')
+ cluster.deploy_flow(invoke_flow, name='minifi-invoke')
+
+ assert cluster.check_output()
Review comment:
We could check the acces logs of the http-proxy, that's what I was checking manually, I will add it to the test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484344801
##########
File path: docker/test/integration/test_http.py
##########
@@ -34,3 +33,26 @@ def test_invoke_listen():
cluster.deploy_flow(invoke_flow, name='minifi-invoke')
assert cluster.check_output()
+
+def test_invoke_listen_with_proxy():
+ """
+ Verify sending through a proxy using InvokeHTTP to a receiver using ListenHTTP.
+ """
+ invoke_flow = (GetFile('/tmp/input')
+ >> LogAttribute()
+ >> InvokeHTTP('http://minifi-listen:8080/contentListener',
+ method='POST',
+ proxy_host='http-proxy',
+ proxy_port='3128',
+ proxy_username='admin',
+ proxy_password='test101'))
+
+ listen_flow = ListenHTTP(8080) >> LogAttribute() >> PutFile('/tmp/output')
+
+ with DockerTestCluster(SingleFileOutputValidator('test')) as cluster:
+ cluster.put_test_data('test')
+ cluster.deploy_flow(None, engine='http-proxy')
+ cluster.deploy_flow(listen_flow, name='minifi-listen')
+ cluster.deploy_flow(invoke_flow, name='minifi-invoke')
+
+ assert cluster.check_output()
Review comment:
Do we have a way to verify that the traffic really went through the proxy and didn't bypass it like before this PR?
##########
File path: libminifi/include/utils/ByteArrayCallback.h
##########
@@ -57,6 +57,7 @@ class ByteInputCallBack : public InputStreamCallback {
}
virtual void seek(size_t pos) {
+ ptr = &vec[pos];
Review comment:
What's the purpose of this change? I think this may break `getBuffer`.
##########
File path: extensions/http-curl/processors/InvokeHTTP.h
##########
@@ -155,8 +155,10 @@ class InvokeHTTP : public core::Processor {
bool use_chunked_encoding_{false};
// penalize on no retry
bool penalize_no_retry_{false};
- // disable peer verification ( makes susceptible for MITM attacks )
- bool disable_peer_verification_{false};
+ bool disable_peer_verification_{false}; // ( makes susceptible for MITM attacks )
+ bool follow_redirects_{true};
Review comment:
I think `follow_redirects_` is unused
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
arpadboda commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484283890
##########
File path: extensions/http-curl/processors/InvokeHTTP.cpp
##########
@@ -246,6 +246,20 @@ void InvokeHTTP::onSchedule(const std::shared_ptr<core::ProcessContext> &context
if (context->getProperty(DisablePeerVerification.getName(), disablePeerVerification)) {
utils::StringUtils::StringToBool(disablePeerVerification, disable_peer_verification_);
}
+
+ std::string proxy_value;
+ if (context->getProperty(ProxyHost.getName(), proxy_value) && !proxy_value.empty()) {
+ proxy_.host = proxy_value;
+ }
+ if (context->getProperty(ProxyPort.getName(), proxy_value) && !proxy_value.empty()) {
+ proxy_.port = std::stoi(proxy_value);
Review comment:
The other reason to remove if statements is updating the config: we should null the members, otherwise proxy properties can't be removed without restarting minifi.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #897: MINIFICPP-1357 Implement and test proxy handling in InvokeHTTP
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #897:
URL: https://github.com/apache/nifi-minifi-cpp/pull/897#discussion_r484405469
##########
File path: extensions/http-curl/processors/InvokeHTTP.cpp
##########
@@ -246,6 +246,20 @@ void InvokeHTTP::onSchedule(const std::shared_ptr<core::ProcessContext> &context
if (context->getProperty(DisablePeerVerification.getName(), disablePeerVerification)) {
utils::StringUtils::StringToBool(disablePeerVerification, disable_peer_verification_);
}
+
+ std::string proxy_value;
+ if (context->getProperty(ProxyHost.getName(), proxy_value) && !proxy_value.empty()) {
+ proxy_.host = proxy_value;
+ }
+ if (context->getProperty(ProxyPort.getName(), proxy_value) && !proxy_value.empty()) {
+ proxy_.port = std::stoi(proxy_value);
Review comment:
Done in [2bc2b75](https://github.com/apache/nifi-minifi-cpp/pull/897/commits/429675f5d9a526c529080f4468d170e320f21dca)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org