You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by benqiu2016 <gi...@git.apache.org> on 2017/06/21 21:11:23 UTC

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

GitHub user benqiu2016 opened a pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114

    site2site port negotiation

    Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
    
    In order to streamline the review of the contribution we ask you
    to ensure the following steps have been taken:
    
    ### For all changes:
    - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
         in the commit message?
    
    - [ ] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    - [ ] Is your initial contribution a single, squashed commit?
    
    ### For code changes:
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [ ] If applicable, have you updated the LICENSE file?
    - [ ] If applicable, have you updated the NOTICE file?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
    
    ### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/benqiu2016/nifi-minifi-cpp port_negotiation

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi-minifi-cpp/pull/114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #114
    
----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126214904
  
    --- Diff: libminifi/include/utils/HTTPUtils.h ---
    @@ -88,6 +90,40 @@ struct HTTPRequestResponse {
     
     };
     
    +static void parse_url(std::string &url, std::string &host, int &port, std::string &protocol) {
    --- End diff --
    
    IPv6 was just one example. Another case would be username:password@example.com (Not that that is a typical NiFi deployment case, but for the sake of argument as we can't anticipate how these tools will be deployed in every scenario.) I am not saying we necessarily need full RFC URI/L compliance, but I do think whatever is / is not supported in terms of remote s2s urls should be documented somewhere that the users / admins can find it, or logged if parsing fails so that users can more easily discover at runtime why they cannot connect to their IPv6 URL :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126458724
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +194,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    --- End diff --
    
    Thanks for posting that @benqiu2016. I learned something new about the NiFI REST API :).
    
    Are you planning to add support for default protocol ports?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126457339
  
    --- Diff: libminifi/include/utils/HTTPUtils.h ---
    @@ -88,6 +90,40 @@ struct HTTPRequestResponse {
     
     };
     
    +static void parse_url(std::string &url, std::string &host, int &port, std::string &protocol) {
    --- End diff --
    
    Sure, I think documentation of the YAML format is a good thing to have, and probably also a method comment on the interface for HTTPUtils for future developers to know what format the method expects to be passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123527427
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -54,19 +57,29 @@ bool create = true) {
       std::unique_ptr<Site2SiteClientProtocol> nextProtocol = nullptr;
       if (!available_protocols_.try_dequeue(nextProtocol)) {
         if (create) {
    +      refreshPeerList();
           // create
    -      nextProtocol = std::unique_ptr<Site2SiteClientProtocol>(new Site2SiteClientProtocol(nullptr));
    -      nextProtocol->setPortId(protocol_uuid_);
    -      std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str = std::unique_ptr<org::apache::nifi::minifi::io::DataStream>(stream_factory_->createSocket(host_, port_));
    -      std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr<Site2SitePeer>(new Site2SitePeer(std::move(str), host_, port_));
    -      nextProtocol->setPeer(std::move(peer_));
    +      for (auto peer : site2site_peer_status_list_) {
    +        std::unique_ptr<Site2SiteClientProtocol> protocol = nullptr;
    +        protocol = std::unique_ptr < Site2SiteClientProtocol
    +            > (new Site2SiteClientProtocol(nullptr));
    +        protocol->setPortId(protocol_uuid_);
    +        std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str =
    +            std::unique_ptr < org::apache::nifi::minifi::io::DataStream
    +                > (stream_factory_->createSocket(peer.host_, peer.port_));
    +        std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr < Site2SitePeer
    +            > (new Site2SitePeer(std::move(str), peer.host_, peer.port_));
    +        protocol->setPeer(std::move(peer_));
    +        returnProtocol(std::move(protocol));
    +      }
         }
    +    available_protocols_.try_dequeue(nextProtocol);
       }
       return std::move(nextProtocol);
     }
     
     void RemoteProcessorGroupPort::returnProtocol(std::unique_ptr<Site2SiteClientProtocol> return_protocol) {
    -  if (available_protocols_.size_approx() >= max_concurrent_tasks_) {
    +  if (available_protocols_.size_approx() >= (max_concurrent_tasks_ * site2site_peer_status_list_.size())) {
    --- End diff --
    
    i need to round robin the peer even for a single thread to distribute the load cross all peers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123500940
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -278,26 +301,28 @@ bool Site2SiteClientProtocol::handShake() {
       }
     
       switch (code) {
    -    case PROPERTIES_OK:
    -      logger_->log_info("Site2Site HandShake Completed");
    -      _peerState = HANDSHAKED;
    -      return true;
    -    case PORT_NOT_IN_VALID_STATE:
    -    case UNKNOWN_PORT:
    -    case PORTS_DESTINATION_FULL:
    -      logger_->log_error("Site2Site HandShake Failed because destination port is either invalid or full");
    -      ret = -1;
    -      /*
    -       peer_->yield();
    -       tearDown(); */
    -      return false;
    -    default:
    -      logger_->log_info("HandShake Failed because of unknown respond code %d", code);
    -      ret = -1;
    -      /*
    -       peer_->yield();
    -       tearDown(); */
    -      return false;
    +  case PROPERTIES_OK:
    +    logger_->log_info("Site2Site HandShake Completed");
    +    _peerState = HANDSHAKED;
    +    return true;
    +  case PORT_NOT_IN_VALID_STATE:
    +  case UNKNOWN_PORT:
    +  case PORTS_DESTINATION_FULL:
    +    logger_->log_error(
    +        "Site2Site HandShake Failed because destination port is either invalid or full");
    +    ret = -1;
    +    /*
    --- End diff --
    
    is the commented code needed here? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123768801
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -54,19 +57,29 @@ bool create = true) {
       std::unique_ptr<Site2SiteClientProtocol> nextProtocol = nullptr;
       if (!available_protocols_.try_dequeue(nextProtocol)) {
         if (create) {
    +      refreshPeerList();
           // create
    -      nextProtocol = std::unique_ptr<Site2SiteClientProtocol>(new Site2SiteClientProtocol(nullptr));
    -      nextProtocol->setPortId(protocol_uuid_);
    -      std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str = std::unique_ptr<org::apache::nifi::minifi::io::DataStream>(stream_factory_->createSocket(host_, port_));
    -      std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr<Site2SitePeer>(new Site2SitePeer(std::move(str), host_, port_));
    -      nextProtocol->setPeer(std::move(peer_));
    +      for (auto peer : site2site_peer_status_list_) {
    +        std::unique_ptr<Site2SiteClientProtocol> protocol = nullptr;
    +        protocol = std::unique_ptr < Site2SiteClientProtocol
    +            > (new Site2SiteClientProtocol(nullptr));
    +        protocol->setPortId(protocol_uuid_);
    +        std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str =
    +            std::unique_ptr < org::apache::nifi::minifi::io::DataStream
    +                > (stream_factory_->createSocket(peer.host_, peer.port_));
    +        std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr < Site2SitePeer
    +            > (new Site2SitePeer(std::move(str), peer.host_, peer.port_));
    +        protocol->setPeer(std::move(peer_));
    +        returnProtocol(std::move(protocol));
    +      }
         }
    +    available_protocols_.try_dequeue(nextProtocol);
       }
       return std::move(nextProtocol);
     }
     
     void RemoteProcessorGroupPort::returnProtocol(std::unique_ptr<Site2SiteClientProtocol> return_protocol) {
    -  if (available_protocols_.size_approx() >= max_concurrent_tasks_) {
    +  if (available_protocols_.size_approx() >= (max_concurrent_tasks_ * site2site_peer_status_list_.size())) {
    --- End diff --
    
    I think the logic works for a single thread, but what happens in the case where you have ten threads, returning at different times? Typical algorithms use multiple queues for fairness. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    Thanks for the review. Can we merge the PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123500630
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -38,7 +39,8 @@ namespace minifi {
     
     bool Site2SiteClientProtocol::establish() {
       if (_peerState != IDLE) {
    -    logger_->log_error("Site2Site peer state is not idle while try to establish");
    +    logger_->log_error(
    --- End diff --
    
    Why the change in log statements? Our line length is 200. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123528501
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -278,26 +301,28 @@ bool Site2SiteClientProtocol::handShake() {
       }
     
       switch (code) {
    -    case PROPERTIES_OK:
    -      logger_->log_info("Site2Site HandShake Completed");
    -      _peerState = HANDSHAKED;
    -      return true;
    -    case PORT_NOT_IN_VALID_STATE:
    -    case UNKNOWN_PORT:
    -    case PORTS_DESTINATION_FULL:
    -      logger_->log_error("Site2Site HandShake Failed because destination port is either invalid or full");
    -      ret = -1;
    -      /*
    -       peer_->yield();
    -       tearDown(); */
    -      return false;
    -    default:
    -      logger_->log_info("HandShake Failed because of unknown respond code %d", code);
    -      ret = -1;
    -      /*
    -       peer_->yield();
    -       tearDown(); */
    -      return false;
    +  case PROPERTIES_OK:
    +    logger_->log_info("Site2Site HandShake Completed");
    +    _peerState = HANDSHAKED;
    +    return true;
    +  case PORT_NOT_IN_VALID_STATE:
    +  case UNKNOWN_PORT:
    +  case PORTS_DESTINATION_FULL:
    +    logger_->log_error(
    +        "Site2Site HandShake Failed because destination port is either invalid or full");
    +    ret = -1;
    +    /*
    --- End diff --
    
    will remove that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    Looks good, but am curious about the use of lock vs a stack based lock guard. What was the reason for that? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126208386
  
    --- Diff: libminifi/include/utils/HTTPUtils.h ---
    @@ -88,6 +90,40 @@ struct HTTPRequestResponse {
     
     };
     
    +static void parse_url(std::string &url, std::string &host, int &port, std::string &protocol) {
    --- End diff --
    
    now it was designed for support ipv4 now. If we need a RFC compliant URL parser, we can certainly use a open source compliant URL parser.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r127535449
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -84,19 +107,41 @@ void RemoteProcessorGroupPort::initialize() {
       std::set<core::Relationship> relationships;
       relationships.insert(relation);
       setSupportedRelationships(relationships);
    +  curl_global_init(CURL_GLOBAL_DEFAULT);
    +  site2site_peer_mutex_.lock();
    --- End diff --
    
    I agree this seems to be a good use case for lock_guard as the lock is obtained at the start of initialize() and released just before returning.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123529970
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -344,7 +427,8 @@ int Site2SiteClientProtocol::readRequestType(RequestType &type) {
       return -1;
     }
     
    -int Site2SiteClientProtocol::readRespond(RespondCode &code, std::string &message) {
    +int Site2SiteClientProtocol::readRespond(RespondCode &code,
    --- End diff --
    
    i will revert that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    @phrocker used https://github.com/apache/nifi-minifi-cpp/pull/119 which has a single merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126172307
  
    --- Diff: libminifi/include/Site2SiteClientProtocol.h ---
    @@ -387,6 +387,15 @@ class DataPacket {
     
     };
     
    +/**
    +  * Site2Site Peer
    +  */
    + typedef struct Site2SitePeerStatus {
    +   std::string host_;
    +   int port_;
    +   bool isSecure_;
    + } Site2SitePeerStatus;
    --- End diff --
    
    What about flowFileCount, the number of flow files the peer holds? That is discoverable using the `/site-to-site/peers` endpoint and could be useful if the client wants to use it for load balancing decisions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123528403
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -38,7 +39,8 @@ namespace minifi {
     
     bool Site2SiteClientProtocol::establish() {
       if (_peerState != IDLE) {
    -    logger_->log_error("Site2Site peer state is not idle while try to establish");
    +    logger_->log_error(
    --- End diff --
    
    OK, i will change the format, i want to reduce the line length to make it easy to read


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123530192
  
    --- Diff: libminifi/src/core/yaml/YamlConfiguration.cpp ---
    @@ -314,10 +314,8 @@ void YamlConfiguration::parseProvenanceReportingYaml(YAML::Node *reportNode, cor
       auto schedulingStrategyStr = node["scheduling strategy"].as<std::string>();
       checkRequiredField(&node, "scheduling period", CONFIG_YAML_PROVENANCE_REPORT_KEY);
       auto schedulingPeriodStr = node["scheduling period"].as<std::string>();
    -  checkRequiredField(&node, "host", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto hostStr = node["host"].as<std::string>();
    -  checkRequiredField(&node, "port", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto portStr = node["port"].as<std::string>();
    +  checkRequiredField(&node, "url", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    --- End diff --
    
    i want to avoid confusion so that we have a consistent way to config the same. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126240211
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -20,6 +20,9 @@
     
     #include "../include/RemoteProcessorGroupPort.h"
     
    +#include <curl/curl.h>
    +#include <curl/curlbuild.h>
    +#include <curl/easy.h>
    --- End diff --
    
    for now, i think we are tied couple curl with our code like others InvokeHttp, etc. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126239506
  
    --- Diff: libminifi/include/utils/HTTPUtils.h ---
    @@ -88,6 +90,40 @@ struct HTTPRequestResponse {
     
     };
     
    +static void parse_url(std::string &url, std::string &host, int &port, std::string &protocol) {
    --- End diff --
    
    for now, we can document the same in the README.md to specify the URL format if it is OK with you. 
    As for the access control, i looked at the doc, it looks like we need to use token, etc for a secure cluster. I will need to ask around.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126160245
  
    --- Diff: libminifi/include/RemoteProcessorGroupPort.h ---
    @@ -84,6 +93,17 @@ class RemoteProcessorGroupPort : public core::Processor {
       void setTransmitting(bool val) {
         transmitting_ = val;
       }
    +  // setURL
    +  void setURL(std::string val) {
    +    url_ = val;
    +    utils::parse_url(url_, host_, port_, protocol_);
    --- End diff --
    
    do we need a null check here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    @benqiu2016 thanks for making that round of changes! They look good. I'll be happy to test the branch when a get a chance to setup a NiFi environment that I can use to test some of the more advanced configurations that are supported (clustered peers, secure client). I should have a chance to do that this week, hopefully in the next couple days. As a general comment, it looks like there are quite a few cases that are not covered by corresponding automated tests so a lot of this will rely manual verification. You may want to open a JIRA ticket for adding some additional unit tests and integration tests as a way to verify this feature over time as changes are made to the codebase. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126207949
  
    --- Diff: libminifi/include/RemoteProcessorGroupPort.h ---
    @@ -84,6 +93,17 @@ class RemoteProcessorGroupPort : public core::Processor {
       void setTransmitting(bool val) {
         transmitting_ = val;
       }
    +  // setURL
    +  void setURL(std::string val) {
    +    url_ = val;
    +    utils::parse_url(url_, host_, port_, protocol_);
    --- End diff --
    
    if the val is empty, the url will be empty and other will empty also.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123769093
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -54,19 +57,29 @@ bool create = true) {
       std::unique_ptr<Site2SiteClientProtocol> nextProtocol = nullptr;
       if (!available_protocols_.try_dequeue(nextProtocol)) {
         if (create) {
    +      refreshPeerList();
           // create
    -      nextProtocol = std::unique_ptr<Site2SiteClientProtocol>(new Site2SiteClientProtocol(nullptr));
    -      nextProtocol->setPortId(protocol_uuid_);
    -      std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str = std::unique_ptr<org::apache::nifi::minifi::io::DataStream>(stream_factory_->createSocket(host_, port_));
    -      std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr<Site2SitePeer>(new Site2SitePeer(std::move(str), host_, port_));
    -      nextProtocol->setPeer(std::move(peer_));
    +      for (auto peer : site2site_peer_status_list_) {
    --- End diff --
    
    What is the time to establish sockets? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126455887
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -84,19 +107,41 @@ void RemoteProcessorGroupPort::initialize() {
       std::set<core::Relationship> relationships;
       relationships.insert(relation);
       setSupportedRelationships(relationships);
    +  curl_global_init(CURL_GLOBAL_DEFAULT);
    +  site2site_peer_mutex_.lock();
    --- End diff --
    
    why did you choose to use lock over a  lock guard? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123501252
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -344,7 +427,8 @@ int Site2SiteClientProtocol::readRequestType(RequestType &type) {
       return -1;
     }
     
    -int Site2SiteClientProtocol::readRespond(RespondCode &code, std::string &message) {
    +int Site2SiteClientProtocol::readRespond(RespondCode &code,
    --- End diff --
    
    Why the change in the function prototype? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126210180
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +194,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    --- End diff --
    
    the /nifi-api/controller return 
    it return the site2site port and we can use that to do the site2site negotiation with master to find the peer info
    {
      "revision": {
        "clientId": "d40fb824-070b-4547-9b1c-f50f5ba0a677"
      },
      "controller": {
        "id": "fe4a3a42-53b6-4af1-a80d-6fdfe60de97f",
        "name": "NiFi Flow",
        "comments": "",
        "runningCount": 3,
        "stoppedCount": 2,
        "invalidCount": 0,
        "disabledCount": 0,
        "inputPortCount": 1,
        "outputPortCount": 1,
        "remoteSiteListeningPort": 10001,
        "siteToSiteSecure": false,
        "instanceId": "9d841c51-ab00-422e-811c-53c6dc2f8e59",
        "inputPorts": [
          {
            "id": "471deef6-2a6e-4a7d-912a-81cc17e3a204",
            "name": " From Node A",
            "comments": "",
            "state": "RUNNING"
          }
        ],
        "outputPorts": [
          {
            "id": "75f88005-0a87-4fef-8320-6219cdbcf18b",
            "name": "To A",
            "comments": "",
            "state": "STOPPED"
          }
        ]
      }
    }


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 closed the pull request at:

    https://github.com/apache/nifi-minifi-cpp/pull/114


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r127531375
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +194,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    --- End diff --
    
    I looked into it further and found out that `/controller` is a NiFi 0.x endpoint. The `/site-to-site` endpoint was introduced in NiFi 1.x, and NiFi 1.x maps `/controller` to `/site-to-site` so that NiFi 0.x instances can communicate with NiFI 1.x instances.
    
    I wasn't sure what minifi-cpp should do. `/controller` currently works for both major versions of NiFi, but there is no guarantee it will forever, and `/controller` is not documented in the 1.x REST API, making the intended behavior of this client code a little less obvious/discoverable for future developers (it was not obvious to me just from the API documentation). The `/site-to-site` endpoint is documented, but would not be available on 0.x NiFi instances.
    
    I decided to look at the minifi-java implementation. It uses the site-to-site client module from NiFi, which has this behavior: It tries `/site-to-site` first, and if it gets a 404 (resource not found) error, then it falls back to `/controller`. From [SiteToSiteRestAPiClient](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/util/SiteToSiteRestApiClient.java):
    
    ```java
    private ControllerDTO getController() throws IOException {
            try {
                HttpGet get = this.createGetControllerRequest();
                return ((ControllerEntity)this.execute(get, ControllerEntity.class)).getController();
            } catch (SiteToSiteRestApiClient.HttpGetFailedException var3) {
                if(404 == var3.getResponseCode()) {
                    logger.debug("getController received NOT_FOUND, trying to access the old NiFi version resource url...");
                    HttpGet get = this.createGet("/controller");
                    return ((ControllerEntity)this.execute(get, ControllerEntity.class)).getController();
                } else {
                    throw var3;
                }
            }
        }
    
        private HttpGet createGetControllerRequest() {
            HttpGet get = this.createGet("/site-to-site");
            get.setHeader("x-nifi-site-to-site-protocol-version", String.valueOf(this.transportProtocolVersionNegotiator.getVersion()));
            return get;
        }
    ```
    
    That seems like a reasonable approach for minifi-cpp as well. I'm interested in hearing thoughts from others. @apiri - should we try to keep compatibility with 0.x NiFi's in this implementation? If not, we could just go with `/site-to-site` in this implementation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123526843
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -54,19 +57,29 @@ bool create = true) {
       std::unique_ptr<Site2SiteClientProtocol> nextProtocol = nullptr;
       if (!available_protocols_.try_dequeue(nextProtocol)) {
         if (create) {
    +      refreshPeerList();
           // create
    -      nextProtocol = std::unique_ptr<Site2SiteClientProtocol>(new Site2SiteClientProtocol(nullptr));
    -      nextProtocol->setPortId(protocol_uuid_);
    -      std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str = std::unique_ptr<org::apache::nifi::minifi::io::DataStream>(stream_factory_->createSocket(host_, port_));
    -      std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr<Site2SitePeer>(new Site2SitePeer(std::move(str), host_, port_));
    -      nextProtocol->setPeer(std::move(peer_));
    +      for (auto peer : site2site_peer_status_list_) {
    --- End diff --
    
    i want to prepopulate to reduce the time to establish the socket/negotiation. Also we need to round robin them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123503614
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -54,19 +57,29 @@ bool create = true) {
       std::unique_ptr<Site2SiteClientProtocol> nextProtocol = nullptr;
       if (!available_protocols_.try_dequeue(nextProtocol)) {
         if (create) {
    +      refreshPeerList();
           // create
    -      nextProtocol = std::unique_ptr<Site2SiteClientProtocol>(new Site2SiteClientProtocol(nullptr));
    -      nextProtocol->setPortId(protocol_uuid_);
    -      std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str = std::unique_ptr<org::apache::nifi::minifi::io::DataStream>(stream_factory_->createSocket(host_, port_));
    -      std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr<Site2SitePeer>(new Site2SitePeer(std::move(str), host_, port_));
    -      nextProtocol->setPeer(std::move(peer_));
    +      for (auto peer : site2site_peer_status_list_) {
    +        std::unique_ptr<Site2SiteClientProtocol> protocol = nullptr;
    +        protocol = std::unique_ptr < Site2SiteClientProtocol
    +            > (new Site2SiteClientProtocol(nullptr));
    +        protocol->setPortId(protocol_uuid_);
    +        std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str =
    +            std::unique_ptr < org::apache::nifi::minifi::io::DataStream
    +                > (stream_factory_->createSocket(peer.host_, peer.port_));
    +        std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr < Site2SitePeer
    +            > (new Site2SitePeer(std::move(str), peer.host_, peer.port_));
    +        protocol->setPeer(std::move(peer_));
    +        returnProtocol(std::move(protocol));
    +      }
         }
    +    available_protocols_.try_dequeue(nextProtocol);
       }
       return std::move(nextProtocol);
     }
     
     void RemoteProcessorGroupPort::returnProtocol(std::unique_ptr<Site2SiteClientProtocol> return_protocol) {
    -  if (available_protocols_.size_approx() >= max_concurrent_tasks_) {
    +  if (available_protocols_.size_approx() >= (max_concurrent_tasks_ * site2site_peer_status_list_.size())) {
    --- End diff --
    
    Why should this change? 
    With 10 concurrent tasks ( threads ) and 10 peers, you need 100 queued elements? Shouldn't you only need ten 
     ( +/- a few ) since you only have ten threads to pull objects? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126211327
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +194,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    +
    +  this->site2site_port_ = -1;
    +  CURL *http_session = curl_easy_init();
    +
    +  curl_easy_setopt(http_session, CURLOPT_URL, fullUrl.c_str());
    +
    +  utils::HTTPRequestResponse content;
    +  curl_easy_setopt(http_session, CURLOPT_WRITEFUNCTION,
    +      &utils::HTTPRequestResponse::recieve_write);
    +
    +  curl_easy_setopt(http_session, CURLOPT_WRITEDATA,
    +      static_cast<void*>(&content));
    +
    +  CURLcode res = curl_easy_perform(http_session);
    +
    +  if (res == CURLE_OK) {
    +    std::string response_body(content.data.begin(), content.data.end());
    +    int64_t http_code = 0;
    +    curl_easy_getinfo(http_session, CURLINFO_RESPONSE_CODE, &http_code);
    +    char *content_type;
    +    /* ask for the content-type */
    +    curl_easy_getinfo(http_session, CURLINFO_CONTENT_TYPE, &content_type);
    +
    +    bool isSuccess = ((int32_t) (http_code / 100)) == 2
    +        && res != CURLE_ABORTED_BY_CALLBACK;
    +    bool body_empty = IsNullOrEmpty(content.data);
    +
    +    if (isSuccess && !body_empty) {
    +      std::string controller = std::move(response_body);
    +      logger_->log_debug("controller config %s", controller.c_str());
    +      Json::Value value;
    +      Json::Reader reader;
    +      bool parsingSuccessful = reader.parse(controller, value);
    +      if (parsingSuccessful && !value.empty()) {
    +        Json::Value controllerValue = value["controller"];
    +        if (!controllerValue.empty()) {
    +          Json::Value port = controllerValue["remoteSiteListeningPort"];
    +          if (!port.empty())
    +            this->site2site_port_ = port.asInt();
    +          Json::Value secure = controllerValue["siteToSiteSecure"];
    +          if (!secure.empty())
    +            this->site2site_secure_ = secure.asBool();
    +        }
    +        logger_->log_info("process group remote site2site port %d, is secure %d", site2site_port_, site2site_secure_);
    +      }
    +    } else {
    +      logger_->log_error("Cannot output body to content for ProcessGroup::refreshRemoteSite2SiteInfo");
    +    }
    +  } else {
    +    logger_->log_error(
    +        "ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed %s\n",
    +        curl_easy_strerror(res));
    +  }
    +  curl_easy_cleanup(http_session);
    +}
    +
    +void RemoteProcessorGroupPort::refreshPeerList() {
    +  refreshRemoteSite2SiteInfo();
    +  if (site2site_port_ == -1)
    +    return;
    +
    +  this->site2site_peer_status_list_.clear();
    --- End diff --
    
    refreshRemoteSite2SiteInfo can fail.
    if it is failed. it will set the this->site2site_port_ = -1;
    if that's the case, we do not need to clear the list.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    @phrocker please review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123527545
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -76,27 +89,20 @@ void RemoteProcessorGroupPort::returnProtocol(std::unique_ptr<Site2SiteClientPro
     void RemoteProcessorGroupPort::initialize() {
     // Set the supported properties
       std::set<core::Property> properties;
    -  properties.insert(hostName);
    -  properties.insert(port);
       properties.insert(portUUID);
       setSupportedProperties(properties);
     // Set the supported relationships
       std::set<core::Relationship> relationships;
       relationships.insert(relation);
       setSupportedRelationships(relationships);
    +  curl_global_init(CURL_GLOBAL_DEFAULT);
    --- End diff --
    
    OK.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123503771
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -54,19 +57,29 @@ bool create = true) {
       std::unique_ptr<Site2SiteClientProtocol> nextProtocol = nullptr;
       if (!available_protocols_.try_dequeue(nextProtocol)) {
         if (create) {
    +      refreshPeerList();
           // create
    -      nextProtocol = std::unique_ptr<Site2SiteClientProtocol>(new Site2SiteClientProtocol(nullptr));
    -      nextProtocol->setPortId(protocol_uuid_);
    -      std::unique_ptr<org::apache::nifi::minifi::io::DataStream> str = std::unique_ptr<org::apache::nifi::minifi::io::DataStream>(stream_factory_->createSocket(host_, port_));
    -      std::unique_ptr<Site2SitePeer> peer_ = std::unique_ptr<Site2SitePeer>(new Site2SitePeer(std::move(str), host_, port_));
    -      nextProtocol->setPeer(std::move(peer_));
    +      for (auto peer : site2site_peer_status_list_) {
    --- End diff --
    
    Could you not create one on demand and simply return it versus enqueueing it and dequeuing it? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126455434
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -20,6 +20,9 @@
     
     #include "../include/RemoteProcessorGroupPort.h"
     
    +#include <curl/curl.h>
    +#include <curl/curlbuild.h>
    +#include <curl/easy.h>
    --- End diff --
    
    InvokeHTTP's sole responsibility is HTTP communications. I don't think the same translates to RemoteProcessGroupPort. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126168614
  
    --- Diff: libminifi/include/utils/HTTPUtils.h ---
    @@ -88,6 +90,40 @@ struct HTTPRequestResponse {
     
     };
     
    +static void parse_url(std::string &url, std::string &host, int &port, std::string &protocol) {
    --- End diff --
    
    This URL parsing method is a subset of valid URLs. Are we ok with that? For example, it will fail if passed a Literal IPv6 URL as defined by [RFC 3986](https://tools.ietf.org/html/rfc3986) or [RFC 2732](https://tools.ietf.org/html/rfc2732), e.g.:
    
     `http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/nifi`
    
    If we are ok with this parser being designed to work with URLs of a certain format, we should document those assumptions of inputs.
    
    If we would like this part of the code to be a bit more robust for cases we haven't thought of (both present and future), we could consider adding a dependency to a RFC-compliant URI parser, or, if we want to avoid adding a dependency, just expanding the logic to cover more valid URLs.
    
    Lastly, should the url input arg be const and checked for null?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123748661
  
    --- Diff: libminifi/src/core/yaml/YamlConfiguration.cpp ---
    @@ -314,10 +314,8 @@ void YamlConfiguration::parseProvenanceReportingYaml(YAML::Node *reportNode, cor
       auto schedulingStrategyStr = node["scheduling strategy"].as<std::string>();
       checkRequiredField(&node, "scheduling period", CONFIG_YAML_PROVENANCE_REPORT_KEY);
       auto schedulingPeriodStr = node["scheduling period"].as<std::string>();
    -  checkRequiredField(&node, "host", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto hostStr = node["host"].as<std::string>();
    -  checkRequiredField(&node, "port", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto portStr = node["port"].as<std::string>();
    +  checkRequiredField(&node, "url", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    --- End diff --
    
    If we tagged today would people who have tried MiNiFi in cpp 0.2.0 be able to use the same config files in cpp 0.3.0? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    @phrocker @kevdoran use lock, extract token code, set default port. we need to support /nifi-api/controller which is common to 1.x and 0.x. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123529541
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -319,6 +344,64 @@ void Site2SiteClientProtocol::tearDown() {
       _peerState = IDLE;
     }
     
    +bool Site2SiteClientProtocol::getPeerList(std::vector<minifi::Site2SitePeerStatus> &peer) {
    +  if (establish() && handShake()) {
    +    int status = this->writeRequestType(REQUEST_PEER_LIST);
    +
    +    if (status <= 0) {
    +      tearDown();
    +      return false;
    +    }
    +
    +    uint32_t number;
    +    status = peer_->read(number);
    +
    +    if (status <= 0) {
    +      tearDown();
    +      return false;
    +    }
    +
    +    for (int i = 0; i < number; i++) {
    --- End diff --
    
    it is config by the NiFI cluster server to let client evenly distribute the node cross peers.
    MiNiFI java is doing the similar thing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126181684
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +194,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    --- End diff --
    
    If port is not available this should default to 80 for http and 443 for https.
    
    Also, I'm new to the NiFi REST API... what does /nifi-api/controller return? the site2site impl's I see pull from /site-to-site/peers for this info. curious about the difference (will try it out later today)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123529125
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +147,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    +
    +  this->site2site_port_ = -1;
    +  CURL *http_session = curl_easy_init();
    +
    +  curl_easy_setopt(http_session, CURLOPT_URL, fullUrl.c_str());
    +
    +  utils::HTTPRequestResponse content;
    +  curl_easy_setopt(http_session, CURLOPT_WRITEFUNCTION,
    +      &utils::HTTPRequestResponse::recieve_write);
    +
    +  curl_easy_setopt(http_session, CURLOPT_WRITEDATA,
    +      static_cast<void*>(&content));
    +
    +  CURLcode res = curl_easy_perform(http_session);
    +
    +  if (res == CURLE_OK) {
    +    std::string response_body(content.data.begin(), content.data.end());
    +    int64_t http_code = 0;
    +    curl_easy_getinfo(http_session, CURLINFO_RESPONSE_CODE, &http_code);
    +    char *content_type;
    +    /* ask for the content-type */
    +    curl_easy_getinfo(http_session, CURLINFO_CONTENT_TYPE, &content_type);
    +
    +    bool isSuccess = ((int32_t) (http_code / 100)) == 2
    +        && res != CURLE_ABORTED_BY_CALLBACK;
    +    bool body_empty = IsNullOrEmpty(content.data);
    +
    +    if (isSuccess && !body_empty) {
    +      std::string controller = std::move(response_body);
    +      logger_->log_debug("controller config %s", controller.c_str());
    +      Json::Value value;
    +      Json::Reader reader;
    +      bool parsingSuccessful = reader.parse(controller, value);
    +      if (parsingSuccessful && !value.empty()) {
    +        Json::Value controllerValue = value["controller"];
    +        if (!controllerValue.empty()) {
    +          Json::Value port = controllerValue["remoteSiteListeningPort"];
    +          if (!port.empty())
    +            this->site2site_port_ = port.asInt();
    +          Json::Value secure = controllerValue["siteToSiteSecure"];
    +          if (!secure.empty())
    +            this->site2site_secure_ = secure.asBool();
    +        }
    +        logger_->log_info("process group remote site2site port %d, is secure %d", site2site_port_, site2site_secure_);
    +      }
    +    } else {
    +      logger_->log_error("Cannot output body to content for ProcessGroup::refreshRemoteSite2SiteInfo");
    +    }
    +  } else {
    +    logger_->log_error(
    +        "ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed %s\n",
    +        curl_easy_strerror(res));
    +  }
    +  curl_easy_cleanup(http_session);
    +}
    +
    +void RemoteProcessorGroupPort::refreshPeerList() {
    +  refreshRemoteSite2SiteInfo();
    +  if (site2site_port_ == -1)
    +    return;
    +
    +  this->site2site_peer_status_list_.clear();
    +
    +  std::unique_ptr < Site2SiteClientProtocol> protocol;
    +  protocol = std::unique_ptr < Site2SiteClientProtocol
    +      > (new Site2SiteClientProtocol(nullptr));
    --- End diff --
    
    refresh is done once, i would like to avoid hanging a protocol around.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126173718
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +194,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    +
    +  this->site2site_port_ = -1;
    +  CURL *http_session = curl_easy_init();
    +
    +  curl_easy_setopt(http_session, CURLOPT_URL, fullUrl.c_str());
    +
    +  utils::HTTPRequestResponse content;
    +  curl_easy_setopt(http_session, CURLOPT_WRITEFUNCTION,
    +      &utils::HTTPRequestResponse::recieve_write);
    +
    +  curl_easy_setopt(http_session, CURLOPT_WRITEDATA,
    +      static_cast<void*>(&content));
    +
    +  CURLcode res = curl_easy_perform(http_session);
    +
    +  if (res == CURLE_OK) {
    +    std::string response_body(content.data.begin(), content.data.end());
    +    int64_t http_code = 0;
    +    curl_easy_getinfo(http_session, CURLINFO_RESPONSE_CODE, &http_code);
    +    char *content_type;
    +    /* ask for the content-type */
    +    curl_easy_getinfo(http_session, CURLINFO_CONTENT_TYPE, &content_type);
    +
    +    bool isSuccess = ((int32_t) (http_code / 100)) == 2
    +        && res != CURLE_ABORTED_BY_CALLBACK;
    +    bool body_empty = IsNullOrEmpty(content.data);
    +
    +    if (isSuccess && !body_empty) {
    +      std::string controller = std::move(response_body);
    +      logger_->log_debug("controller config %s", controller.c_str());
    +      Json::Value value;
    +      Json::Reader reader;
    +      bool parsingSuccessful = reader.parse(controller, value);
    +      if (parsingSuccessful && !value.empty()) {
    +        Json::Value controllerValue = value["controller"];
    +        if (!controllerValue.empty()) {
    +          Json::Value port = controllerValue["remoteSiteListeningPort"];
    +          if (!port.empty())
    +            this->site2site_port_ = port.asInt();
    +          Json::Value secure = controllerValue["siteToSiteSecure"];
    +          if (!secure.empty())
    +            this->site2site_secure_ = secure.asBool();
    +        }
    +        logger_->log_info("process group remote site2site port %d, is secure %d", site2site_port_, site2site_secure_);
    +      }
    +    } else {
    +      logger_->log_error("Cannot output body to content for ProcessGroup::refreshRemoteSite2SiteInfo");
    +    }
    +  } else {
    +    logger_->log_error(
    +        "ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed %s\n",
    +        curl_easy_strerror(res));
    +  }
    +  curl_easy_cleanup(http_session);
    +}
    +
    +void RemoteProcessorGroupPort::refreshPeerList() {
    +  refreshRemoteSite2SiteInfo();
    +  if (site2site_port_ == -1)
    +    return;
    +
    +  this->site2site_peer_status_list_.clear();
    --- End diff --
    
    Can the logic below this fail under certain conditions? If so, is it work waiting to clear the current list until we are sure we have an updated list?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123504131
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -76,27 +89,20 @@ void RemoteProcessorGroupPort::returnProtocol(std::unique_ptr<Site2SiteClientPro
     void RemoteProcessorGroupPort::initialize() {
     // Set the supported properties
       std::set<core::Property> properties;
    -  properties.insert(hostName);
    -  properties.insert(port);
       properties.insert(portUUID);
       setSupportedProperties(properties);
     // Set the supported relationships
       std::set<core::Relationship> relationships;
       relationships.insert(relation);
       setSupportedRelationships(relationships);
    +  curl_global_init(CURL_GLOBAL_DEFAULT);
    --- End diff --
    
    the function is not thread safe. initialization isn't guaranteed to be thread safe either since other components could initialize concurrently. It's probably fine for now but seems like this should be done in a thread safe way. 
    https://curl.haxx.se/libcurl/c/curl_global_init.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123501069
  
    --- Diff: libminifi/src/Site2SiteClientProtocol.cpp ---
    @@ -319,6 +344,64 @@ void Site2SiteClientProtocol::tearDown() {
       _peerState = IDLE;
     }
     
    +bool Site2SiteClientProtocol::getPeerList(std::vector<minifi::Site2SitePeerStatus> &peer) {
    +  if (establish() && handShake()) {
    +    int status = this->writeRequestType(REQUEST_PEER_LIST);
    +
    +    if (status <= 0) {
    +      tearDown();
    +      return false;
    +    }
    +
    +    uint32_t number;
    +    status = peer_->read(number);
    +
    +    if (status <= 0) {
    +      tearDown();
    +      return false;
    +    }
    +
    +    for (int i = 0; i < number; i++) {
    --- End diff --
    
    Does it make sense to have a limit on number so we don't have a virtual DoS?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    @phrocker please review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123502816
  
    --- Diff: libminifi/src/core/yaml/YamlConfiguration.cpp ---
    @@ -314,10 +314,8 @@ void YamlConfiguration::parseProvenanceReportingYaml(YAML::Node *reportNode, cor
       auto schedulingStrategyStr = node["scheduling strategy"].as<std::string>();
       checkRequiredField(&node, "scheduling period", CONFIG_YAML_PROVENANCE_REPORT_KEY);
       auto schedulingPeriodStr = node["scheduling period"].as<std::string>();
    -  checkRequiredField(&node, "host", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto hostStr = node["host"].as<std::string>();
    -  checkRequiredField(&node, "port", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto portStr = node["port"].as<std::string>();
    +  checkRequiredField(&node, "url", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    --- End diff --
    
    We should maintain prior functionality. Though we're not GA, we should avoid jarring consumers who may be using a prior config. We should able to support both conditionally. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    @phrocker if we use a stack based lock guard, in order to control the period of lock, i need to add scope for the lock guard inside {}. it make code less readable.
    Anyway. i can change my code to use that if you prefer the same.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by benqiu2016 <gi...@git.apache.org>.
Github user benqiu2016 commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126209183
  
    --- Diff: libminifi/include/Site2SiteClientProtocol.h ---
    @@ -387,6 +387,15 @@ class DataPacket {
     
     };
     
    +/**
    +  * Site2Site Peer
    +  */
    + typedef struct Site2SitePeerStatus {
    +   std::string host_;
    +   int port_;
    +   bool isSecure_;
    + } Site2SitePeerStatus;
    --- End diff --
    
    yes, there is a flowFileCount, i want to reduce the complexity of looking at flowFileCount and doing round robin. If we need to use flowFileCount, it means that we need refresh the peer status periodically and even that, we can only catch a snapshot of the flowFileCount based on the refreshing interval, it is not accurate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123764442
  
    --- Diff: libminifi/src/core/yaml/YamlConfiguration.cpp ---
    @@ -314,10 +314,8 @@ void YamlConfiguration::parseProvenanceReportingYaml(YAML::Node *reportNode, cor
       auto schedulingStrategyStr = node["scheduling strategy"].as<std::string>();
       checkRequiredField(&node, "scheduling period", CONFIG_YAML_PROVENANCE_REPORT_KEY);
       auto schedulingPeriodStr = node["scheduling period"].as<std::string>();
    -  checkRequiredField(&node, "host", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto hostStr = node["host"].as<std::string>();
    -  checkRequiredField(&node, "port", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    -  auto portStr = node["port"].as<std::string>();
    +  checkRequiredField(&node, "url", CONFIG_YAML_PROVENANCE_REPORT_KEY);
    --- End diff --
    
    I agree it would be good to support both. We might be able to support both in the same config loader by preferring the new way and falling back to the old way, or if not we could version the flow file YAML schema similar to the Java implementation of this YAML config loader: https://github.com/apache/nifi-minifi/tree/master/minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r126219277
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -20,6 +20,9 @@
     
     #include "../include/RemoteProcessorGroupPort.h"
     
    +#include <curl/curl.h>
    +#include <curl/curlbuild.h>
    +#include <curl/easy.h>
    --- End diff --
    
    Encapsulating the curl / http communication in this implementation has some pros and cons. On the upside, it abstracts the details of the network communication from the users of the interface, making calling it simple. A downside is that it's harder to unit test the logic in this class as the networking component is not mockable / stub-able without setting up an HTTP server. Also if we ever want to change HTTP client libraries, it is more work as it is tightly coupled here. I don't know if it's worth changing now as it would be a significant refactoring, and it's certainly not just in this case but something that goes for a lot of our code base.  It would be nice if there was more decoupled modularity in these components.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on the issue:

    https://github.com/apache/nifi-minifi-cpp/pull/114
  
    @benqiu2016 Can you rebase? https://github.com/apache/nifi-minifi-cpp/commit/4042aa0ed2cd16aee740978e3575cc58a7e9cff7 appears to be a merge commit since there are two parents for this commit. Can you rebase and we'll get it merged, thanks. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #114: site2site port negotiation

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/114#discussion_r123500565
  
    --- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
    @@ -150,6 +147,87 @@ void RemoteProcessorGroupPort::onTrigger(core::ProcessContext *context, core::Pr
       return;
     }
     
    +void RemoteProcessorGroupPort::refreshRemoteSite2SiteInfo() {
    +  if (this->host_.empty() || this->port_ == -1 || this->protocol_.empty())
    +      return;
    +
    +  std::string fullUrl = this->protocol_ + this->host_ + ":" + std::to_string(this->port_) + "/nifi-api/controller/";
    +
    +  this->site2site_port_ = -1;
    +  CURL *http_session = curl_easy_init();
    +
    +  curl_easy_setopt(http_session, CURLOPT_URL, fullUrl.c_str());
    +
    +  utils::HTTPRequestResponse content;
    +  curl_easy_setopt(http_session, CURLOPT_WRITEFUNCTION,
    +      &utils::HTTPRequestResponse::recieve_write);
    +
    +  curl_easy_setopt(http_session, CURLOPT_WRITEDATA,
    +      static_cast<void*>(&content));
    +
    +  CURLcode res = curl_easy_perform(http_session);
    +
    +  if (res == CURLE_OK) {
    +    std::string response_body(content.data.begin(), content.data.end());
    +    int64_t http_code = 0;
    +    curl_easy_getinfo(http_session, CURLINFO_RESPONSE_CODE, &http_code);
    +    char *content_type;
    +    /* ask for the content-type */
    +    curl_easy_getinfo(http_session, CURLINFO_CONTENT_TYPE, &content_type);
    +
    +    bool isSuccess = ((int32_t) (http_code / 100)) == 2
    +        && res != CURLE_ABORTED_BY_CALLBACK;
    +    bool body_empty = IsNullOrEmpty(content.data);
    +
    +    if (isSuccess && !body_empty) {
    +      std::string controller = std::move(response_body);
    +      logger_->log_debug("controller config %s", controller.c_str());
    +      Json::Value value;
    +      Json::Reader reader;
    +      bool parsingSuccessful = reader.parse(controller, value);
    +      if (parsingSuccessful && !value.empty()) {
    +        Json::Value controllerValue = value["controller"];
    +        if (!controllerValue.empty()) {
    +          Json::Value port = controllerValue["remoteSiteListeningPort"];
    +          if (!port.empty())
    +            this->site2site_port_ = port.asInt();
    +          Json::Value secure = controllerValue["siteToSiteSecure"];
    +          if (!secure.empty())
    +            this->site2site_secure_ = secure.asBool();
    +        }
    +        logger_->log_info("process group remote site2site port %d, is secure %d", site2site_port_, site2site_secure_);
    +      }
    +    } else {
    +      logger_->log_error("Cannot output body to content for ProcessGroup::refreshRemoteSite2SiteInfo");
    +    }
    +  } else {
    +    logger_->log_error(
    +        "ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed %s\n",
    +        curl_easy_strerror(res));
    +  }
    +  curl_easy_cleanup(http_session);
    +}
    +
    +void RemoteProcessorGroupPort::refreshPeerList() {
    +  refreshRemoteSite2SiteInfo();
    +  if (site2site_port_ == -1)
    +    return;
    +
    +  this->site2site_peer_status_list_.clear();
    +
    +  std::unique_ptr < Site2SiteClientProtocol> protocol;
    +  protocol = std::unique_ptr < Site2SiteClientProtocol
    +      > (new Site2SiteClientProtocol(nullptr));
    --- End diff --
    
    why the need to create a new object each refresh? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---