You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by arpadboda <gi...@git.apache.org> on 2018/10/18 09:16:56 UTC

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

GitHub user arpadboda opened a pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421

    Minificpp 641 - C API: add support to register failure callback

    Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
    
    In order to streamline the review of the contribution we ask you
    to ensure the following steps have been taken:
    
    ### For all changes:
    - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
         in the commit message?
    
    - [ ] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    - [ ] Is your initial contribution a single, squashed commit?
    
    ### For code changes:
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [ ] If applicable, have you updated the LICENSE file?
    - [ ] If applicable, have you updated the NOTICE file?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
    
    ### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-641

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi-minifi-cpp/pull/421.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #421
    
----
commit b3c292c2bb0f326d2d39c6620ce4cc7d73f5a393
Author: Arpad Boda <ab...@...>
Date:   2018-10-09T15:56:38Z

    MINIFICPP-638 - C API: add unit tests

commit 00376b84fea6497d62c995575f1377439e678480
Author: Arpad Boda <ab...@...>
Date:   2018-10-16T14:33:09Z

    MINIFICPP-641 - C API: add support to register failure callback

----


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227422808
  
    --- Diff: libminifi/include/capi/Plan.h ---
    @@ -93,11 +95,50 @@ class ExecutionPlan {
         next_ff_ = ptr;
       }
     
    +  static std::shared_ptr<core::Processor> createProcessor(const std::string &processor_name, const std::string &name);
    +
      protected:
    +  class FailureHandler {
    +   public:
    +    FailureHandler() {
    +      callback_ = nullptr;
    +    }
    +    void setCallback(void (*onerror_callback)(const flow_file_record*)) {
    +      callback_=onerror_callback;
    +    }
    +    void operator()(const processor_session* ps)
    +    {
    --- End diff --
    
    These are features that exist within NiFi and are things users will undoubtedly expect. Rollback is a feature that enables us to roll back that flow file. As you state in the comment in capi.h it is the dev's responsibility to copy the content -- but the content could be modified at this point and not the original content -- that's something that should happen at rollback. Therefore even left it at at this state, they are potentially losing any ability to correct or even copy the data before a processor may have failed or modified the content. Happy to move this to a ticket referencing this PR if you want to create one, that way we can implement that discussion before release and avoid keeping this PR alive forever. Also happy to keep discussion here, too. 


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by arpadboda <gi...@git.apache.org>.
Github user arpadboda commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227361733
  
    --- Diff: libminifi/include/capi/Plan.h ---
    @@ -93,11 +95,50 @@ class ExecutionPlan {
         next_ff_ = ptr;
       }
     
    +  static std::shared_ptr<core::Processor> createProcessor(const std::string &processor_name, const std::string &name);
    +
      protected:
    +  class FailureHandler {
    +   public:
    +    FailureHandler() {
    +      callback_ = nullptr;
    +    }
    +    void setCallback(void (*onerror_callback)(const flow_file_record*)) {
    +      callback_=onerror_callback;
    +    }
    +    void operator()(const processor_session* ps)
    +    {
    --- End diff --
    
    Makes sense, but I'm not aware of all possible use-cases here, so let's discuss before doing further implementation. 


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by arpadboda <gi...@git.apache.org>.
Github user arpadboda commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227360200
  
    --- Diff: libminifi/include/capi/Plan.h ---
    @@ -93,11 +95,50 @@ class ExecutionPlan {
         next_ff_ = ptr;
       }
     
    +  static std::shared_ptr<core::Processor> createProcessor(const std::string &processor_name, const std::string &name);
    +
      protected:
    +  class FailureHandler {
    +   public:
    +    FailureHandler() {
    +      callback_ = nullptr;
    +    }
    +    void setCallback(void (*onerror_callback)(const flow_file_record*)) {
    +      callback_=onerror_callback;
    +    }
    +    void operator()(const processor_session* ps)
    +    {
    +      auto ses = static_cast<core::ProcessSession*>(ps->session);
    +
    +      auto ff = ses->get();
    +      if (ff == nullptr) {
    +        return;
    +      }
    +      auto claim = ff->getResourceClaim();
    +
    +      if (claim != nullptr && callback_ != nullptr) {
    +        // create a flow file.
    +        auto path = claim->getContentFullPath();
    +        auto ffr = create_ff_object_na(path.c_str(), path.length(), ff->getSize());
    +        ffr->attributes = ff->getAttributesPtr();
    +        ffr->ffp = ff.get();
    +        callback_(ffr);
    +      }
    +      // This deletes the content of the flowfile as ff gets out of scope
    +      // It's the users responsibility to copy all the data
    +      ses->remove(ff);
    +
    +    }
    +   private:
    +    void (*callback_)(const flow_file_record*);
    --- End diff --
    
    Done


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226306211
  
    --- Diff: libminifi/include/capi/Plan.h ---
    @@ -93,11 +95,50 @@ class ExecutionPlan {
         next_ff_ = ptr;
       }
     
    +  static std::shared_ptr<core::Processor> createProcessor(const std::string &processor_name, const std::string &name);
    +
      protected:
    +  class FailureHandler {
    +   public:
    +    FailureHandler() {
    +      callback_ = nullptr;
    +    }
    +    void setCallback(void (*onerror_callback)(const flow_file_record*)) {
    +      callback_=onerror_callback;
    +    }
    +    void operator()(const processor_session* ps)
    +    {
    +      auto ses = static_cast<core::ProcessSession*>(ps->session);
    +
    +      auto ff = ses->get();
    +      if (ff == nullptr) {
    +        return;
    +      }
    +      auto claim = ff->getResourceClaim();
    +
    +      if (claim != nullptr && callback_ != nullptr) {
    +        // create a flow file.
    +        auto path = claim->getContentFullPath();
    +        auto ffr = create_ff_object_na(path.c_str(), path.length(), ff->getSize());
    +        ffr->attributes = ff->getAttributesPtr();
    +        ffr->ffp = ff.get();
    +        callback_(ffr);
    +      }
    +      // This deletes the content of the flowfile as ff gets out of scope
    +      // It's the users responsibility to copy all the data
    +      ses->remove(ff);
    +
    +    }
    +   private:
    +    void (*callback_)(const flow_file_record*);
    --- End diff --
    
    ^ no std::function here either? 


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by arpadboda <gi...@git.apache.org>.
Github user arpadboda commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227360126
  
    --- Diff: libminifi/include/capi/Plan.h ---
    @@ -93,11 +95,50 @@ class ExecutionPlan {
         next_ff_ = ptr;
       }
     
    +  static std::shared_ptr<core::Processor> createProcessor(const std::string &processor_name, const std::string &name);
    +
      protected:
    +  class FailureHandler {
    +   public:
    +    FailureHandler() {
    +      callback_ = nullptr;
    +    }
    +    void setCallback(void (*onerror_callback)(const flow_file_record*)) {
    --- End diff --
    
    Done


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227411014
  
    --- Diff: libminifi/include/capi/api.h ---
    @@ -79,6 +79,13 @@ processor *add_processor_with_linkage(flow *flow, const char *processor_name);
     
     processor *add_python_processor(flow *, void (*ontrigger_callback)(processor_session *session));
     
    +/**
    +* Register your callback to received flow files that the flow failed to process
    +* The flow file is deleted after the callback is executed, make sure to copy all the data you need!
    +* The first callback should be registered before the flow is used. Can be changed later during runtime.
    +*/
    --- End diff --
    
    Sorry that was a bit terse -- I'm asking about the choice to remove the flow file by default. It seems that there is no choice to handle rolling back events ( to retry ) or keeping the flow file content for future operations without completely copying the data. That seems like it should be something that's optional  


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by arpadboda <gi...@git.apache.org>.
Github user arpadboda commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227360238
  
    --- Diff: libminifi/src/capi/Plan.cpp ---
    @@ -202,29 +180,31 @@ std::shared_ptr<core::ProcessSession> ExecutionPlan::getCurrentSession() {
       return current_session_;
     }
     
    -std::shared_ptr<minifi::Connection> ExecutionPlan::buildFinalConnection(std::shared_ptr<core::Processor> processor, bool setDest) {
    -  std::stringstream connection_name;
    -  std::shared_ptr<core::Processor> last = processor;
    -  connection_name << last->getUUIDStr() << "-to-" << processor->getUUIDStr();
    -  std::shared_ptr<minifi::Connection> connection = std::make_shared<minifi::Connection>(flow_repo_, content_repo_, connection_name.str());
    -  connection->setRelationship(termination_);
    +std::shared_ptr<minifi::Connection> ExecutionPlan::buildFinalConnection(std::shared_ptr<core::Processor> processor, bool set_dst) {
    +  return connectProcessors(processor, processor, termination_, set_dst);
    +}
     
    -  // link the connections so that we can test results at the end for this
    -  connection->setSource(last);
    -  if (setDest)
    -    connection->setDestination(processor);
    +void ExecutionPlan::finalize() {
    +  if (failure_handler_) {
    +    auto failure_proc = createProcessor("CallbackProcessor", "CallbackProcessor");
     
    -  utils::Identifier uuid_copy;
    -  last->getUUID(uuid_copy);
    -  connection->setSourceUUID(uuid_copy);
    -  if (setDest)
    -    connection->setDestinationUUID(uuid_copy);
    +    std::shared_ptr<processors::CallbackProcessor> callback_proc = std::static_pointer_cast<processors::CallbackProcessor>(failure_proc);
    +    callback_proc->setCallback(nullptr, std::bind(&FailureHandler::operator(), failure_handler_, std::placeholders::_1));
     
    -  processor->addConnection(connection);
    -  return connection;
    -}
    +    for (const auto& proc : processor_queue_) {
    +      relationships_.push_back(connectProcessors(proc, failure_proc, core::Relationship("failure", "failure collector"), true));
    --- End diff --
    
    Done


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226309091
  
    --- Diff: libminifi/src/capi/Plan.cpp ---
    @@ -202,29 +180,31 @@ std::shared_ptr<core::ProcessSession> ExecutionPlan::getCurrentSession() {
       return current_session_;
     }
     
    -std::shared_ptr<minifi::Connection> ExecutionPlan::buildFinalConnection(std::shared_ptr<core::Processor> processor, bool setDest) {
    -  std::stringstream connection_name;
    -  std::shared_ptr<core::Processor> last = processor;
    -  connection_name << last->getUUIDStr() << "-to-" << processor->getUUIDStr();
    -  std::shared_ptr<minifi::Connection> connection = std::make_shared<minifi::Connection>(flow_repo_, content_repo_, connection_name.str());
    -  connection->setRelationship(termination_);
    +std::shared_ptr<minifi::Connection> ExecutionPlan::buildFinalConnection(std::shared_ptr<core::Processor> processor, bool set_dst) {
    +  return connectProcessors(processor, processor, termination_, set_dst);
    +}
     
    -  // link the connections so that we can test results at the end for this
    -  connection->setSource(last);
    -  if (setDest)
    -    connection->setDestination(processor);
    +void ExecutionPlan::finalize() {
    +  if (failure_handler_) {
    +    auto failure_proc = createProcessor("CallbackProcessor", "CallbackProcessor");
     
    -  utils::Identifier uuid_copy;
    -  last->getUUID(uuid_copy);
    -  connection->setSourceUUID(uuid_copy);
    -  if (setDest)
    -    connection->setDestinationUUID(uuid_copy);
    +    std::shared_ptr<processors::CallbackProcessor> callback_proc = std::static_pointer_cast<processors::CallbackProcessor>(failure_proc);
    +    callback_proc->setCallback(nullptr, std::bind(&FailureHandler::operator(), failure_handler_, std::placeholders::_1));
     
    -  processor->addConnection(connection);
    -  return connection;
    -}
    +    for (const auto& proc : processor_queue_) {
    +      relationships_.push_back(connectProcessors(proc, failure_proc, core::Relationship("failure", "failure collector"), true));
    --- End diff --
    
    Are you checking processors to see if they have a failure relationship?


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by arpadboda <gi...@git.apache.org>.
Github user arpadboda commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227361035
  
    --- Diff: libminifi/include/capi/api.h ---
    @@ -79,6 +79,13 @@ processor *add_processor_with_linkage(flow *flow, const char *processor_name);
     
     processor *add_python_processor(flow *, void (*ontrigger_callback)(processor_session *session));
     
    +/**
    +* Register your callback to received flow files that the flow failed to process
    +* The flow file is deleted after the callback is executed, make sure to copy all the data you need!
    +* The first callback should be registered before the flow is used. Can be changed later during runtime.
    +*/
    --- End diff --
    
    Not sure I understand the Q properly. 
    The current implementation only requires to add the callback before finalising the flow (as there is no way to add the callback processor later). Later the callback can be replaced.


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226716905
  
    --- Diff: libminifi/include/capi/Plan.h ---
    @@ -93,11 +95,50 @@ class ExecutionPlan {
         next_ff_ = ptr;
       }
     
    +  static std::shared_ptr<core::Processor> createProcessor(const std::string &processor_name, const std::string &name);
    +
      protected:
    +  class FailureHandler {
    +   public:
    +    FailureHandler() {
    +      callback_ = nullptr;
    +    }
    +    void setCallback(void (*onerror_callback)(const flow_file_record*)) {
    +      callback_=onerror_callback;
    +    }
    +    void operator()(const processor_session* ps)
    +    {
    --- End diff --
    
    hmm, this doesn't give the option of what's called a rollback ( go back and potentially re-try). I'm curious if this could be done more simply with a function ptr and that gives them the option and provide them a set of ready-made ones that rollback and retry, drop, and continue, etc?


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/nifi-minifi-cpp/pull/421


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r227425691
  
    --- Diff: libminifi/include/capi/api.h ---
    @@ -79,6 +79,13 @@ processor *add_processor_with_linkage(flow *flow, const char *processor_name);
     
     processor *add_python_processor(flow *, void (*ontrigger_callback)(processor_session *session));
     
    +/**
    +* Register your callback to received flow files that the flow failed to process
    +* The flow file is deleted after the callback is executed, make sure to copy all the data you need!
    +* The first callback should be registered before the flow is used. Can be changed later during runtime.
    +*/
    --- End diff --
    
    ^ again, happy to move this discussion to that same ticket referenced above and merge this. Seems like any of these ideas can be done after more discussion on this topic. Thanks!


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226717192
  
    --- Diff: libminifi/include/capi/api.h ---
    @@ -79,6 +79,13 @@ processor *add_processor_with_linkage(flow *flow, const char *processor_name);
     
     processor *add_python_processor(flow *, void (*ontrigger_callback)(processor_session *session));
     
    +/**
    +* Register your callback to received flow files that the flow failed to process
    +* The flow file is deleted after the callback is executed, make sure to copy all the data you need!
    +* The first callback should be registered before the flow is used. Can be changed later during runtime.
    +*/
    --- End diff --
    
    what if failure is an optional condition? Why remove and not reuse? curious about comment above.


---

[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...

Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226302333
  
    --- Diff: libminifi/include/capi/Plan.h ---
    @@ -93,11 +95,50 @@ class ExecutionPlan {
         next_ff_ = ptr;
       }
     
    +  static std::shared_ptr<core::Processor> createProcessor(const std::string &processor_name, const std::string &name);
    +
      protected:
    +  class FailureHandler {
    +   public:
    +    FailureHandler() {
    +      callback_ = nullptr;
    +    }
    +    void setCallback(void (*onerror_callback)(const flow_file_record*)) {
    --- End diff --
    
    std::function for consistency?


---