[GitHub] incubator-fluo issue #795: May need to add post-commit call back to loader

[keith-turner <gi...@git.apache.org>]

GitHub user keith-turner opened an issue:

    https://github.com/apache/incubator-fluo/issues/795

    May need to add post-commit call back to loader

    I strongly suspect that the Loader interface will need a post commit callback in order to support apache/incubator-fluo-recipes#123 and apache/incubator-fluo-recipes#124.  If something like Kafka or Nifi is initiating load transactions, then those system will need to know when those load transactions are successful.
    
    This could be accomplished currently by executing individual transactions.  However using the LoaderExecutor is much more efficient at committing batches of transactions.

----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-fluo issue #795: May need to add post-commit call back to loader

[keith-turner <gi...@git.apache.org>]

Github user keith-turner commented on the issue:

    https://github.com/apache/incubator-fluo/issues/795
  
    Another possible solution to this problem, would be to add an async commit.  This would allow transactions to get a callback when a commit completes.  This offer similar functionality as adding a post commit callback to the loader interface.  This would avoid having a thread wait for commit and allow many transactions to be working their way through the commit process.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-fluo issue #795: May need to add post-commit call back to loader

[keith-turner <gi...@git.apache.org>]

Github user keith-turner commented on the issue:

    https://github.com/apache/incubator-fluo/issues/795
  
    I described a solution without describing the problem very well.  The problem this issue is trying to solve is getting good throughput from a source of continuous information that needs feedback on when that information was successfully processed by Fluo.  The LoaderExecutor gives good throughput for committing transactions, but lacks the feedback mechanism.
    
    I was thinking about how to solve  this w/ the current 1.0.0 release.  One way would be to process batches with the loader executor.  For example process 10,000 transactions with the LoaderExecutor and then shut the LoaderExecutor down.  After its shut down, then feedback can be given to the information source that all 10,000 transactions completed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---