[jira] [Commented] (NIFI-2259) HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL as expected

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378930#comment-15378930 ] 

ASF GitHub Bot commented on NIFI-2259:
--------------------------------------

GitHub user ijokarumawak opened a pull request:

    https://github.com/apache/nifi/pull/656

    Issue HTTP Site-to-Site with lots of queued flow-file

    This PR is not an actual fix for NIFI-2259. I just want to share it with @markap14.
    
    @markap14 I've been trying to reproduce the issue about PipedInputStream.awaitSpace stops working when there're lots of small flow-files to be sent, like 10,000 files, 1kb each. But couldn't reproduce it so far.
    
    So, I would like to investigate the issue differently. While I was trying to reproduce it, I found another issue related to the amount of queued flow-files. That is NIFI-2259.
    
    With this fix, I was able to queue up 60,000 1kb flow files at Site-to-Site client, then successfully transferred it to remote input-port using HTTP Site-to-Site. (I think it can hold more than that, but stopped increasing at that point)
    
    I also tested by adding RateControl after remote input-port, simulate the slow data consumption. It worked, too, without stopping or losing data.
    
    Would you pull this branch and test it again? I am not thinking this will fix your problem, but certainly make HTTP Site-to-Site works better in a case with lots of flow-files.
    
    Also, if you can reproduce the issue:
    - Please modify org.apache.nifi.remote log level to DEBUG, and share nifi-app.log
    - Please share whole thread dump. I'd like to see the counter part thread which is draining data from PipedInputStream.
    - Also, flow template would be appreciated, too
    
    Thanks!

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ijokarumawak/nifi pr-530-with-NIFI-2259

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/656.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #656
    
----
commit 8a2f03e46ef86f213507980f6f24f76588bbcc5e
Author: Mark Payne <ma...@hotmail.com>
Date:   2016-06-14T18:47:24Z

    NIFI-1992: Updated site-to-site client and server to support clustered nifi instances

commit 715906a12608c17c770759769b834151add30af5
Author: Koji Kawamura <ij...@gmail.com>
Date:   2016-07-15T06:33:36Z

    NIFI-2259: HTTP Site-to-Site can't handle DEST_FULL
    
    HTTP Site-to-Site can't handle TRANSACTION_FINISHED_BUT_DESTINATION_FULL
    scenario as expected.
    
    That happens if the remote NiFi's input port destination relationship
    becomes full during Site-to-Site client sends data. The data which has
    already sent to the remote NiFi has to be committed successfully,
    however, the remote NiFi returns 503 as a response of commit HTTP
    request. Because it does check port availability.
    
    The port availability check shouldn't be done with commit request, since
    the session at source NiFi has already been committed. The remote NiFi
    should commit its session as well, and return
    TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.
    
    This fix make remote NiFi keep the handshaken properties when it holds
    transaction to be committed. Then if a transaction already has
    handshaken properties, then use it, instead of doing handshake process
    again.

----


> HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL as expected
> ---------------------------------------------------------------------------------------
>
>                 Key: NIFI-2259
>                 URL: https://issues.apache.org/jira/browse/NIFI-2259
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.0.0
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
>             Fix For: 1.0.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> If a remote NiFi instance input-port's destination becomes full, It's expected that remote NiFi instance returns TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.
> However, an IOException occurred due to DESTINATION_FULL handshake error when org.apache.nifi.remote.Transaction.complete() is called.
> The cause of this issue is SiteToSiteResource.commitInputPortTransaction() checks if all of input-port's destination is full. Which shouldn't be executed at complete(). The transaction is already committed on source NiFi.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)