[jira] [Commented] (NIFI-3730) Allow exclusion of begin/commit events in CaptureChangeMySQL

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981152#comment-15981152 ] 

ASF GitHub Bot commented on NIFI-3730:
--------------------------------------

GitHub user mattyb149 opened a pull request:

    https://github.com/apache/nifi/pull/1689

    NIFI-3730: Allow exclusion of BEGIN/COMMIT event output in CaptureChangeMySQL

    Thank you for submitting a contribution to Apache NiFi.
    
    In order to streamline the review of the contribution we ask you
    to ensure the following steps have been taken:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? Is it referenced 
         in the commit message?
    
    - [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    - [x] Is your initial contribution a single, squashed commit?
    
    ### For code changes:
    - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
    - [x] Have you written or updated unit tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
    - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
    - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered?
    
    ### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mattyb149/nifi NIFI-3730

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/1689.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1689
    
----
commit a9ee4015166ac64c80e963ab7dd7139abdf20cbd
Author: Matt Burgess <ma...@apache.org>
Date:   2017-04-24T13:23:02Z

    NIFI-3730: Allow exclusion of BEGIN/COMMIT event output in CaptureChangeMySQL

----


> Allow exclusion of begin/commit events in CaptureChangeMySQL
> ------------------------------------------------------------
>
>                 Key: NIFI-3730
>                 URL: https://issues.apache.org/jira/browse/NIFI-3730
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.2.0
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
>             Fix For: 1.2.0
>
>
> Notes from [~ijokarumawak]:
> In an end-to-end CDC flow, we would use CaptureChangeMySQL to get database events to stream into PutSQL or PutDatabaseRecord. Those database events include SQL statements such as begin and commit, and such statements are executed literally here.
> I think it would be great if users can configure CaptureChangeMySQL whether to emit begin and commit event or not, as these statements do not have significant meaning for synchronizing change since CaptureChangeMySQL emits FlowFile per updated record, but just for EnforceOrder to order events correctly.
> If we can eliminate these events, we can minimize the number of FlowFiles which would lead us to a better performance.
> Also, (for MySQL at least) begin and commit is not a database or table specific event. When I replicate changes from table A to table B using CaptureChangeMySQL and PutDatabaseRecord, I saw following behavior and felt begin and commit are a little bit disturbing:
> Insert a row into table A
> 3 events are emitted, begin, insert A and commit. Then CaptureChangeMySQL emitted 3 FlowFiles.
> PutDatabaseRecord execute 3 SQLs. begin, insert B and commit.
> Another 3 events are emitted derived from insert B via MySQL bin-log, begin, insert B and commit.
> Since I have configured CaptureChangeMySQL only listens to table A, it only emitted 2 FlowFiles, begin and commit.
> PutDatabaseRecord executes the begin and commit SQL.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)