[GitHub] nifi pull request #3159: NIFI-5794 Allow empty string demarcator in Consume/...

[pvillard31 <gi...@git.apache.org>]

GitHub user pvillard31 opened a pull request:

    https://github.com/apache/nifi/pull/3159

    NIFI-5794 Allow empty string demarcator in Consume/PublishKafka proce…

    …ssors
    
    Thank you for submitting a contribution to Apache NiFi.
    
    In order to streamline the review of the contribution we ask you
    to ensure the following steps have been taken:
    
    ### For all changes:
    - [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
         in the commit message?
    
    - [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    - [ ] Is your initial contribution a single, squashed commit?
    
    ### For code changes:
    - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
    - [ ] Have you written or updated unit tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
    - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
    - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
    
    ### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pvillard31/nifi NIFI-5794

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/3159.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3159
    
----
commit 88fbcffdc3fa6aadac14c23af9a5012edaa21e0f
Author: Pierre Villard <pi...@...>
Date:   2018-11-07T09:55:10Z

    NIFI-5794 Allow empty string demarcator in Consume/PublishKafka processors

----


---

[GitHub] nifi pull request #3159: NIFI-5794 Allow empty string demarcator in Consume/...

[asfgit <gi...@git.apache.org>]

Github user asfgit closed the pull request at:

    https://github.com/apache/nifi/pull/3159


---

[GitHub] nifi issue #3159: NIFI-5794 Allow empty string demarcator in Consume/Publish...

[pvillard31 <gi...@git.apache.org>]

Github user pvillard31 commented on the issue:

    https://github.com/apache/nifi/pull/3159
  
    In the end, I only changed the ConsumeKafka processors since it does not make sense in PublishKafka (one would just have to not set any demarcator to get the same result: all messages in the flow files are sent as one single message in Kafka). The main interest of this PR is to increase efficiency when we consume Avro data from Kafka: there is no need to set a demarcator and we do want to have multiples messages in the same flow file. I've used this [template](https://gist.github.com/pvillard31/57e627d206e7965cfb87cc27c09caf44) to test the change and to confirm the behavior: when pulling Avro data from Kafka with "empty string" as demarcator to have multiple messages in the same flow file, we are able to process the records in the next processor using a ConvertRecord for instance.


---

[GitHub] nifi issue #3159: NIFI-5794 Allow empty string demarcator in Consume/Publish...

[ijokarumawak <gi...@git.apache.org>]

Github user ijokarumawak commented on the issue:

    https://github.com/apache/nifi/pull/3159
  
    LGTM +1. I was able to confirm the expected benefit of using an empty demarcator string to combine multiple Avro formatted Kafka messages into a single FlowFile. Thanks @pvillard31!
    
    For those who want to use this improvement with older NiFi versions, using EL `${empty}` can mimic the non-empty validation and result the same effect . This assumes there's no variable named `empty`, the EL evaluation result will be an empty string while this is a valid configuration since it's not an empty property value.


---