[jira] [Commented] (METRON-1571) Correct KAFKA_TAIL Seek to End Logic

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484726#comment-16484726 ] 

ASF GitHub Bot commented on METRON-1571:
----------------------------------------

GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/1023

    METRON-1571 Correct KAFKA_TAIL Seek to End Logic

    
    ## Changes
    
    * KAFKA_TAIL now performs manual partition assignment and correctly uses the Kafka API to seek to the end of the topic.
     
        * Previously, the function messed with the 'group.id' to effectively seek to end.  This is an abuse of the API.  Also, a fixed group.id would be required for using the KAFKA_* functions in a Kerberized cluster.
     
    * KAKFA_GET and KAFKA_TAIL now use a more accurate mechanism for adhering to the user's requested max wait time.
    
        * Previously, the max wait was divided by the max poll timeout to estimate how many times Kafka should be polled before returning to the user.  This method is not always accurate when some poll requests return values and others do not.
    
    * The MPack now defines a global property `bootstrap.servers` which allow the KAFKA_* functions to work out-of-the-box after a Metron installation.
    
        * Previously, a user had to manually define this property before using the KAKFA_* functions.  If the property is not defined correctly, they would have to wait an excessive amount of time for the connection to timeout since there is no means in the REPL to interrupt a function call.
    
    ## Pull Request Checklist
    
    - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1571

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1023.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1023
    
----
commit 10fe2bfd04db6633b73e9d0f37d458f4ce83aaf9
Author: Nick Allen <ni...@...>
Date:   2018-04-20T20:18:16Z

    METRON-1571

----


> Correct KAFKA_TAIL Seek to End Logic
> ------------------------------------
>
>                 Key: METRON-1571
>                 URL: https://issues.apache.org/jira/browse/METRON-1571
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Nick Allen
>            Priority: Major
>
> KAFKA_TAIL always needs to tail from the end of a topic.  The current implementation does not correctly seek to the end of the topic as the Kafka API supports.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)