[jira] [Commented] (METRON-557) Create Stellar Functions for Kafka

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655126#comment-15655126 ] 

ASF GitHub Bot commented on METRON-557:
---------------------------------------

GitHub user nickwallen opened a pull request:

    https://github.com/apache/incubator-metron/pull/354

    METRON-557 Create Stellar Functions for Kafka

    ## [METRON-557](https://issues.apache.org/jira/browse/METRON-557)
    
    Add functions to read and write messages to Kafka topics.  This should enable a user to debug a Grok expression or Enrichment configuration inside the REPL. 
    
    For example, I can pull a message off of the error topic to see why it failed to parse. Then fix the configuration and resubmit the message to try again. I also won’t have to go outside of the Stellar REPL to monitor topic activity.
    
    ### `KAFKA_GET`
     
    Retrieves messages from a Kafka topic.  Subsequent calls will continue retrieving messages sequentially from the original offset.
     
    Example: Retrieve one message from a topic.
    ```
    KAFKA_GET('topic')
    ```
    Example: Retrieve 10 messages from a topic.
    ```
     KAFKA_GET('topic', 10)
    ```
    Example: Retrieve the first message in a topic.  This must be the first retrieval from the topic, otherwise the messages will be retrieved starting from the previously stored consumer offset.
    ```
    KAFKA_GET('topic', 1, { "auto.offset.reset": "earliest" })
    ```
    
    ### `KAFKA_TAIL`
    
    Retrieves messages from a Kafka topic always starting with the most recent message first.
    
    Example: Retrieve the latest message from a topic.
     ```
    KAFKA_TAIL('topic')
    ```
    Example: Retrieve 10 messages from a topic starting with the latest.
     ```
    KAFKA_TAIL('topic', 10)
    ```
    
    ### `KAFKA_PUT`
    
    Sends messages to a Kafka topic.
    
    Example: Put two messages on the topic 'topic'.
    ```
     KAFKA_PUT('topic', ["message1", "message2"])
    ```
    Example: Put a message on a topic and also define an alternative Kafka broker.
    ```
    KAFKA_PUT('topic', ["message1"], { "bootstrap.servers": "kafka-broker-1:6667" })
    ```
    
    ### `KAFKA_PROPS`
    
    Retrieves the Kafka properties that are used by other KAFKA_* functions like KAFKA_GET and KAFKA_PUT.  The Kafka properties are compiled from a set of default properties, the global properties, and any overrides.
    
    Example: Retrieve the current Kafka properties.
    ```
    KAFKA_PROPS()
    ```
    Example: Retrieve the current Kafka properties taking into account a set of overrides.
    ```
    KAFKA_PROPS({ "max.poll.records": 1 })
    ```
    
    ## Testing
    
    These were tested on a local Stellar Shell in addition to running within the "Quick Dev" environment.  Note that in the "Quick Dev" environment, you must manually copy the jar built within the Management project (`metron-platform/metron-management/target/...`) to `/usr/metron/0.2.1BETA/lib`.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/incubator-metron METRON-557

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/354.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #354
    
----
commit 4c2d5daaa603405a0ba8b21b75cd55d8e59efd02
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-11-10T16:32:14Z

    METRON-557 Create Stellar Functions for Kafka

----


> Create Stellar Functions for Kafka
> ----------------------------------
>
>                 Key: METRON-557
>                 URL: https://issues.apache.org/jira/browse/METRON-557
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Nick Allen
>            Assignee: Nick Allen
>
> Add functions to read and write messages to Kafka topics.
> This should allow a user the to debug a Grok expression or Enrichment configuration inside the REPL.  For example, I can pull a message off of the error topic to see why it failed to parse.  Then fix the configuration and resubmit the message to try again.  I also won’t have to go outside of the Stellar REPL to monitor topic activity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)