[GitHub] metron issue #958: METRON-1483: Create a tool to monitor performance of the ...

[cestella <gi...@git.apache.org>]

Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/958
  
    # Testing Plan:
    We presume
    * `ZOOKEEPER` is an environment variable set to the zk quorum (e.g. `node1:2181`)
    * `BROKER` is an environment variable set to the broker (e.g. `node1:6667`)
    
    ## Unbiased Generation
    
    In order to test this, we will:
    1. create a new topic
    ```
    # Create dummy_12_unbiased with 12 partitions
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic dummy_12_unbiased --partitions 12 --replication-factor 1
    ```
    2. Create some dummy data to send into kafka
    ```
    # Generate some templates of dummy data that look like CSV in ~/dummy.templates
    # With a number, a string, a GUID and the timestamp
    for i in $(seq 1 100);do echo "$i,foo,\$METRON_GUID,\$METRON_TS";done > ~/dummy.templates
    ```
    3. Write into that topic from a set of templates at a specified rate
    ```
    # Write out 1000 messages per second to dummy_12_unbiased
    # Monitor dummy_12_unbiased
    # Each message is drawn from the set of templates in step 2.
    # Stop after 60 seconds and report how much you've written
    $METRON_HOME/bin/load_tool.sh -p 5 -ot dummy_12_unbiased -mt dummy_12_unbiased -t ~/dummy.template -eps 1000 -z $ZOOKEEPER -tl 60000
    ```
      * It should indicate that it generated something like 60k messages
    4. Measure if the messages generated roughly match the requested rate.
    ```
    # You will have to ctrl-c this in a few seconds
    /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server node1:6667 --topic dummy_12_topic --from-beginning | wc -l
    ```
      * It should output something like `Processed a total of 60400 messages`, which would be about 1000 messages per second.



---