[jira] [Commented] (METRON-1051) Enable the ability to update indexed messages

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110902#comment-16110902 ] 

ASF GitHub Bot commented on METRON-1051:
----------------------------------------

Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/666
  
    Testing Instructions beyond the normal smoke test (i.e. letting data
    flow through to the indices and checking them).
    
    # Preliminaries
    
    Set an environment variable to indicate `METRON_HOME`:
    * `export METRON_HOME=/usr/metron/0.4.0` 
    
    # Deploy the dummy parser
    * Edit `$METRON_HOME/config/zookeeper/parsers/dummy.json`:
    ```
    {
      "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
      "sensorTopic":"dummy"
    }
    ```
    * Create the dummy kafka topic:
      `/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic dummy --partitions 1 --replication-factor 1`
    * Persist config changes: `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z node1:2181`
    * Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s dummy`
    
    # Send dummy data through
    * Edit `~/msg.json` with the following content:
    ```
    { "guid" : "guid0", "sensor.type" : "dummy", "timestamp" : 100 }
    ```
    * Send `msg.json` through to kafka via `cat ~/msg.json | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic dummy`
    
    ## Test Case: Update via patch
    * Patch the message in ES and create a new field 'project' by executing
      the following:
    ```
    curl -u user:password -X PATCH --header 'Content-Type: application/json' --header 'Accept: */*' -d '{
      "uuid" : "guid0",
        "sensorType" : "dummy",
        "patch" : [
        {
          "op": "add"
            , "path": "/project"
            , "value": "metron"
        }
      ]
    }' 'http://node1:8082/api/v1/update/patch'
    ```
    * Validate that the message has a field 'project':
    ```
    curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
    {
      "_source" : [ "project" ]
    }
    '
    ```
    
    ## Test Case: Update via replace 
    * Replace the message in ES and create a couple of modifications:
      * new field `new_field` == "brand new"
      * modified `timestamp` == 7
    Execute the following:
    ```
    curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: */*' -d '{
         "uuid" : "guid0",
         "sensorType" : "dummy",
         "replacement" : {
           "source:type": "dummy",
           "guid" : "guid0",
           "new_field" : "brand new",
           "timestamp" : 7
          }
       }' 'http://node1:8082/api/v1/update/replace'
    ```
    * Validate that the message has a field 'project':
    ```
    curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
    {
      "_source" : [ "new_field", "timestamp" ]
    }
    '
    ```
    ## Test Case: Validate that the changes persist to HBase
    * Execute `echo "scan 'metron_update'" | hbase shell` 
    * Ensure that 
      * There is one row: `guid0`
      * There are 2 columns 
      * The values associated with the columns correspond to the two modifications done in the tests above



> Enable the ability to update indexed messages
> ---------------------------------------------
>
>                 Key: METRON-1051
>                 URL: https://issues.apache.org/jira/browse/METRON-1051
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Casey Stella
>
> Currently our indexing layer is immutable.  It would be useful to be able to:
> * Update data in the indices
> * Track lineage for updates
> Current version of data should be able to be exposed via a batch interface. 
> Updates should be available via a Java DAO API as well as via the REST API.
> Implementation should ensure that the ability to read the current data in batch is possible. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)