[jira] [Commented] (METRON-326) Error Handling in ElasticsearchWriter

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535827#comment-15535827 ] 

ASF GitHub Bot commented on METRON-326:
---------------------------------------

GitHub user justinleet opened a pull request:

    https://github.com/apache/incubator-metron/pull/286

    METRON-326 Error Handling in ElasticsearchWriter

    To facilitate partial successes writing in Elasticsearch (i.e. subset of Tuples succeeds and another subset fails), the BulkMessageWriter.write() now returns a BulkWriterResponse object which contains both the successful and failed Tuples (So that we know which ones to ack successfully and which to send to the error stream).
    
    ElasticsearchWriter has been modified to be able to take advantage of partial writes.  The other implementations of the BulkWriter have only been affected to produce total success at the end of their methods (which is the same functionality previously implied by not hitting an Exception.
    
    Unit tests have been added for the ElasticsearchWriter writers implementation of the partial successes and failures, along with modifying existing tests elsewhere to account for the BulkWriterResponse.
    
    One question I'd like resolved in review, is where to actually place the BulkWriterResponse in the code hierarchy.  It's obviously closely tied to the BulkMessageWriter (really only existing to add functionality to it), but that lives in an interface package and BWR is a concrete class. If there are opinions on where that should live, I'm happy to move it and update the PR.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/incubator-metron es_error_handling

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #286
    
----
commit 8d7475d0f8857f40ab2a025a0d20b429ef084053
Author: justinjleet <ju...@gmail.com>
Date:   2016-09-27T15:02:14Z

    Adding functionality for partial write success in Elasticsearch

commit 83161afd1b1407c47a5565340bdfa96173955e7f
Author: justinjleet <ju...@gmail.com>
Date:   2016-09-30T11:40:21Z

    fix

----


> Error Handling in ElasticsearchWriter
> -------------------------------------
>
>                 Key: METRON-326
>                 URL: https://issues.apache.org/jira/browse/METRON-326
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Ajay Yadav
>            Assignee: Justin Leet
>
> In Elasticsearch writer we raise a exception if BulkResponse object has failures and that results in failing the whole batch even if some objects failed in it. This has spiral effect specially when there is continuous stream of bad messages and errorStream is tied to indexingBolt. 
> If possible we should iterate through items in BulkResponse object and send only failed messages to errorStream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)