[jira] [Commented] (AMBARI-20378) LogFeeder: Add de-duplication support

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903357#comment-15903357 ] 

Hadoop QA commented on AMBARI-20378:
------------------------------------

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12857030/AMBARI-20378.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 6 new or modified test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of javac compiler warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in ambari-logsearch/ambari-logsearch-logfeeder.

Test results: https://builds.apache.org/job/Ambari-trunk-test-patch/10945//testReport/
Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/10945//console

This message is automatically generated.

> LogFeeder: Add de-duplication support
> -------------------------------------
>
>                 Key: AMBARI-20378
>                 URL: https://issues.apache.org/jira/browse/AMBARI-20378
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-logsearch
>    Affects Versions: 2.5.0
>            Reporter: Olivér Szabó
>            Assignee: Olivér Szabó
>             Fix For: trunk
>
>         Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at input config block. For example (the values below are the defaults):
> {code:java}
> {
>   "input": [
>     {
>       ...
>       "cache_enabled" : "true",
>       "cache_size" : "100",
>       "cache_dedup_interval" : "1000",
>       "cache_last_dedup_enabled" : "false",
>       "cache_key_field" : "log_message"
>     }
>   ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is not reached between 2 log messages (same content), then the new log will be dropped. 
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the same as the last one, the new log will be dropped. (dedup interval wont have impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and stored in the cache (as keys)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)