You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Miklos Gergely <mg...@hortonworks.com> on 2017/02/02 13:00:33 UTC
Review Request 56224: Improve Log Feeder simulation to help scale
testing
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56224/
-----------------------------------------------------------
Review request for Ambari, Oliver Szabo and Robert Nettleton.
Bugs: AMBARI-19847
https://issues.apache.org/jira/browse/AMBARI-19847
Repository: ambari
Description
-------
Simulated Log Feeder input writes almost identical log messages, which is not really simulates real logs.
Instead of the logfeeder.simulate.log_message_size property there will be 3 new properties (the simulated Log Feeders would enforce a minimum, and a maximum value, and also set a default for them, if they are not specified):
logfeeder.simulate.number_of_words - the number of different words that may appear in the simulated logs (50 / 1000000, default 1000)
logfeeder.simulate.min_log_words - the minimum number of words in a simulated log message (1 / 10, default 5)
logfeeder.simulate.max_log_words - the maximum number of words in a log message (10 / 20, default 10)
After the parameters are set each logfeeder will start to load messages like this. In each message all the words are different:
Word000912 Word000903 Word000940 Word000760 Word000564 Word000762 Word000955 Word000553
Word000320 Word000218 Word000872 Word000204 Word000120 Word000298 Word000727 Word000720 Word000475
Word000413 Word000642 Word000872 Word000464 Word000746 Word000250
The rest of the properties remain the same:
logfeeder.simulate.input_number - number of paralell inputs (threads) loading the logs, if it is set and not 0 then all the rest of the configured inputs are ignored (running in simulation mode)!!
logfeeder.simulate.log_ids - comma separated list of the log ids to propagate at random, if not set by default all the available logs are propagated at random, example: storm_drpc,storm_logviewer,storm_nimbus
logfeeder.simulate.log_level - the level of the simulated log messages, by default WARN
logfeeder.simulate.sleep_milliseconds - the time interval at which each simulated inputs writes one log message at random
Diffs
-----
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java 743be69
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java 5bf600e
Diff: https://reviews.apache.org/r/56224/diff/
Testing
-------
Tested on local cluster
Thanks,
Miklos Gergely
Re: Review Request 56224: Improve Log Feeder simulation to help scale
testing
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56224/#review164012
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Nettleton
On Feb. 2, 2017, 1 p.m., Miklos Gergely wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56224/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2017, 1 p.m.)
>
>
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
>
>
> Bugs: AMBARI-19847
> https://issues.apache.org/jira/browse/AMBARI-19847
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Simulated Log Feeder input writes almost identical log messages, which is not really simulates real logs.
>
> Instead of the logfeeder.simulate.log_message_size property there will be 3 new properties (the simulated Log Feeders would enforce a minimum, and a maximum value, and also set a default for them, if they are not specified):
>
> logfeeder.simulate.number_of_words - the number of different words that may appear in the simulated logs (50 / 1000000, default 1000)
> logfeeder.simulate.min_log_words - the minimum number of words in a simulated log message (1 / 10, default 5)
> logfeeder.simulate.max_log_words - the maximum number of words in a log message (10 / 20, default 10)
>
> After the parameters are set each logfeeder will start to load messages like this. In each message all the words are different:
>
> Word000912 Word000903 Word000940 Word000760 Word000564 Word000762 Word000955 Word000553
> Word000320 Word000218 Word000872 Word000204 Word000120 Word000298 Word000727 Word000720 Word000475
> Word000413 Word000642 Word000872 Word000464 Word000746 Word000250
>
> The rest of the properties remain the same:
>
> logfeeder.simulate.input_number - number of paralell inputs (threads) loading the logs, if it is set and not 0 then all the rest of the configured inputs are ignored (running in simulation mode)!!
> logfeeder.simulate.log_ids - comma separated list of the log ids to propagate at random, if not set by default all the available logs are propagated at random, example: storm_drpc,storm_logviewer,storm_nimbus
> logfeeder.simulate.log_level - the level of the simulated log messages, by default WARN
> logfeeder.simulate.sleep_milliseconds - the time interval at which each simulated inputs writes one log message at random
>
>
> Diffs
> -----
>
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java 743be69
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java 5bf600e
>
> Diff: https://reviews.apache.org/r/56224/diff/
>
>
> Testing
> -------
>
> Tested on local cluster
>
>
> Thanks,
>
> Miklos Gergely
>
>
Re: Review Request 56224: Improve Log Feeder simulation to help scale
testing
Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56224/#review164000
-----------------------------------------------------------
Ship it!
Ship It!
- Oliver Szabo
On Feb. 2, 2017, 1 p.m., Miklos Gergely wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56224/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2017, 1 p.m.)
>
>
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
>
>
> Bugs: AMBARI-19847
> https://issues.apache.org/jira/browse/AMBARI-19847
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Simulated Log Feeder input writes almost identical log messages, which is not really simulates real logs.
>
> Instead of the logfeeder.simulate.log_message_size property there will be 3 new properties (the simulated Log Feeders would enforce a minimum, and a maximum value, and also set a default for them, if they are not specified):
>
> logfeeder.simulate.number_of_words - the number of different words that may appear in the simulated logs (50 / 1000000, default 1000)
> logfeeder.simulate.min_log_words - the minimum number of words in a simulated log message (1 / 10, default 5)
> logfeeder.simulate.max_log_words - the maximum number of words in a log message (10 / 20, default 10)
>
> After the parameters are set each logfeeder will start to load messages like this. In each message all the words are different:
>
> Word000912 Word000903 Word000940 Word000760 Word000564 Word000762 Word000955 Word000553
> Word000320 Word000218 Word000872 Word000204 Word000120 Word000298 Word000727 Word000720 Word000475
> Word000413 Word000642 Word000872 Word000464 Word000746 Word000250
>
> The rest of the properties remain the same:
>
> logfeeder.simulate.input_number - number of paralell inputs (threads) loading the logs, if it is set and not 0 then all the rest of the configured inputs are ignored (running in simulation mode)!!
> logfeeder.simulate.log_ids - comma separated list of the log ids to propagate at random, if not set by default all the available logs are propagated at random, example: storm_drpc,storm_logviewer,storm_nimbus
> logfeeder.simulate.log_level - the level of the simulated log messages, by default WARN
> logfeeder.simulate.sleep_milliseconds - the time interval at which each simulated inputs writes one log message at random
>
>
> Diffs
> -----
>
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java 743be69
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java 5bf600e
>
> Diff: https://reviews.apache.org/r/56224/diff/
>
>
> Testing
> -------
>
> Tested on local cluster
>
>
> Thanks,
>
> Miklos Gergely
>
>