You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Will McQueen (JIRA)" <ji...@apache.org> on 2012/08/16 05:47:37 UTC
[jira] [Created] (FLUME-1490) Option to limit number of events sent
in Stress source
Will McQueen created FLUME-1490:
-----------------------------------
Summary: Option to limit number of events sent in Stress source
Key: FLUME-1490
URL: https://issues.apache.org/jira/browse/FLUME-1490
Project: Flume
Issue Type: Improvement
Components: Sinks+Sources
Affects Versions: v1.2.0
Reporter: Will McQueen
Fix For: v1.3.0
Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1490) Option to limit number of events sent
in Stress source
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated FLUME-1490:
-----------------------------------
Attachment: FLUME-1490.v1.patch.txt
This patch implements a count limit, both for overall (successful + failed) events and for successful events. Seems like both could be useful in testing. Includes unit test and a bit of code clean-up.
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1490.v1.patch.txt
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-1490) Option to limit number of events
sent in Stress source
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell reassigned FLUME-1490:
--------------------------------------
Assignee: Patrick Wendell
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1490) Option to limit number of events
sent in Stress source
Posted by "Will McQueen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Will McQueen resolved FLUME-1490.
---------------------------------
Resolution: Fixed
Patch committed (5da69a2ca8560b6dec660e927d0d7e0b062207ed)
Thanks, Patrick!
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1490.v1.patch.txt
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1490) Option to limit number of events
sent in Stress source
Posted by "Will McQueen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436010#comment-13436010 ]
Will McQueen commented on FLUME-1490:
-------------------------------------
Hi Patrick,
As an example with the 'count' feature, a test can limit the stress test to 1M events so that later it can confirm whether exactly 1M events were received at the sink... and if not then whether there were more (duplicates) or less (sink is still catching up).
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Fix For: v1.3.0
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (FLUME-1490) Option to limit number of events sent
in Stress source
Posted by "Will McQueen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Will McQueen closed FLUME-1490.
-------------------------------
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1490.v1.patch.txt
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1490) Option to limit number of events
sent in Stress source
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437198#comment-13437198 ]
Hudson commented on FLUME-1490:
-------------------------------
Integrated in flume-trunk #283 (See [https://builds.apache.org/job/flume-trunk/283/])
FLUME-1490. Option to limit number of events sent in Stress source. (Revision 5da69a2ca8560b6dec660e927d0d7e0b062207ed)
Result = SUCCESS
will : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=5da69a2ca8560b6dec660e927d0d7e0b062207ed
Files :
* flume-ng-core/pom.xml
* flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java
* flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
* flume-ng-core/src/test/java/org/apache/flume/source/TestStressSource.java
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1490.v1.patch.txt, FLUME-1490.v2.patch.txt
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1490) Option to limit number of events
sent in Stress source
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435749#comment-13435749 ]
Patrick Wendell commented on FLUME-1490:
----------------------------------------
I had started a discussion about a load-generating source earlier that was configurable. My thought was that rather than a fixed event count, you really wanted something that generates events at a fixed _rate_. For instance, to mimic a given rate you'd expect from your data source.
Will - what is the use case for having a fixed count? Just to limit the length of an experiment?
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Fix For: v1.3.0
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1490) Option to limit number of events sent
in Stress source
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated FLUME-1490:
-----------------------------------
Attachment: FLUME-1490.v2.patch.txt
Great! For posterity I'm including the patch that was actually committed.
> Option to limit number of events sent in Stress source
> ------------------------------------------------------
>
> Key: FLUME-1490
> URL: https://issues.apache.org/jira/browse/FLUME-1490
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Will McQueen
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1490.v1.patch.txt, FLUME-1490.v2.patch.txt
>
>
> Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira