You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Ari Rabkin (JIRA)" <ji...@apache.org> on 2009/08/04 20:52:14 UTC
[jira] Created: (CHUKWA-368) proposed stress test
proposed stress test
--------------------
Key: CHUKWA-368
URL: https://issues.apache.org/jira/browse/CHUKWA-368
Project: Hadoop Chukwa
Issue Type: New Feature
Components: build and test code
Reporter: Ari Rabkin
I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748204#action_12748204 ]
Ari Rabkin commented on CHUKWA-368:
-----------------------------------
Will commit tomorrow afternoon, I meant to say.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-368:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I think this is now the last round of fixes.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368-c.patch, CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin reopened CHUKWA-368:
-------------------------------
Can improve on previous code.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-368:
------------------------------
Attachment: CHUKWA-368-c.patch
- Cleaner separation of IO from state tracking.
- Track which split data was in.
- Make replay a bit more deterministic.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368-c.patch, CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on CHUKWA-368 started by Ari Rabkin.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-368:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Passes unit tests on my end, so I'm committing this. One non-trivial change. I'm moving src/test/org/apache/hadoop/chukwa/datacollection/TempFileUtil.java out of src/test and into the main src dir. The reason for this change is to make the temp file generation code available to contrib unit tests.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-368:
------------------------------
Status: Patch Available (was: In Progress)
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739091#action_12739091 ]
Ari Rabkin commented on CHUKWA-368:
-----------------------------------
My proposal is as follows: modify ConstRateSender to emit, instead of random bytes, a byte pattern that depends on the sequence number of the Chunk being produced. (This is a small and painless change).
Second, a MapReduce job that looks for that pattern and verifies that each chunk shows up at least once, that counts the number of duplicates, and that verifies that Chunks aren't corrupted.
Really, I'm worried about lost or duplicate chunks; the data pattern thing is just easy to do at the same time.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin reassigned CHUKWA-368:
---------------------------------
Assignee: Ari Rabkin
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-368) proposed stress test
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751766#action_12751766 ]
Hudson commented on CHUKWA-368:
-------------------------------
Integrated in Chukwa-trunk #128 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/128/])
. Some improvements to display of results
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368-c.patch, CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748203#action_12748203 ]
Ari Rabkin commented on CHUKWA-368:
-----------------------------------
I'm going to commit this, with local mods, barring objection.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-368) proposed stress test
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749158#action_12749158 ]
Hudson commented on CHUKWA-368:
-------------------------------
Integrated in Chukwa-trunk #121 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/121/])
. Delete old file
. New data integrity validation tool
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-368:
------------------------------
Attachment: CHUKWA-368.patch
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-368) proposed stress test
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-368:
------------------------------
Status: Patch Available (was: Reopened)
Tested at scale at Berkeley; will commit tomorrow afternoon barring objection. Code only in use at Berkeley so this shouldn't inconvenience anybody else.
> proposed stress test
> --------------------
>
> Key: CHUKWA-368
> URL: https://issues.apache.org/jira/browse/CHUKWA-368
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: build and test code
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-368-c.patch, CHUKWA-368.patch
>
>
> I'd like to build a stress-test for Chukwa, to verify that collectors don't lose or corrupt data under high load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.