You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "John Fung (JIRA)" <ji...@apache.org> on 2012/06/22 23:54:42 UTC
[jira] [Created] (KAFKA-373) Broker Failure Test needs to be fixed
to work with Mirror Maker in TRUNK
John Fung created KAFKA-373:
-------------------------------
Summary: Broker Failure Test needs to be fixed to work with Mirror Maker in TRUNK
Key: KAFKA-373
URL: https://issues.apache.org/jira/browse/KAFKA-373
Project: Kafka
Issue Type: Task
Reporter: John Fung
Assignee: John Fung
Embedded consumer is no longer supported and Mirror Maker is introduced to TRUNK. Broker Failure test needs to be fixed to use Mirror Maker for mirroring.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-373) Broker Failure Test needs to be fixed
to work with Mirror Maker in TRUNK
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-373:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk.
> Broker Failure Test needs to be fixed to work with Mirror Maker in TRUNK
> ------------------------------------------------------------------------
>
> Key: KAFKA-373
> URL: https://issues.apache.org/jira/browse/KAFKA-373
> Project: Kafka
> Issue Type: Task
> Reporter: John Fung
> Assignee: John Fung
> Attachments: kafka-373-v1.patch, kafka-373-v2.patch
>
>
> Embedded consumer is no longer supported and Mirror Maker is introduced to TRUNK. Broker Failure test needs to be fixed to use Mirror Maker for mirroring.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-373) Broker Failure Test needs to be
fixed to work with Mirror Maker in TRUNK
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399781#comment-13399781 ]
Joel Koshy commented on KAFKA-373:
----------------------------------
John, thanks for the patch. Some of these comments would also apply to the
test in 0.8:
- The header of the script is mostly in the README which is more
appropriate. i.e., we should just have a README.
- Non-blocker: I think it would be good to switch to getopt for the test
parameters - I think we should do this in our system tests going forward... and some
of the variable parameters in the script (i.e., the ones you mark as
"change as needed") can be made command-line parameters.
- The script is getting to be big, so a number of functions can/should be
collapsed, and use arguments to handle the variants:
- wait_for_zero.. functions
- start_source/target cluster/server functions
- start_console_consumer functions
- Shouldn't shutdown_producer do a kill_child_processes? in which case why
have the file to notify? FWIW, there's some bug in the script's shutdown -
don't remember what I did, but I had run the system test in 0.8 and my
hard-disk was full in the morning.
- Not sure why you need to check the num_iterations ge iter inside the
start_test loop - also you can move the increment outside those blocks -
or switch to a for loop over a range.
- I don't think you need to sleep after starting the mirror makers.
- Can you elaborate on the hanging console consumer/mirror maker issue?
> Broker Failure Test needs to be fixed to work with Mirror Maker in TRUNK
> ------------------------------------------------------------------------
>
> Key: KAFKA-373
> URL: https://issues.apache.org/jira/browse/KAFKA-373
> Project: Kafka
> Issue Type: Task
> Reporter: John Fung
> Assignee: John Fung
> Attachments: kafka-373-v1.patch
>
>
> Embedded consumer is no longer supported and Mirror Maker is introduced to TRUNK. Broker Failure test needs to be fixed to use Mirror Maker for mirroring.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-373) Broker Failure Test needs to be fixed
to work with Mirror Maker in TRUNK
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Fung updated KAFKA-373:
----------------------------
Attachment: kafka-373-v2.patch
Hi Joel,
Thanks for reviewing this patch.
Uploaded kafka-373-v2.patch and here are the changes:
1. In "initialize" function, added code to find the location of the zk & kafka log4j log files.
2. In "cleanup" function, added code to remove the zk & kafka log4j log files
3. The header of the script is now removed and the description are in README
4. Use getopt to process command line arguments
5. Consolidated the following functions:
* start_console_consumer_for_source_producer
* start_console_consumer_for_mirror_producer
* wait_for_zero_source_console_consumer_lags
* wait_for_zero_mirror_console_consumer_lags
6. The file to notify producer to stop:
The producer is sent to the background to run in a while-loop. If a file is used to notify the producer process in the background, the producer will exit properly inside the while loop.
7. The following check is required for each of the source, target and mirror maker. It is because the following 2 lines are needed for:
* Line 1: find out if the $bounce_source_id is a char in the string $svr_to_bounce
* Line 2: check to see if $num_iterations is already reached and if $svr_idx > 0 (meaning this server needs to be bounced)
* Line 1: svr_idx=`expr index $svr_to_bounce $bounce_source_id`
* Line 2: if [[ $num_iterations -ge $iter && $svr_idx -gt 0 ]]; then
> Broker Failure Test needs to be fixed to work with Mirror Maker in TRUNK
> ------------------------------------------------------------------------
>
> Key: KAFKA-373
> URL: https://issues.apache.org/jira/browse/KAFKA-373
> Project: Kafka
> Issue Type: Task
> Reporter: John Fung
> Assignee: John Fung
> Attachments: kafka-373-v1.patch, kafka-373-v2.patch
>
>
> Embedded consumer is no longer supported and Mirror Maker is introduced to TRUNK. Broker Failure test needs to be fixed to use Mirror Maker for mirroring.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-373) Broker Failure Test needs to be fixed
to work with Mirror Maker in TRUNK
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Fung updated KAFKA-373:
----------------------------
Attachment: kafka-373-v1.patch
Uploaded kafka-373-v1.patch. The changes are:
1. Modified mirror_producer*.properties files to use zk.connect as "broker.list" is no longer supported
2. The main test script run-test.sh is modified to invoke Mirror Maker for mirroring.
3. The bouncing option now supports source broker, target broker and mirror maker.
4. Some methods which can be shared between tests such as "info", "kill_child_processes" are moved to a util.sh script located under "system_test/common"
5. Updated README
> Broker Failure Test needs to be fixed to work with Mirror Maker in TRUNK
> ------------------------------------------------------------------------
>
> Key: KAFKA-373
> URL: https://issues.apache.org/jira/browse/KAFKA-373
> Project: Kafka
> Issue Type: Task
> Reporter: John Fung
> Assignee: John Fung
> Attachments: kafka-373-v1.patch
>
>
> Embedded consumer is no longer supported and Mirror Maker is introduced to TRUNK. Broker Failure test needs to be fixed to use Mirror Maker for mirroring.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-373) Broker Failure Test needs to be
fixed to work with Mirror Maker in TRUNK
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406073#comment-13406073 ]
Joel Koshy commented on KAFKA-373:
----------------------------------
+1
The script can be made more concise (basically, a lot more functions can and
should be moved to utils).
However, I just realized that since development is primarily on 0.8 and
because there are a couple of differences in this script between 0.8 and
trunk, there does not seem to be much point in putting in much effort for
this in trunk.
So if there are no objections, I think we can check this in and work on such
improvements in 0.8 only.
> Broker Failure Test needs to be fixed to work with Mirror Maker in TRUNK
> ------------------------------------------------------------------------
>
> Key: KAFKA-373
> URL: https://issues.apache.org/jira/browse/KAFKA-373
> Project: Kafka
> Issue Type: Task
> Reporter: John Fung
> Assignee: John Fung
> Attachments: kafka-373-v1.patch, kafka-373-v2.patch
>
>
> Embedded consumer is no longer supported and Mirror Maker is introduced to TRUNK. Broker Failure test needs to be fixed to use Mirror Maker for mirroring.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-373) Broker Failure Test needs to be fixed
to work with Mirror Maker in TRUNK
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Fung updated KAFKA-373:
----------------------------
Status: Patch Available (was: Open)
> Broker Failure Test needs to be fixed to work with Mirror Maker in TRUNK
> ------------------------------------------------------------------------
>
> Key: KAFKA-373
> URL: https://issues.apache.org/jira/browse/KAFKA-373
> Project: Kafka
> Issue Type: Task
> Reporter: John Fung
> Assignee: John Fung
> Attachments: kafka-373-v1.patch
>
>
> Embedded consumer is no longer supported and Mirror Maker is introduced to TRUNK. Broker Failure test needs to be fixed to use Mirror Maker for mirroring.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira