You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/05/16 18:31:03 UTC
[jira] [Created] (KAFKA-344) migration tool from 0.7 to 0.8
Jun Rao created KAFKA-344:
-----------------------------
Summary: migration tool from 0.7 to 0.8
Key: KAFKA-344
URL: https://issues.apache.org/jira/browse/KAFKA-344
Project: Kafka
Issue Type: Sub-task
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
Fix For: 0.8
Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-344:
--------------------------
Attachment: kafka_07_producer_performance.patch
zkclient-0.1.jar
kafka-0.7.0.jar
kafka_344_v1.diff
1. The current system test assume there is kafka 0.7 project at location, "~/Dev/kafka_0.7" (otherwise you need to specify the kafka 07 location at "cluster_config.json" file
2. The patch make the replication_testsuite not working because they share the same cluster_config.json, I've talked with John to create three level cluster configuration: testcase, testsuite and global level. If configuration file is found at "testcase" level, the framework won't search the other two level, otherwise if it's found in testsuite level, the framework won't search the global level. Will rebase after John added this feature
3. Added a few roles: "broker07", "zookeeper07", "producer_performance07", and "migration_tool" as specified in the cluster_config.json file
4. add a few functions in Kafka_system_test_utils.py in accordance with the new roles, like start_zookeeper07(), start_producer_performance07(), etc
5. patch the kafka 07 producer performance, by adding message id to the produced messages.
6. please manually add the two jar files to "system_test/migration_tool_testsuite" directory as they cannot be included in the patch file
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye reassigned KAFKA-344:
-----------------------------
Assignee: Yang Ye
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Labels: tools
> Fix For: 0.8
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457175#comment-13457175 ]
Jun Rao commented on KAFKA-344:
-------------------------------
Thanks for the draft patch. Very tricky to get right. It does seem to work. Good job! Look forward to the final patch.
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka_344_draft.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede closed KAFKA-344.
-------------------------------
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_standalone.diff, kafka_344_standalone_v2.diff, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede resolved KAFKA-344.
---------------------------------
Resolution: Fixed
Yes, this is done now
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_standalone.diff, kafka_344_standalone_v2.diff, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461879#comment-13461879 ]
Jun Rao commented on KAFKA-344:
-------------------------------
Thanks for the patch. Committed the tool itself and the change to config/producer.properties to 0.8. Will close the jira once the system test change is done.
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_standalone.diff, kafka_344_standalone_v2.diff, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-344:
-----------------------------
Labels: tools (was: )
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: tools
> Fix For: 0.8
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-344:
--------------------------
Attachment: kafka_344_standalone_v2.diff
Thank you for the comments, it's really careful of you and careless of me...
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_standalone.diff, kafka_344_standalone_v2.diff, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502950#comment-13502950 ]
Jay Kreps commented on KAFKA-344:
---------------------------------
This is done now, right?
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_standalone.diff, kafka_344_standalone_v2.diff, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-344:
--------------------------
Attachment: kafka_344_standalone.diff
All comments address. Also in the code, force rewrite the 08 producer's property "serializer.class" to "kafka.serializer.DefaultEncoder", and the property "producer.type" to "sync"
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_standalone.diff, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460604#comment-13460604 ]
Jun Rao commented on KAFKA-344:
-------------------------------
Thanks for patch v2. There are still comments in the previous review that are not addressed:
1.2 Since this is a command line tool, if the program stops because of an error, we need to print out the exception message and the stacktrace to stdout, instead of log4j so that users can see what's wrong. log4j output may not always be in stdout.
1.3. Currently, one can still omit both whitelist and blacklist without being warned. We need to fix this.
2.1 We shouldn't print the message in the following line since it's binary. Instead, we should log the size and the topic.
logger.debug("send 08 message: " + bytes.toString());
Good change to force write "serializer.class" to "kafka.serializer.DefaultEncoder". We shouldn't force write the property "producer.type" to "sync" though. In most cases, we should probably use async for better throughput. However, we can let users make that decision. Please change the comment in the file accordingly.
Also, broker.list in config/producer.properties has the wrong format and comment. Could you fix it in this patch too?
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_standalone.diff, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459816#comment-13459816 ]
Jun Rao commented on KAFKA-344:
-------------------------------
Thanks for patch v1. Some comments.
1. main:
1.1 Add some comments at the beginning that describe how the tool is built.
1.2 print out an error message on exception in addition to the stack trace.
1.3 Most required command line options are not enforced, e.g. consumer.config, zkclient.01.jar, kafka.07.jar. Also, at least one of the whilelist/blacklist needs to be specified.
1.4 There is a weird regex to subscribe to all topics. Could you add that in the description of whitelist? I can't seem that get wildcard working. Tried things like --whilelist "test*". However, it seems to only pick up topic "test", but not "test1".
2. MigrationThread:
2.1 The following line should be written to log4j under DEBUG level, instead of stdout. Also, since message is binary, printing out the content is useless. We can probably just output the topic name and the message length.
System.out.println("send 08 message: " + bytes.toString());
2.2 Normally, we should never finish the while loop. However, it would be useful to add an info level logging to indicate if the loop is finished.
2.3 IterableNextMethod is not used.
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka-0.7.0.jar, kafka_07_producer_performance.patch, kafka_344_draft.patch, kafka_344_v1.diff, zkclient-0.1.jar
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-344:
--------------------------------
Parent Issue: KAFKA-327 (was: KAFKA-50)
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka_344_draft.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-344:
--------------------------
Attachment: kafka_344_draft.patch
Sorry the system test is still not done yet. The current system test framework assume a global "cluster_config.json" file, it's used by all testSuites and all test cases under each suite. But current cluster config (3 broker, 1 zookeeper, 1 consumer, 1 producer) does not apply to migration test, so I have to create a new one (1 07 zookeeper, 1 08 zookeeper, 3 07 brokers, 3 08 brokers, 1 07 producer, 1 migration tool, 1 08 consumer),
But the current replication_testSuite cannot work with this configration, I have to delete it temporarily. (I've talked with John to enable local cluster configuration instead of just global one)
I created new "role"s of "broker07", "zookeeper07" and "producer_performance07", and changed a few files in the system test framework. Currently it is still not working, continuing on it...
Please help review the migration tool itself
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka_344_draft.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-344) migration tool from 0.7 to 0.8
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-344:
-----------------------------
Priority: Blocker (was: Major)
> migration tool from 0.7 to 0.8
> ------------------------------
>
> Key: KAFKA-344
> URL: https://issues.apache.org/jira/browse/KAFKA-344
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: tools
> Fix For: 0.8
>
> Attachments: kafka_344_draft.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Since 0.8 is a non-backward compatible release, we need a migration tool. One way to do that is to have a tool that can mirror data from an 0.7 Kafka cluster into an 0.8 Kafka cluster. Once the mirror is set up, we can then first migrate the consumers to the 0.8 cluster, followed by the producers. After that, we can decommission the 0.7 cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira