You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Joel Koshy (Created) (JIRA)" <ji...@apache.org> on 2011/11/11 03:28:52 UTC
[jira] [Created] (KAFKA-199) Add a how-to-mirror document
Add a how-to-mirror document
----------------------------
Key: KAFKA-199
URL: https://issues.apache.org/jira/browse/KAFKA-199
Project: Kafka
Issue Type: Improvement
Components: website
Reporter: Joel Koshy
I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-199) Add a how-to-mirror document
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148238#comment-13148238 ]
Jun Rao commented on KAFKA-199:
-------------------------------
Thanks, Joel. A couple of quick comments:
1. I feel that it's better to put this in our wiki.
2. We probably don't need to include new property files. We just need to list those properties unique to mirroring in the wiki.
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch, mirroring.png
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-199) Add a how-to-mirror document
Posted by "Joel Koshy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-199:
-----------------------------
Attachment: KAFKA-199.patch
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-199) Add a how-to-mirror document
Posted by "Joel Koshy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148221#comment-13148221 ]
Joel Koshy commented on KAFKA-199:
----------------------------------
Here is a draft of a mirroring document. Any comments? I'm not sure if it is
at the right level of detail. Does the diagram help much? How about the
quickstart/demo? Would a wiki work better than a patch for this?
Some other notes: the socket buffer size setting on the server is currently
ignored, but someone at LinkedIn is experimenting with it. Maybe we should
comment that out until that is done.
As discussed on the kafka-user mailing list, there is an interest in open
sourcing LinkedIn's ETL and audit system - the audit component is very
useful in checking the mirror. When that is available, this doc can be
updated.
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch, mirroring.png
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-199) Add a how-to-mirror document
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148258#comment-13148258 ]
Jay Kreps commented on KAFKA-199:
---------------------------------
Also, one comment, the bit about using mirroring for scalability doesn't quite make sense if you think about it. A single cluster of N machines should be able to handle more reads then two clusters of N/2 (even if you don't count the reads the mirror needs to do). It does help isolation for different SLAs (i.e. ad hoc usage), and multi data center stuff.
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch, mirroring.png
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-199) Add a how-to-mirror document
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148256#comment-13148256 ]
Jay Kreps commented on KAFKA-199:
---------------------------------
This is great! Personally I really prefer the site html to the wiki for reading, though the wiki is easier to edit.
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch, mirroring.png
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-199) Add a how-to-mirror document
Posted by "Joel Koshy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148771#comment-13148771 ]
Joel Koshy commented on KAFKA-199:
----------------------------------
That's right - doesn't help scalability, but facilitates isolation.
I also prefer the website to wiki. I moved it to a wiki for now (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring). We can let it sit there until it stabilizes. After the ETL stuff has been open-sourced, will add a section on auditing and then consider moving it to the html docs.
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch, mirroring.png
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-199) Add a how-to-mirror document
Posted by "Joel Koshy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-199:
-----------------------------
Attachment: mirroring.png
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch, mirroring.png
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-199) Add a how-to-mirror document
Posted by "Joel Koshy (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy resolved KAFKA-199.
------------------------------
Resolution: Later
Holding in wiki for now.
> Add a how-to-mirror document
> ----------------------------
>
> Key: KAFKA-199
> URL: https://issues.apache.org/jira/browse/KAFKA-199
> Project: Kafka
> Issue Type: Improvement
> Components: website
> Reporter: Joel Koshy
> Attachments: KAFKA-199.patch, mirroring.png
>
>
> I owe kafka-dev a mirroring document, and someone on the user mailing list had questions on mirroring which reminded me to work on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira