You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Sijie Guo (Created) (JIRA)" <ji...@apache.org> on 2011/10/08 14:00:30 UTC
[jira] [Created] (BOOKKEEPER-77) Add a console client for hedwig
Add a console client for hedwig
-------------------------------
Key: BOOKKEEPER-77
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
Project: Bookkeeper
Issue Type: New Feature
Components: hedwig-client
Reporter: Sijie Guo
Add a console client for hedwig:
1) show topics / hubs : list topics / available topics
2) pub/sub/unsub topic :
3) describe topic <topic> : show detail information of topic (including owner/ledgers/subscriptions)
Details:
sub <topic> <subscriberId> <mode> : subscribe a topic as subId
pub <topic> <message> : publish a message to topic
unsub <topic> <subscriberId> : unsubscribe topic for sub Id
show topics : list all the topics
show hubs: show available hub servers
describe topic <topicname> : show detail information about a specific topic
history : history commands
redo <cmdno> : redo the command by command no
redo ! : redo previous command
quit / exit : quit console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138243#comment-13138243 ]
jiraposter@reviews.apache.org commented on BOOKKEEPER-77:
---------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2570/
-----------------------------------------------------------
(Updated 2011-10-28 11:59:22.626468)
Review request for bookkeeper.
Changes
-------
made changed according to Ivan's suggestions:
1) move it to hedwig-server, so the console can be distributed with hedwig-server, which ease administration.
2) move commands and options to a separated file, change them to final static strings.
3) can use 'help <command>' to display detail usage for a specific command
4) change zkquorum to zkserver
since, 'describe topic <topic>' can display subscriptions of a specific topic, so I didn't add a new command for 'show subscriptions'.
Summary
-------
using zookeeper client to read metadata, use hedwig client to interact with hedwig, use bookkeeper client to openLedgerNoRecovery to read messages to implement a hedwig console
This addresses bug BOOKKEEPER-77.
http://issues.apache.org/jira/browse/BOOKKEEPER-77
Diffs (updated)
-----
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/bin/hedwig 1188996
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/pom.xml 1188996
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/HedwigCommands.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/HedwigConsole.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/JLineHedwigCompletor.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/ReadTopic.java PRE-CREATION
Diff: https://reviews.apache.org/r/2570/diff
Testing
-------
Thanks,
Sijie
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-77:
--------------------------------
Description:
implement a console client to use/admin hedwig system.
Usage : hedwig_console [options] COMMAND [argument ...]
(if no COMMAND specified, hedwig_console will enter interactive mode.)
OPTIONS:
{quote}
--zkquorums the quorum list of zookeeper cluster
--zktimeout timeout of zookeeper client
--zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
--region which region of hedwig to connect
--consume_interval the consume interval of hub server
{quote}
COMMANDS:
{quote}
pub {topic} {message}
Publish {message} to the specified {topic}.
sub {topic} {subscriber_id} [mode]
Subscribe the specified {topic} as subscriber {subscriber_id}. (NOTE: only run in INTERACTIVE mode now)
* mode: subscription mode. available values are 0, 1, 2.
o 0 = CREATE : create the subscription if not subscription before.
o 1 = ATTACH (default) : attach the subscription
o 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
closesub {topic} {subscriber_id}
Close subscription of subscriber {subscriber_id}. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
unsub {topic} {subscriber_id}
Remove subscription state of subscriber {subscriber_id}. the subscription state of subscriber {subscriber_id} will be removed from server side.
consume {topic} {subscriber_id} {num_messages_to_consume}
Move the subscription ptr of subscriber {subscriber_id} from ptr to ptr + num_messages_to_consume.
consumeto {topic} {subscriber_id} {message_id}
Move the subscription ptr of subscriber {subscriber_id} from ptr to {message_id}.
* NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
pubsub {topic} {subscriber_id_prefix} {timeout_secs} {message_prefix}
A test command to test healthy of hedwig cluster.
1. sub {topic} as subscriber {subscriber_id_prefix}_{cur_time} .
2. subscriber {subscriber_id_prefix}_{cur_time} will wait a message until {timeout_secs} secs.
3. publish a message {message_prefix}_{cur_time} to topic {topic} .
4. when subscriber {subscriber_id_prefix} receive the message, it will check the message is the published message
5. received message or timeout, subscriber {subscriber_id_prefix} will unsubscribe the {topic}
6. quit
[hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
Starting PUBSUB test ...
Sub topic ttttttttt, subscriber id test-1319602021044
Pub topic ttttttttt : test_message-1319602021044
Received message : test_message-1319602021044
PUBSUB SUCCESS. TIME: 43 MS
SUCCESS. Finished 0.058 s
show hubs
list all available hub servers. including hostname and how many topics the server owns.
* Example:
Available Hub Servers:
98.137.99.27:9875:9876 : 2
show topics
list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
describe topic {topic}
show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
* Example:
===== Topic Information : ttttt =====
Owner : 98.137.99.27:9875:9876
>>> Persistence Info <<<
Ledger 54729 [ 1 ~ 59 ]
Ledger 54731 [ 60 ~ 60 ]
>>> Subscription Info <<<
Subscriber mysub : consumeSeqId: local:50
readtopic {topic} [start_msg_id]
read messages of a specified {topic}.
* no {start_msg_id} specified : readtopic will start from {least_consumed_message_id} + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
* {start_msg_id} specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( {start_msg_id} , {least_consumed_message_id} ).
* Message Format
o MsgId : include two parts: first part is which region the message is published from, second part is message id.
o SrcRegion : region name
o Message : the message body
---------- MSGID=LOCAL(51) ----------
MsgId: LOCAL(51)
SrcRegion: standalone
Message:
hello
* Example:
[hedwig: (standalone) 5] readtopic ttttt
>>>>> Ledger 54729 [ 1 ~ 59] <<<<<
---------- MSGID=LOCAL(51) ----------
MsgId: LOCAL(51)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(52) ----------
MsgId: LOCAL(52)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(53) ----------
MsgId: LOCAL(53)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(54) ----------
MsgId: LOCAL(54)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(55) ----------
MsgId: LOCAL(55)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(56) ----------
MsgId: LOCAL(56)
SrcRegion: standalone
Message:
hello-56
---------- MSGID=LOCAL(57) ----------
MsgId: LOCAL(57)
SrcRegion: standalone
Message:
hello-57
---------- MSGID=LOCAL(58) ----------
MsgId: LOCAL(58)
SrcRegion: standalone
Message:
hello-58
---------- MSGID=LOCAL(59) ----------
MsgId: LOCAL(59)
SrcRegion: standalone
Message:
hello-59
Press Y to continue...
history
list history commands
redo [{cmdno}|!]
redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
help
print help information
quit|exit
exit the interactive console
{quote}
was:
Add a console client for hedwig:
1) show topics / hubs : list topics / available topics
2) pub/sub/unsub topic :
3) describe topic <topic> : show detail information of topic (including owner/ledgers/subscriptions)
Details:
sub <topic> <subscriberId> <mode> : subscribe a topic as subId
pub <topic> <message> : publish a message to topic
unsub <topic> <subscriberId> : unsubscribe topic for sub Id
show topics : list all the topics
show hubs: show available hub servers
describe topic <topicname> : show detail information about a specific topic
history : history commands
redo <cmdno> : redo the command by command no
redo ! : redo previous command
quit / exit : quit console
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> {quote}
> pub {topic} {message}
> Publish {message} to the specified {topic}.
> sub {topic} {subscriber_id} [mode]
> Subscribe the specified {topic} as subscriber {subscriber_id}. (NOTE: only run in INTERACTIVE mode now)
> * mode: subscription mode. available values are 0, 1, 2.
> o 0 = CREATE : create the subscription if not subscription before.
> o 1 = ATTACH (default) : attach the subscription
> o 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> closesub {topic} {subscriber_id}
> Close subscription of subscriber {subscriber_id}. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> unsub {topic} {subscriber_id}
> Remove subscription state of subscriber {subscriber_id}. the subscription state of subscriber {subscriber_id} will be removed from server side.
> consume {topic} {subscriber_id} {num_messages_to_consume}
> Move the subscription ptr of subscriber {subscriber_id} from ptr to ptr + num_messages_to_consume.
> consumeto {topic} {subscriber_id} {message_id}
> Move the subscription ptr of subscriber {subscriber_id} from ptr to {message_id}.
> * NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> pubsub {topic} {subscriber_id_prefix} {timeout_secs} {message_prefix}
> A test command to test healthy of hedwig cluster.
> 1. sub {topic} as subscriber {subscriber_id_prefix}_{cur_time} .
> 2. subscriber {subscriber_id_prefix}_{cur_time} will wait a message until {timeout_secs} secs.
> 3. publish a message {message_prefix}_{cur_time} to topic {topic} .
> 4. when subscriber {subscriber_id_prefix} receive the message, it will check the message is the published message
> 5. received message or timeout, subscriber {subscriber_id_prefix} will unsubscribe the {topic}
> 6. quit
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
>
> show hubs
> list all available hub servers. including hostname and how many topics the server owns.
> * Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
>
> show topics
> list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> describe topic {topic}
> show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> * Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
>
> readtopic {topic} [start_msg_id]
> read messages of a specified {topic}.
> * no {start_msg_id} specified : readtopic will start from {least_consumed_message_id} + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> * {start_msg_id} specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( {start_msg_id} , {least_consumed_message_id} ).
> * Message Format
> o MsgId : include two parts: first part is which region the message is published from, second part is message id.
> o SrcRegion : region name
> o Message : the message body
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
>
> * Example:
> [hedwig: (standalone) 5] readtopic ttttt
> >>>>> Ledger 54729 [ 1 ~ 59] <<<<<
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> ---------- MSGID=LOCAL(52) ----------
> MsgId: LOCAL(52)
> SrcRegion: standalone
> Message:
> hello
> ---------- MSGID=LOCAL(53) ----------
> MsgId: LOCAL(53)
> SrcRegion: standalone
> Message:
> hello
> ---------- MSGID=LOCAL(54) ----------
> MsgId: LOCAL(54)
> SrcRegion: standalone
> Message:
> hello
> ---------- MSGID=LOCAL(55) ----------
> MsgId: LOCAL(55)
> SrcRegion: standalone
> Message:
> hello
> ---------- MSGID=LOCAL(56) ----------
> MsgId: LOCAL(56)
> SrcRegion: standalone
> Message:
> hello-56
> ---------- MSGID=LOCAL(57) ----------
> MsgId: LOCAL(57)
> SrcRegion: standalone
> Message:
> hello-57
> ---------- MSGID=LOCAL(58) ----------
> MsgId: LOCAL(58)
> SrcRegion: standalone
> Message:
> hello-58
> ---------- MSGID=LOCAL(59) ----------
> MsgId: LOCAL(59)
> SrcRegion: standalone
> Message:
> hello-59
> Press Y to continue...
>
> history
> list history commands
> redo [{cmdno}|!]
> redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> help
> print help information
> quit|exit
> exit the interactive console
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-77:
--------------------------------
Attachment: bookkeeper-77-draft.patch
attach a draft version of hedwig console client.
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch
>
>
> Add a console client for hedwig:
> 1) show topics / hubs : list topics / available topics
> 2) pub/sub/unsub topic :
> 3) describe topic <topic> : show detail information of topic (including owner/ledgers/subscriptions)
> Details:
> sub <topic> <subscriberId> <mode> : subscribe a topic as subId
> pub <topic> <message> : publish a message to topic
> unsub <topic> <subscriberId> : unsubscribe topic for sub Id
> show topics : list all the topics
> show hubs: show available hub servers
> describe topic <topicname> : show detail information about a specific topic
> history : history commands
> redo <cmdno> : redo the command by command no
> redo ! : redo previous command
> quit / exit : quit console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-77:
--------------------------------
Description:
implement a console client to use/admin hedwig system.
Usage : hedwig_console [options] COMMAND [argument ...]
(if no COMMAND specified, hedwig_console will enter interactive mode.)
OPTIONS:
{quote}
--zkquorums the quorum list of zookeeper cluster
--zktimeout timeout of zookeeper client
--zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
--region which region of hedwig to connect
--consume_interval the consume interval of hub server
{quote}
COMMANDS:
* pub <topic> <message>
** Publish <message> to the specified <topic>.
* sub <topic> <subscriber_id> [mode]
** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
** mode: subscription mode. available values are 0, 1, 2.
*** 0 = CREATE : create the subscription if not subscription before.
*** 1 = ATTACH (default) : attach the subscription
*** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
* closesub <topic> <subscriber_id>
** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
* unsub <topic> <subscriber_id>
** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
* consume <topic> <subscriber_id> <num_messages_to_consume>
** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
* consumeto <topic> <subscriber_id> <message_id>
** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
* pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
** A test command to test healthy of hedwig cluster.
*# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
*# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
*# publish a message <message_prefix>_<cur_time> to topic <topic> .
*# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
*# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
*# quit
{quote}
[hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
Starting PUBSUB test ...
Sub topic ttttttttt, subscriber id test-1319602021044
Pub topic ttttttttt : test_message-1319602021044
Received message : test_message-1319602021044
PUBSUB SUCCESS. TIME: 43 MS
SUCCESS. Finished 0.058 s
{quote}
* show hubs
** list all available hub servers. including hostname and how many topics the server owns.
{quote}
Example:
Available Hub Servers:
98.137.99.27:9875:9876 : 2
{quote}
* show topics
** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
* describe topic <topic>
** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
{quote}
Example:
===== Topic Information : ttttt =====
Owner : 98.137.99.27:9875:9876
>>> Persistence Info <<<
Ledger 54729 [ 1 ~ 59 ]
Ledger 54731 [ 60 ~ 60 ]
>>> Subscription Info <<<
Subscriber mysub : consumeSeqId: local:50
{quote}
* readtopic <topic> [start_msg_id]
** read messages of a specified <topic>.
*** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
*** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
*** Message Format
**** MsgId : include two parts: first part is which region the message is published from, second part is message id.
**** SrcRegion : region name
**** Message : the message body
{quote}
---------- MSGID=LOCAL(51) ----------
MsgId: LOCAL(51)
SrcRegion: standalone
Message:
hello
{quote}
* history
** list history commands
* redo [<cmdno>|!]
** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
* help
** print help information
* quit|exit
** exit the interactive console
was:
implement a console client to use/admin hedwig system.
Usage : hedwig_console [options] COMMAND [argument ...]
(if no COMMAND specified, hedwig_console will enter interactive mode.)
OPTIONS:
{quote}
--zkquorums the quorum list of zookeeper cluster
--zktimeout timeout of zookeeper client
--zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
--region which region of hedwig to connect
--consume_interval the consume interval of hub server
{quote}
COMMANDS:
{quote}
pub {topic} {message}
Publish {message} to the specified {topic}.
sub {topic} {subscriber_id} [mode]
Subscribe the specified {topic} as subscriber {subscriber_id}. (NOTE: only run in INTERACTIVE mode now)
* mode: subscription mode. available values are 0, 1, 2.
o 0 = CREATE : create the subscription if not subscription before.
o 1 = ATTACH (default) : attach the subscription
o 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
closesub {topic} {subscriber_id}
Close subscription of subscriber {subscriber_id}. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
unsub {topic} {subscriber_id}
Remove subscription state of subscriber {subscriber_id}. the subscription state of subscriber {subscriber_id} will be removed from server side.
consume {topic} {subscriber_id} {num_messages_to_consume}
Move the subscription ptr of subscriber {subscriber_id} from ptr to ptr + num_messages_to_consume.
consumeto {topic} {subscriber_id} {message_id}
Move the subscription ptr of subscriber {subscriber_id} from ptr to {message_id}.
* NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
pubsub {topic} {subscriber_id_prefix} {timeout_secs} {message_prefix}
A test command to test healthy of hedwig cluster.
1. sub {topic} as subscriber {subscriber_id_prefix}_{cur_time} .
2. subscriber {subscriber_id_prefix}_{cur_time} will wait a message until {timeout_secs} secs.
3. publish a message {message_prefix}_{cur_time} to topic {topic} .
4. when subscriber {subscriber_id_prefix} receive the message, it will check the message is the published message
5. received message or timeout, subscriber {subscriber_id_prefix} will unsubscribe the {topic}
6. quit
[hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
Starting PUBSUB test ...
Sub topic ttttttttt, subscriber id test-1319602021044
Pub topic ttttttttt : test_message-1319602021044
Received message : test_message-1319602021044
PUBSUB SUCCESS. TIME: 43 MS
SUCCESS. Finished 0.058 s
show hubs
list all available hub servers. including hostname and how many topics the server owns.
* Example:
Available Hub Servers:
98.137.99.27:9875:9876 : 2
show topics
list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
describe topic {topic}
show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
* Example:
===== Topic Information : ttttt =====
Owner : 98.137.99.27:9875:9876
>>> Persistence Info <<<
Ledger 54729 [ 1 ~ 59 ]
Ledger 54731 [ 60 ~ 60 ]
>>> Subscription Info <<<
Subscriber mysub : consumeSeqId: local:50
readtopic {topic} [start_msg_id]
read messages of a specified {topic}.
* no {start_msg_id} specified : readtopic will start from {least_consumed_message_id} + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
* {start_msg_id} specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( {start_msg_id} , {least_consumed_message_id} ).
* Message Format
o MsgId : include two parts: first part is which region the message is published from, second part is message id.
o SrcRegion : region name
o Message : the message body
---------- MSGID=LOCAL(51) ----------
MsgId: LOCAL(51)
SrcRegion: standalone
Message:
hello
* Example:
[hedwig: (standalone) 5] readtopic ttttt
>>>>> Ledger 54729 [ 1 ~ 59] <<<<<
---------- MSGID=LOCAL(51) ----------
MsgId: LOCAL(51)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(52) ----------
MsgId: LOCAL(52)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(53) ----------
MsgId: LOCAL(53)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(54) ----------
MsgId: LOCAL(54)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(55) ----------
MsgId: LOCAL(55)
SrcRegion: standalone
Message:
hello
---------- MSGID=LOCAL(56) ----------
MsgId: LOCAL(56)
SrcRegion: standalone
Message:
hello-56
---------- MSGID=LOCAL(57) ----------
MsgId: LOCAL(57)
SrcRegion: standalone
Message:
hello-57
---------- MSGID=LOCAL(58) ----------
MsgId: LOCAL(58)
SrcRegion: standalone
Message:
hello-58
---------- MSGID=LOCAL(59) ----------
MsgId: LOCAL(59)
SrcRegion: standalone
Message:
hello-59
Press Y to continue...
history
list history commands
redo [{cmdno}|!]
redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
help
print help information
quit|exit
exit the interactive console
{quote}
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135859#comment-13135859 ]
jiraposter@reviews.apache.org commented on BOOKKEEPER-77:
---------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2570/
-----------------------------------------------------------
Review request for bookkeeper.
Summary
-------
using zookeeper client to read metadata, use hedwig client to interact with hedwig, use bookkeeper client to openLedgerNoRecovery to read messages to implement a hedwig console
This addresses bug BOOKKEEPER-77.
http://issues.apache.org/jira/browse/BOOKKEEPER-77
Diffs
-----
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/bin/hedwig_console PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/conf/hedwig-env.sh PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/conf/hedwig_console.conf PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/pom.xml PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/HedwigConsole.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/JLineHedwigCompletor.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/ReadTopic.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/pom.xml 1188996
Diff: https://reviews.apache.org/r/2570/diff
Testing
-------
Thanks,
Sijie
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-77:
--------------------------------
Attachment: hedwig-console.tar.gz
since hedwig_console will use zookeeper client to read metadata, use hedwig client to interact with hedwig, use bookkeeper client to openLedgerNoRecovery to read messages.
so let hedwig_console be a independent module.
attach it as a package.
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135854#comment-13135854 ]
Ivan Kelly commented on BOOKKEEPER-77:
--------------------------------------
Could you put this patch in review board?
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198749#comment-13198749 ]
jiraposter@reviews.apache.org commented on BOOKKEEPER-77:
---------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2570/
-----------------------------------------------------------
(Updated 2012-02-02 12:48:24.354170)
Review request for bookkeeper.
Changes
-------
attach a new patch based on trunk. in this patch, I did some cleanup to move all the meta-related operations such as getTopics, show topic info into a admin class.
Summary
-------
using zookeeper client to read metadata, use hedwig client to interact with hedwig, use bookkeeper client to openLedgerNoRecovery to read messages to implement a hedwig console
This addresses bug BOOKKEEPER-77.
http://issues.apache.org/jira/browse/BOOKKEEPER-77
Diffs (updated)
-----
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/bin/hedwig 1239585
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/pom.xml 1239585
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/HedwigAdmin.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/HedwigCommands.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/HedwigConsole.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/JLineHedwigCompletor.java PRE-CREATION
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/ReadTopic.java PRE-CREATION
Diff: https://reviews.apache.org/r/2570/diff
Testing
-------
Thanks,
Sijie
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.1.0
>
> Attachments: BK-77.patch, bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203484#comment-13203484 ]
Hudson commented on BOOKKEEPER-77:
----------------------------------
Integrated in bookkeeper-trunk #349 (See [https://builds.apache.org/job/bookkeeper-trunk/349/])
BOOKKEEPER-77: Add a console client for hedwig (Sijie Guo via ivank)
ivank :
Files :
* /zookeeper/bookkeeper/trunk/CHANGES.txt
* /zookeeper/bookkeeper/trunk/hedwig-server/bin/hedwig
* /zookeeper/bookkeeper/trunk/hedwig-server/pom.xml
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/HedwigAdmin.java
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/HedwigCommands.java
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/HedwigConsole.java
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/JLineHedwigCompletor.java
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/admin/console/ReadTopic.java
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BK-77.patch, bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197878#comment-13197878 ]
Sijie Guo commented on BOOKKEEPER-77:
-------------------------------------
thanks, I would create a patch based on trunk.
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.1.0
>
> Attachments: bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135850#comment-13135850 ]
Sijie Guo commented on BOOKKEEPER-77:
-------------------------------------
implementation of hedwig console:
* pub
** use hedwigclient.getPublisher().publish(...)
* sub
** use hedwigclient.getSubscriber().subscribe()
* closesub
** use hedwigclient.getSubscriber().closeSubscription()
* unsub
** use hedwigclient.getSubscriber().unsubscribe()
* consume/consumeto
** use hedwigclient.getSubscriber().consume()
* pubsub
*# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
*# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
*# publish a message <message_prefix>_<cur_time> to topic <topic> .
*# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
*# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
*# quit
* describe topic
*# check whether the topic existed or not : zk.exists("/hedwig/<region>/topics/<topic>").
*## if existed, go next
*## else return
*# read the topic owner from zookeeper : /hedwig/<region>/topics/<topic>/owner
*# read the persistent information from zookeeper : /hedwig/<region>/topics/<topic>/ledgers
*## for each ledger, parse its data to get its start message id and end message id
*### if the ledger is not closed (which means the ledger is used for persist messages not), we use bookkeeper client to openLedgerNoRecovery to read the lastConfirmed message id
*# read the subscription information from zookeeper : /hedwig/<region>/topics/<topic>/subscribers
*## for each subscriber, parse its data to get the subscription state
* readtopic
*# check whether the topic existed or not : zk.exists("/hedwig/<region>/topics/<topic>").
*# read the persistent information from zookeeper
*# read the subscription information from zookeeper to get the <least_consumed_message_id>
*# find <start_message_id> by MAX( <least_consumed_message_id>, <start_message_id> )
*# loop over the ledgers used for persisting messsages
*## for each ledger, if the messages are all less than <start_message_id>, skip the ledger
*## otherwise, open ledger by calling openLedgerNoRecovery (just for reading) to read the messages and format them
* show hubs
* fetch list of hosts from zookeeper /hedwig/<region>/available , for each host read its data and convert it to integer.
* show topics
** fetch list of topics from zookeeper /hedwig/<region>/topics
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-77:
--------------------------------
Attachment: BK-77.patch
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.1.0
>
> Attachments: BK-77.patch, bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135872#comment-13135872 ]
jiraposter@reviews.apache.org commented on BOOKKEEPER-77:
---------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2570/#review2854
-----------------------------------------------------------
This is very nice :)
I have a few general comments.
- Firstly, I dont think it should be a separate package. I would be inclined to include it in hedwig-server, and put the a cli option in hedwig-server/bin/hedwig. Its an admin tool really, so it should go with the server.
- Is it possible to intergrate some help with the actual commands. I don't know about jline, but when I've done cli interfaces in the past it's always been possible to attach the help to the actual commands.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/HedwigConsole.java
<https://reviews.apache.org/r/2570/#comment6388>
Could you add a "show subscriptions" here.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/HedwigConsole.java
<https://reviews.apache.org/r/2570/#comment6389>
zkquorums should be zkquorum, or even better zkserver. you only need to connect to one server, not a quorum and it's unlikely you'll change zkserver during the lifetime of the client.
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/HedwigConsole.java
<https://reviews.apache.org/r/2570/#comment6390>
These keys should be defined as final static Strings rather than using the actual string for each option.
- Ivan
On 2011-10-26 10:54:04, Sijie Guo wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2570/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-26 10:54:04)
bq.
bq.
bq. Review request for bookkeeper.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. using zookeeper client to read metadata, use hedwig client to interact with hedwig, use bookkeeper client to openLedgerNoRecovery to read messages to implement a hedwig console
bq.
bq.
bq. This addresses bug BOOKKEEPER-77.
bq. http://issues.apache.org/jira/browse/BOOKKEEPER-77
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/bin/hedwig_console PRE-CREATION
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/conf/hedwig-env.sh PRE-CREATION
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/conf/hedwig_console.conf PRE-CREATION
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/pom.xml PRE-CREATION
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/HedwigConsole.java PRE-CREATION
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/JLineHedwigCompletor.java PRE-CREATION
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/hedwig-console/src/main/java/org/apache/hedwig/client/console/ReadTopic.java PRE-CREATION
bq. http://svn.apache.org/repos/asf/zookeeper/bookkeeper/trunk/pom.xml 1188996
bq.
bq. Diff: https://reviews.apache.org/r/2570/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Sijie
bq.
bq.
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Reporter: Sijie Guo
> Attachments: bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (BOOKKEEPER-77) Add a console client for hedwig
Posted by "Ivan Kelly (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly reassigned BOOKKEEPER-77:
------------------------------------
Assignee: Sijie Guo
> Add a console client for hedwig
> -------------------------------
>
> Key: BOOKKEEPER-77
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-77
> Project: Bookkeeper
> Issue Type: New Feature
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.1.0
>
> Attachments: bookkeeper-77-draft.patch, hedwig-console.tar.gz
>
>
> implement a console client to use/admin hedwig system.
> Usage : hedwig_console [options] COMMAND [argument ...]
> (if no COMMAND specified, hedwig_console will enter interactive mode.)
> OPTIONS:
> {quote}
> --zkquorums the quorum list of zookeeper cluster
> --zktimeout timeout of zookeeper client
> --zk_hedwig_prefix the prefix of zookeeper path to store hedwig metadata
> --region which region of hedwig to connect
> --consume_interval the consume interval of hub server
> {quote}
> COMMANDS:
> * pub <topic> <message>
> ** Publish <message> to the specified <topic>.
> * sub <topic> <subscriber_id> [mode]
> ** Subscribe the specified <topic> as subscriber <subscriber_id>. (NOTE: only run in INTERACTIVE mode now)
> ** mode: subscription mode. available values are 0, 1, 2.
> *** 0 = CREATE : create the subscription if not subscription before.
> *** 1 = ATTACH (default) : attach the subscription
> *** 2 = CREATE_OR_ATTACH : if the subscription is not existed, create the subscription then attach.
> * closesub <topic> <subscriber_id>
> ** Close subscription of subscriber <subscriber_id>. (NOTE: it just close the subscription connection and do cleanup work in client-side, without REMOVING subscription state from server side)
> * unsub <topic> <subscriber_id>
> ** Remove subscription state of subscriber <subscriber_id>. the subscription state of subscriber <subscriber_id> will be removed from server side.
> * consume <topic> <subscriber_id> <num_messages_to_consume>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to ptr + num_messages_to_consume.
> * consumeto <topic> <subscriber_id> <message_id>
> ** Move the subscription ptr of subscriber <subscriber_id> from ptr to <message_id>.
> ** NOTE: consume*/*consumeto just sent consume request to hub server and hub server move the subscription ptr in its memory. Hub server lazily persists the subscription ptr to zookeeper. the default persist interval in hub server is 50 messages. so use DESCRIBE TOPIC to show subscription, the subscription ptr might be not changed.
> * pubsub <topic> <subscriber_id_prefix> <timeout_secs> <message_prefix>
> ** A test command to test healthy of hedwig cluster.
> *# sub <topic> as subscriber <subscriber_id_prefix>_<cur_time> .
> *# subscriber <subscriber_id_prefix>_<cur_time> will wait a message until <timeout_secs> secs.
> *# publish a message <message_prefix>_<cur_time> to topic <topic> .
> *# when subscriber <subscriber_id_prefix> receive the message, it will check the message is the published message
> *# received message or timeout, subscriber <subscriber_id_prefix> will unsubscribe the <topic>
> *# quit
> {quote}
> [hedwig: (standalone) 7] pubsub ttttttttt test 10 test_message
> Starting PUBSUB test ...
> Sub topic ttttttttt, subscriber id test-1319602021044
> Pub topic ttttttttt : test_message-1319602021044
> Received message : test_message-1319602021044
> PUBSUB SUCCESS. TIME: 43 MS
> SUCCESS. Finished 0.058 s
> {quote}
> * show hubs
> ** list all available hub servers. including hostname and how many topics the server owns.
> {quote}
> Example:
> Available Hub Servers:
> 98.137.99.27:9875:9876 : 2
> {quote}
>
> * show topics
> ** list all existing topics. (NOTE: since we fetch topic lists from zookeeper, we may got PacketLenException when we have millions of topics. it doesn't affect system, just can't display the topic list)
> * describe topic <topic>
> ** show state of a specified topic, including topic owner, topic persistent information, topic subscriber list and their subscription states.
> {quote}
> Example:
> ===== Topic Information : ttttt =====
> Owner : 98.137.99.27:9875:9876
> >>> Persistence Info <<<
> Ledger 54729 [ 1 ~ 59 ]
> Ledger 54731 [ 60 ~ 60 ]
> >>> Subscription Info <<<
> Subscriber mysub : consumeSeqId: local:50
> {quote}
>
> * readtopic <topic> [start_msg_id]
> ** read messages of a specified <topic>.
> *** no <start_msg_id> specified : readtopic will start from <least_consumed_message_id> + 1 of its subscribers. in above exmaple, "readtopic ttttt" will start from 50. if there is no subscription, it will start from 1.
> *** <start_msg_id> specified : since messages consumed will be removed by garbage collection. so readtopic tries to not read consumed message, it will start from MAX( <start_msg_id> , <least_consumed_message_id> ).
> *** Message Format
> **** MsgId : include two parts: first part is which region the message is published from, second part is message id.
> **** SrcRegion : region name
> **** Message : the message body
> {quote}
> ---------- MSGID=LOCAL(51) ----------
> MsgId: LOCAL(51)
> SrcRegion: standalone
> Message:
> hello
> {quote}
> * history
> ** list history commands
> * redo [<cmdno>|!]
> ** redo the specified command by command no. (NOTE: "*redo *" means redo the previous command)
> * help
> ** print help information
> * quit|exit
> ** exit the interactive console
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira