You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zookeeper.apache.org by fp...@apache.org on 2012/12/11 11:15:21 UTC
svn commit: r1420069 - in /zookeeper/bookkeeper/trunk: CHANGES.txt
doc/hedwigParams.textile doc/index.textile hedwig-server/conf/hw_server.conf
hedwig-server/src/main/java/org/apache/hedwig/server/common/ServerConfiguration.java
Author: fpj
Date: Tue Dec 11 10:15:19 2012
New Revision: 1420069
URL: http://svn.apache.org/viewvc?rev=1420069&view=rev
Log:
BOOKKEEPER-491: Hedwig doc for configuration (fpj, sijie via fpj)
Added:
zookeeper/bookkeeper/trunk/doc/hedwigParams.textile
Modified:
zookeeper/bookkeeper/trunk/CHANGES.txt
zookeeper/bookkeeper/trunk/doc/index.textile
zookeeper/bookkeeper/trunk/hedwig-server/conf/hw_server.conf
zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/common/ServerConfiguration.java
Modified: zookeeper/bookkeeper/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/zookeeper/bookkeeper/trunk/CHANGES.txt?rev=1420069&r1=1420068&r2=1420069&view=diff
==============================================================================
--- zookeeper/bookkeeper/trunk/CHANGES.txt (original)
+++ zookeeper/bookkeeper/trunk/CHANGES.txt Tue Dec 11 10:15:19 2012
@@ -188,6 +188,8 @@ Trunk (unreleased changes)
BOOKKEEPER-476: Log to file during tests (ivank via fpj)
+ BOOKKEEPER-491: Hedwig doc for configuration (fpj, sijie via fpj)
+
bookkeeper-server:
BOOKKEEPER-328: Bookie DeathWatcher is missing thread name (Rakesh via sijie)
Added: zookeeper/bookkeeper/trunk/doc/hedwigParams.textile
URL: http://svn.apache.org/viewvc/zookeeper/bookkeeper/trunk/doc/hedwigParams.textile?rev=1420069&view=auto
==============================================================================
--- zookeeper/bookkeeper/trunk/doc/hedwigParams.textile (added)
+++ zookeeper/bookkeeper/trunk/doc/hedwigParams.textile Tue Dec 11 10:15:19 2012
@@ -0,0 +1,91 @@
+Notice: Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License. You may
+ obtain a copy of the License at "http://www.apache.org/licenses/LICENSE-2.0":http://www.apache.org/licenses/LICENSE-2.0.
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an "AS IS"
+ BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied. See the License for the specific language governing permissions
+ and limitations under the License.
+ .
+
+h1. Hedwig configuration parameters
+
+This page contains detailed information about configuration parameters used for Hubs, Regions, ZooKeeper, and BookKeeper.
+
+h2. Hedwig server configuration parameters
+
+Please also refer to the configuration file that comes with the distribution: _hedwig-server/conf/hw_server.conf_.
+
+h3. Region related parameters
+
+| @region@ | Region identifier. Default is "standalone". |
+| @regions@ | List of region identifiers, space separated. Default is empty. |
+| @inter_region_ssl_enabled (deprecated)@ | Enables SSL across regions. Default is false. *Since this parameter has been deprecated, use __ssl_enabled__ in _hedwig-server/conf/hw_region_client.conf_ to enable SSL across regions instead.* |
+| @retry_remote_subscribe_thread_run_interval@ | This parameter is used to determine how often we run a thread to retry those failed remote subscriptions in asynchronous mode (in milliseconds). Default is 2 minutes. |
+
+h3. Hub server parameters
+
+| @standalone@ | Sets the hub server to run in standalone mode (no regions). Default is false. |
+| @server_port@ | Sets the server port that receives client connections. Default is 4080. |
+| @ssl_enabled@ | Enables SSL. Default is false. |
+| @ssl_server_port@ | Sets the server port for SSL connections. Default is 9876. |
+| @password@ | Password used for pkcs12 certificate.. Default is the empty string. |
+| @cert_name@ | Sets the name of the SSL certificate if available as a resource. Default is the null string. |
+| @cert_path@ | Sets the path to the SSL certificate if it is available as a file. Default is the null string. |
+
+h3. Read-ahead cache parameters
+
+| @readahead_enabled@ | Enables read-ahead. Enabled by default. |
+| @readahead_count@ | Number of messages to read ahead. Default is 10. |
+| @readahead_size@ | Maximum number of bytes to read during a scan. Default is 4 megabytes. |
+
+bq. Upon a range scan request for a given topic, two hints are provided as to when scanning should stop: the number of messages scanned and the total size of messages scanned. Scanning stops whenever one of these limits is exceeded.
+
+| @cache_size@ | Sets the size of the read-ahead cache. Default is the smallest of 2G or half the heap size. |
+| @scan_backoff_ms@ | The backoff time (in milliseconds) to retry scans after failures. Default value is 1s (1000ms). Default is 1s. |
+| @num_readahead_cache_threads@ | Sets the number of threads to be used for the read-ahead mechanism. Default is the number of cores as returned with a call to <code>Runtime.getRuntime().availableProcessors()</code>.|
+
+h3. Publish and subscription parameters
+
+| @max_message_size@ | Sets the maximum message size. Default is 1.2 megabytes. |
+| @default_message_window_size@ | This parameter is used for setting the default maximum number of messages that can be delivered to a subscriber without being consumed. We pause delivery to a subscriber when reaching the window size. Default is unlimited (0). |
+| @consume_interval@ | Sets the number of messages consumed before persisting information about consumed messages. A value greater than one avoids persisting information about consumed messages upon every consumed message. Default is 50.|
+| @retention_secs@ | the interval to release a topic. If this parameter is greater than zero, then schedule a task to release an owned topic. Default is 0 (never released).
+| @messages_consumed_thread_run_interval@ | Time interval (in milliseconds) to run messages consumed timer task to
+delete those consumed ledgers in BookKeeper. Default is 1 minute (60,000 ms). |
+
+
+h3. ZooKeeper parameters
+
+| @zk_host@ | Sets the ZooKeeper list of servers. Default is localhost:2181. |
+| @zk_timeout@ | Sets the ZooKeeper session timeout. Default is 2s. |
+
+h3. BookKeeper parameters
+
+| @bk_ensemble_size@ | Sets the ensemble size. Default is 3. |
+| @bk_write_quorum_size@ | Sets the write quorum size. Default is 2. |
+| @bk_ack_quorum_size@ | Sets the ack quorum size. Default is 2. |
+
+bq. Note that the ack quorum size must be equal or smaller than the write quorum size.
+
+| @max_entries_per_ledger@ | Maximum number of entries before we roll a ledger. Default is unlimited (0). |
+
+h3. Metadata parameters
+
+| @zk_prefix@ | Sets the ZooKeeper path prefix. Default is _/hedwig_. |
+| @metadata_manager_based_topic_manager_enabled@ | Enables the use of a metadata manager for topic management. Default is false. |
+| @metadata_manager_factory_class@ | Sets the default factory for the metadata manager. Default is null. |
+
+h2. Region manager configuration parameters
+
+Please also refer to the configuration file that comes with the distribution: _hedwig-server/conf/hw_region_client.conf_.
+
+| @ssl_enabled@ | This parameter is a boolean flag indicating if communication with the server should be done via SSL for encryption. The Hedwig server hubs also need to be SSL enabled for this to work. Default value is false. |
+| @max_message_size@ | Sets the maximum message size in bytes. The default value is 2 MB (2097152). |
+| @max_server_redirects@ | Sets the maximum number of redirects we permit before signaling an error. Default value is 2. |
+| @auto_send_consume_message_enabled@ | A flag indicating whether the client library should automatically send consume messages to the server. Default value is true. |
+| @consumed_messages_buffer_size@ | Sets the number of messages we buffer before sending a consume message to the server. Default value is 5. |
+| @max_outstanding_messages@ | Support for client side throttling, sets the maximum number of outstanding messages. Default value is 10. |
+| @server_ack_response_timeout@ | Sets the timeout (in milliseconds) before we error out any existing requests. Default value is 30s (30,000). |
+
Modified: zookeeper/bookkeeper/trunk/doc/index.textile
URL: http://svn.apache.org/viewvc/zookeeper/bookkeeper/trunk/doc/index.textile?rev=1420069&r1=1420068&r2=1420069&view=diff
==============================================================================
--- zookeeper/bookkeeper/trunk/doc/index.textile (original)
+++ zookeeper/bookkeeper/trunk/doc/index.textile Tue Dec 11 10:15:19 2012
@@ -37,6 +37,7 @@ h1. Apache Hedwig documentation
* "Building Hedwig, or how to set up Hedwig":./hedwigBuild.html
* "User's Guide, or how to program against the Hedwig API and how to run it":./hedwigUser.html
* "Developer's Guide, or Hedwig internals and hacking details":./hedwigDesign.html
+* "Configuration parameters":./hedwigParams.html
* "Message Filtering":./hedwigMessageFilter.html
h2. Hedwig Admin & Ops
Modified: zookeeper/bookkeeper/trunk/hedwig-server/conf/hw_server.conf
URL: http://svn.apache.org/viewvc/zookeeper/bookkeeper/trunk/hedwig-server/conf/hw_server.conf?rev=1420069&r1=1420068&r2=1420069&view=diff
==============================================================================
--- zookeeper/bookkeeper/trunk/hedwig-server/conf/hw_server.conf (original)
+++ zookeeper/bookkeeper/trunk/hedwig-server/conf/hw_server.conf Tue Dec 11 10:15:19 2012
@@ -51,6 +51,9 @@ ssl_enabled=false
# Password used for pkcs12 certificate.
# password=
+#######################################
+# Publish and subscription parameters
+#######################################
# Max Message Size that a hub server could accept
# max_message_size=1258291
@@ -124,6 +127,11 @@ ssl_enabled=false
# Default value is 1000.
# scan_backoff_ms=1000
+# Sets the number of threads to be used for the read-ahead mechanism.
+# Default is the number of cores as returned with a call to
+# <code>Runtime.getRuntime().availableProcessors()</code>.
+# num_readahead_cache_threads=
+
################################
# Metadata Settings
################################
@@ -146,5 +154,8 @@ ssl_enabled=false
# Ensemble size of a ledger in BookKeeper. Default is 3.
# bk_ensemble_size=3
-# Quorum size of a ledger in BookKeeper. Default is 2.
-# bk_quorum_size=2
+# Write quorum size for a ledger in BookKeeper. Default is 2.
+# bk_write_quorum_size=2
+
+# Ack quorum size for a ledger in BookKeeper. Default is 2.
+# bk_ack_quorum_size=2
Modified: zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/common/ServerConfiguration.java
URL: http://svn.apache.org/viewvc/zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/common/ServerConfiguration.java?rev=1420069&r1=1420068&r2=1420069&view=diff
==============================================================================
--- zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/common/ServerConfiguration.java (original)
+++ zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/common/ServerConfiguration.java Tue Dec 11 10:15:19 2012
@@ -141,32 +141,67 @@ public class ServerConfiguration extends
return myRegionByteString;
}
+ /**
+ * Maximum number of messages to read ahead. Default is 10.
+ *
+ * @return int
+ */
public int getReadAheadCount() {
return conf.getInt(READAHEAD_COUNT, 10);
}
+ /**
+ * Maximum number of bytes to read ahead. Default is 4MB.
+ *
+ * @return long
+ */
public long getReadAheadSizeBytes() {
return conf.getLong(READAHEAD_SIZE, 4 * 1024 * 1024); // 4M
}
+ /**
+ * Maximum cache size. By default is the smallest of 2G or
+ * half the heap size.
+ *
+ * @return long
+ */
public long getMaximumCacheSize() {
// 2G or half of the maximum amount of memory the JVM uses
return conf.getLong(CACHE_SIZE, Math.min(2 * 1024L * 1024L * 1024L, Runtime.getRuntime().maxMemory() / 2));
}
- // After a scan of a log fails, how long before we retry (in msec)
+ /**
+ * After a scan of a log fails, how long before we retry (in msec)
+ *
+ * @return long
+ */
public long getScanBackoffPeriodMs() {
return conf.getLong(SCAN_BACKOFF_MSEC, 1000);
}
-
+
+ /**
+ * Returns server port.
+ *
+ * @return int
+ */
public int getServerPort() {
return conf.getInt(SERVER_PORT, 4080);
}
+ /**
+ * Returns SSL server port.
+ *
+ * @return int
+ */
public int getSSLServerPort() {
return conf.getInt(SSL_SERVER_PORT, 9876);
}
+ /**
+ * Returns ZooKeeper path prefix.
+ *
+ * @return string
+ */
public String getZkPrefix() {
return conf.getString(ZK_PREFIX, "/hedwig");
}
@@ -205,24 +240,49 @@ public class ServerConfiguration extends
return myServerAddress;
}
+ /**
+ * Return ZooKeeper list of servers. Default is localhost.
+ *
+ * @return String
+ */
public String getZkHost() {
return conf.getString(ZK_HOST, "localhost");
}
+ /**
+ * Return ZooKeeper session timeout. Default is 2s.
+ *
+ * @return int
+ */
public int getZkTimeout() {
return conf.getInt(ZK_TIMEOUT, 2000);
}
+ /**
+ * Returns true if read-ahead enabled. Default is true.
+ *
+ * @return boolean
+ */
public boolean getReadAheadEnabled() {
return conf.getBoolean(READAHEAD_ENABLED, true)
|| conf.getBoolean("readhead_enabled");
// the key was misspelt in a previous version, so compensate here
}
+ /**
+ * Returns true if standalone. Default is false.
+ *
+ * @return boolean
+ */
public boolean isStandalone() {
return conf.getBoolean(STANDALONE, false);
}
+ /**
+ * Returns list of regions.
+ *
+ * @return List<String>
+ */
public List<String> getRegions() {
if (regionList == null) {
refreshRegionList();
@@ -230,12 +290,20 @@ public class ServerConfiguration extends
return regionList;
}
- // This is the name of the SSL certificate if available as a resource.
+ /**
+ * Returns the name of the SSL certificate if available as a resource.
+ *
+ * @return String
+ */
public String getCertName() {
return conf.getString(CERT_NAME, "");
}
- // This is the path to the SSL certificate if it is available as a file.
+ /**
+ * This is the path to the SSL certificate if it is available as a file.
+ *
+ * @return String
+ */
public String getCertPath() {
return conf.getString(CERT_PATH, "");
}
@@ -255,57 +323,110 @@ public class ServerConfiguration extends
throw new ConfigurationException("SSL Certificate configuration does not have resource name or path set!");
}
+ /**
+ * Returns the password used for BookKeeper ledgers. Default
+ * is the empty string.
+ *
+ * @return
+ */
public String getPassword() {
return conf.getString(PASSWORD, "");
}
+ /**
+ * Returns true if SSL is enabled. Default is false.
+ *
+ * @return boolean
+ */
public boolean isSSLEnabled() {
return conf.getBoolean(SSL_ENABLED, false);
}
+ /**
+ * Gets the number of messages consumed before persisting
+ * information about consumed messages. A value greater than
+ * one avoids persisting information about consumed messages
+ * upon every consumed message. Default is 50.
+ *
+ * @return int
+ */
public int getConsumeInterval() {
return conf.getInt(CONSUME_INTERVAL, 50);
}
+ /**
+ * Returns the interval to release a topic. If this
+ * parameter is greater than zero, then schedule a
+ * task to release an owned topic. Default is 0 (never released).
+ *
+ * @return int
+ */
public int getRetentionSecs() {
return conf.getInt(RETENTION_SECS, 0);
}
+ /**
+ * True if SSL is enabled across regions.
+ *
+ * @return boolean
+ */
public boolean isInterRegionSSLEnabled() {
return conf.getBoolean(INTER_REGION_SSL_ENABLED, false);
}
- // This parameter is used to determine how often we run the
- // SubscriptionManager's Messages Consumed timer task thread (in
- // milliseconds).
+ /**
+ * This parameter is used to determine how often we run the
+ * SubscriptionManager's Messages Consumed timer task thread
+ * (in milliseconds).
+ *
+ * @return int
+ */
public int getMessagesConsumedThreadRunInterval() {
return conf.getInt(MESSAGES_CONSUMED_THREAD_RUN_INTERVAL, 60000);
}
- // This parameter is used to determine how often we run a thread
- // to retry those failed remote subscriptions in asynchronous mode
- // (in milliseconds).
+ /**
+ * This parameter is used to determine how often we run a thread
+ * to retry those failed remote subscriptions in asynchronous mode
+ * (in milliseconds).
+ *
+ * @return int
+ */
public int getRetryRemoteSubscribeThreadRunInterval() {
return conf.getInt(RETRY_REMOTE_SUBSCRIBE_THREAD_RUN_INTERVAL, 120000);
}
- // This parameter is for setting the default maximum number of messages which
- // can be delivered to a subscriber without being consumed.
- // we pause messages delivery to a subscriber when reaching the window size
+ /**
+ * This parameter is for setting the default maximum number of messages which
+ * can be delivered to a subscriber without being consumed.
+ * we pause messages delivery to a subscriber when reaching the window size
+ *
+ * @return int
+ */
public int getDefaultMessageWindowSize() {
return conf.getInt(DEFAULT_MESSAGE_WINDOW_SIZE, 0);
}
- // This parameter is used when Bookkeeper is the persistence store
- // and indicates what the ensemble size is (i.e. how many bookie
- // servers to stripe the ledger entries across).
+ /**
+ * This parameter is used when Bookkeeper is the persistence
+ * store and indicates what the ensemble size is (i.e. how
+ * many bookie servers to stripe the ledger entries across).
+ *
+ * @return int
+ */
public int getBkEnsembleSize() {
return conf.getInt(BK_ENSEMBLE_SIZE, 3);
}
- // This parameter is used when Bookkeeper is the persistence store
- // and indicates what the quorum size is (i.e. how many redundant
- // copies of each ledger entry is written).
+
+ /**
+ * This parameter is used when Bookkeeper is the persistence store
+ * and indicates what the quorum size is (i.e. how many redundant
+ * copies of each ledger entry is written).
+ *
+ * @return int
+ */
+ @Deprecated
protected int getBkQuorumSize() {
return conf.getInt(BK_QUORUM_SIZE, 2);
}