You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Boyang Jerry Peng (JIRA)" <ji...@apache.org> on 2015/10/16 21:29:05 UTC
[jira] [Updated] (STORM-1111) Fix Validation for lots of different
configs
[ https://issues.apache.org/jira/browse/STORM-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boyang Jerry Peng updated STORM-1111:
-------------------------------------
Description:
Once https://github.com/apache/storm/pull/785 goes in the validation logic is more obvious about what is happening, and we have a lot of configs that the validation is incomplete. We should look at all of the configs and update the validation logic + comments to show what can be stored in these configs, and that we validate them correctly. The following is an incomplete list of some of these changes that need to be made.
```
TOPOLOGY_ISOLATED_MACHINES needs @isPositiveNumber and @isInteger
All of the 'ZMQ_` configs should be deprecated.
TRANSACTIONAL_ZOOKEEPER_PORTneeds @isPositiveNumber and @isInteger
It would be great if we could restrict TOPOLOGY_LOGGING_SENSITIVITY to one of the allowed values "S0", "S1", "S2", "S3"
TOPOLOGY_SHELLBOLT_MAX_PENDING needs @isPositiveNumber
TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS needs @isPositiveNumber
TOPOLOGY_MAX_ERROR_REPORT_PER_INTERVAL and TOPOLOGY_ERROR_THROTTLE_INTERVAL_SECS both seem to need @isPositiveNumber
TOPOLOGY_TRANSFER_BUFFER_SIZE needs to be @isPowerOf2
TOPOLOGY_ENVIRONMENT should be @isMapEntryType(keyType = String.class, valueType = String.class).
TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS needs @isPositiveNumber(includeZero = true)
TOPOLOGY_MAX_SPOUT_PENDING needs @isPositiveNumber
TOPOLOGY_MAX_TASK_PARALLELISM needs @isPositiveNumber
WORKER_METRICS and TOPOLOGY_WORKER_METRICS should be @isMapEntryType(keyType = String.class, valueType = String.class).
TOPOLOGY_METRICS_CONSUMER_REGISTER should have a custom validator (you might not have time to do it, so we might need a follow on JIRA for this). Something like
@isListEntryCustom(entryValidatorClasses={MetricRegistryValidator.class})
MetricRegistryValidator.class needs to check that it is a map, with a "class" key that points to a string, a "parallelism.hint" key that points to a positive non-null integer.
TOPOLOGY_EVENTLOGGER_EXECUTORS needs @isPositiveNumber
TOPOLOGY_ACKER_EXECUTORS needs @isPositiveNumber
TOPOLOGY_TASKS and TOPOLOGY_WORKERS need @isPositiveNumber
TASK_CREDENTIALS_POLL_SECS needs @isPositiveNumber
TASK_REFRESH_POLL_SECS TASK_HEARTBEAT_FREQUENCY_SECS WORKER_HEARTBEAT_FREQUENCY_SECS and WORKER_RECEIVER_THREAD_COUNT need @isPositiveNumber
SUPERVISOR_MONITOR_FREQUENCY_SECS and SUPERVISOR_HEARTBEAT_FREQUENCY_SECS need @isPositiveNumber
SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS needs @isPositiveNumber
DRPC_HTTP_FILTER_PARAMS should be @isMapEntryType(keyType = String.class, valueType = String.class).
DRPC_INVOCATIONS_THREADS and DRPC_INVOCATIONS_PORT need @isPositiveNumber
DRPC_QUEUE_SIZE DRPC_MAX_BUFFER_SIZE and DRPC_WORKER_THREADS need @isPositiveNumber
DRPC_AUTHORIZER_ACL needs to be a Map>>. This too probably needs a custom validator in a follow on JIRA.
DRPC_PORT needs @isPositiveNumber
DRPC_HTTPS_PORT and DRPC_HTTP_PORT need @isPositiveNumber
UI_HTTPS_PORT and UI_HEADER_BUFFER_BYTES need @isPositiveNumber
UI_FILTER_PARAMS should be @isMapEntryType(keyType = String.class, valueType = String.class).
LOGVIEWER_HTTPS_PORT needs @isPositiveNumber
LOGVIEWER_PORT and UI_PORT need @isPositiveNumber
NIMBUS_CREDENTIAL_RENEW_FREQ_SECS needs @isPositiveNumber
NIMBUS_IMPERSONATION_ACL needs to be updated, because I don't think Map of string to map. It is more complex then that.
NIMBUS_TASK_LAUNCH_SECS NIMBUS_SUPERVISOR_TIMEOUT_SECS NIMBUS_INBOX_JAR_EXPIRATION_SECS NIMBUS_CLEANUP_INBOX_FREQ_SECS NIMBUS_MONITOR_FREQ_SECS and NIMBUS_TASK_TIMEOUT_SECS need @isPositiveNumber
NIMBUS_THRIFT_MAX_BUFFER_SIZE needs @isPositiveNumber
NIMBUS_THRIFT_THREADS and NIMBUS_THRIFT_PORT need @isPositiveNumber
```
was:
Once https://github.com/apache/storm/pull/785 goes in the validation logic is more obvious about what is happening, and we have a lot of configs that the validation is incomplete. We should look at all of the configs and update the validation logic + comments to show what can be stored in these configs, and that we validate them correctly. The following is an incomplete list of some of these changes that need to be made.
```
TOPOLOGY_ISOLATED_MACHINES needs @isPositiveNumber
All of the 'ZMQ_` configs should be deprecated.
TRANSACTIONAL_ZOOKEEPER_PORTneeds @isPositiveNumber
It would be great if we could restrict TOPOLOGY_LOGGING_SENSITIVITY to one of the allowed values "S0", "S1", "S2", "S3"
TOPOLOGY_SHELLBOLT_MAX_PENDING needs @isPositiveNumber
TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS needs @isPositiveNumber
TOPOLOGY_MAX_ERROR_REPORT_PER_INTERVAL and TOPOLOGY_ERROR_THROTTLE_INTERVAL_SECS both seem to need @isPositiveNumber
TOPOLOGY_TRANSFER_BUFFER_SIZE needs to be @isPowerOf2
TOPOLOGY_ENVIRONMENT should be @isMapEntryType(keyType = String.class, valueType = String.class).
TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS needs @isPositiveNumber(includeZero = true)
TOPOLOGY_MAX_SPOUT_PENDING needs @isPositiveNumber
TOPOLOGY_MAX_TASK_PARALLELISM needs @isPositiveNumber
WORKER_METRICS and TOPOLOGY_WORKER_METRICS should be @isMapEntryType(keyType = String.class, valueType = String.class).
TOPOLOGY_METRICS_CONSUMER_REGISTER should have a custom validator (you might not have time to do it, so we might need a follow on JIRA for this). Something like
@isListEntryCustom(entryValidatorClasses={MetricRegistryValidator.class})
MetricRegistryValidator.class needs to check that it is a map, with a "class" key that points to a string, a "parallelism.hint" key that points to a positive non-null integer.
TOPOLOGY_EVENTLOGGER_EXECUTORS needs @isPositiveNumber
TOPOLOGY_ACKER_EXECUTORS needs @isPositiveNumber
TOPOLOGY_TASKS and TOPOLOGY_WORKERS need @isPositiveNumber
TASK_CREDENTIALS_POLL_SECS needs @isPositiveNumber
TASK_REFRESH_POLL_SECS TASK_HEARTBEAT_FREQUENCY_SECS WORKER_HEARTBEAT_FREQUENCY_SECS and WORKER_RECEIVER_THREAD_COUNT need @isPositiveNumber
SUPERVISOR_MONITOR_FREQUENCY_SECS and SUPERVISOR_HEARTBEAT_FREQUENCY_SECS need @isPositiveNumber
SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS needs @isPositiveNumber
DRPC_HTTP_FILTER_PARAMS should be @isMapEntryType(keyType = String.class, valueType = String.class).
DRPC_INVOCATIONS_THREADS and DRPC_INVOCATIONS_PORT need @isPositiveNumber
DRPC_QUEUE_SIZE DRPC_MAX_BUFFER_SIZE and DRPC_WORKER_THREADS need @isPositiveNumber
DRPC_AUTHORIZER_ACL needs to be a Map>>. This too probably needs a custom validator in a follow on JIRA.
DRPC_PORT needs @isPositiveNumber
DRPC_HTTPS_PORT and DRPC_HTTP_PORT need @isPositiveNumber
UI_HTTPS_PORT and UI_HEADER_BUFFER_BYTES need @isPositiveNumber
UI_FILTER_PARAMS should be @isMapEntryType(keyType = String.class, valueType = String.class).
LOGVIEWER_HTTPS_PORT needs @isPositiveNumber
LOGVIEWER_PORT and UI_PORT need @isPositiveNumber
NIMBUS_CREDENTIAL_RENEW_FREQ_SECS needs @isPositiveNumber
NIMBUS_IMPERSONATION_ACL needs to be updated, because I don't think Map of string to map. It is more complex then that.
NIMBUS_TASK_LAUNCH_SECS NIMBUS_SUPERVISOR_TIMEOUT_SECS NIMBUS_INBOX_JAR_EXPIRATION_SECS NIMBUS_CLEANUP_INBOX_FREQ_SECS NIMBUS_MONITOR_FREQ_SECS and NIMBUS_TASK_TIMEOUT_SECS need @isPositiveNumber
NIMBUS_THRIFT_MAX_BUFFER_SIZE needs @isPositiveNumber
NIMBUS_THRIFT_THREADS and NIMBUS_THRIFT_PORT need @isPositiveNumber
```
> Fix Validation for lots of different configs
> --------------------------------------------
>
> Key: STORM-1111
> URL: https://issues.apache.org/jira/browse/STORM-1111
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Robert Joseph Evans
> Assignee: Boyang Jerry Peng
>
> Once https://github.com/apache/storm/pull/785 goes in the validation logic is more obvious about what is happening, and we have a lot of configs that the validation is incomplete. We should look at all of the configs and update the validation logic + comments to show what can be stored in these configs, and that we validate them correctly. The following is an incomplete list of some of these changes that need to be made.
> ```
> TOPOLOGY_ISOLATED_MACHINES needs @isPositiveNumber and @isInteger
> All of the 'ZMQ_` configs should be deprecated.
> TRANSACTIONAL_ZOOKEEPER_PORTneeds @isPositiveNumber and @isInteger
> It would be great if we could restrict TOPOLOGY_LOGGING_SENSITIVITY to one of the allowed values "S0", "S1", "S2", "S3"
> TOPOLOGY_SHELLBOLT_MAX_PENDING needs @isPositiveNumber
> TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS needs @isPositiveNumber
> TOPOLOGY_MAX_ERROR_REPORT_PER_INTERVAL and TOPOLOGY_ERROR_THROTTLE_INTERVAL_SECS both seem to need @isPositiveNumber
> TOPOLOGY_TRANSFER_BUFFER_SIZE needs to be @isPowerOf2
> TOPOLOGY_ENVIRONMENT should be @isMapEntryType(keyType = String.class, valueType = String.class).
> TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS needs @isPositiveNumber(includeZero = true)
> TOPOLOGY_MAX_SPOUT_PENDING needs @isPositiveNumber
> TOPOLOGY_MAX_TASK_PARALLELISM needs @isPositiveNumber
> WORKER_METRICS and TOPOLOGY_WORKER_METRICS should be @isMapEntryType(keyType = String.class, valueType = String.class).
> TOPOLOGY_METRICS_CONSUMER_REGISTER should have a custom validator (you might not have time to do it, so we might need a follow on JIRA for this). Something like
> @isListEntryCustom(entryValidatorClasses={MetricRegistryValidator.class})
> MetricRegistryValidator.class needs to check that it is a map, with a "class" key that points to a string, a "parallelism.hint" key that points to a positive non-null integer.
> TOPOLOGY_EVENTLOGGER_EXECUTORS needs @isPositiveNumber
> TOPOLOGY_ACKER_EXECUTORS needs @isPositiveNumber
> TOPOLOGY_TASKS and TOPOLOGY_WORKERS need @isPositiveNumber
> TASK_CREDENTIALS_POLL_SECS needs @isPositiveNumber
> TASK_REFRESH_POLL_SECS TASK_HEARTBEAT_FREQUENCY_SECS WORKER_HEARTBEAT_FREQUENCY_SECS and WORKER_RECEIVER_THREAD_COUNT need @isPositiveNumber
> SUPERVISOR_MONITOR_FREQUENCY_SECS and SUPERVISOR_HEARTBEAT_FREQUENCY_SECS need @isPositiveNumber
> SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS needs @isPositiveNumber
> DRPC_HTTP_FILTER_PARAMS should be @isMapEntryType(keyType = String.class, valueType = String.class).
> DRPC_INVOCATIONS_THREADS and DRPC_INVOCATIONS_PORT need @isPositiveNumber
> DRPC_QUEUE_SIZE DRPC_MAX_BUFFER_SIZE and DRPC_WORKER_THREADS need @isPositiveNumber
> DRPC_AUTHORIZER_ACL needs to be a Map>>. This too probably needs a custom validator in a follow on JIRA.
> DRPC_PORT needs @isPositiveNumber
> DRPC_HTTPS_PORT and DRPC_HTTP_PORT need @isPositiveNumber
> UI_HTTPS_PORT and UI_HEADER_BUFFER_BYTES need @isPositiveNumber
> UI_FILTER_PARAMS should be @isMapEntryType(keyType = String.class, valueType = String.class).
> LOGVIEWER_HTTPS_PORT needs @isPositiveNumber
> LOGVIEWER_PORT and UI_PORT need @isPositiveNumber
> NIMBUS_CREDENTIAL_RENEW_FREQ_SECS needs @isPositiveNumber
> NIMBUS_IMPERSONATION_ACL needs to be updated, because I don't think Map of string to map. It is more complex then that.
> NIMBUS_TASK_LAUNCH_SECS NIMBUS_SUPERVISOR_TIMEOUT_SECS NIMBUS_INBOX_JAR_EXPIRATION_SECS NIMBUS_CLEANUP_INBOX_FREQ_SECS NIMBUS_MONITOR_FREQ_SECS and NIMBUS_TASK_TIMEOUT_SECS need @isPositiveNumber
> NIMBUS_THRIFT_MAX_BUFFER_SIZE needs @isPositiveNumber
> NIMBUS_THRIFT_THREADS and NIMBUS_THRIFT_PORT need @isPositiveNumber
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)