You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/11/28 17:43:00 UTC
[jira] [Commented] (METRON-1336) Patching Can Result in Bad
Configuration
[ https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269129#comment-16269129 ]
ASF GitHub Bot commented on METRON-1336:
----------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/851
METRON-1336 Patching Can Result in Bad Configuration
The following problems are addressed in this PR.
* A patch can be constructed that when applied creates an invalid configuration. The invalid configuration is not discovered until attempting to "pull" it back out of Zookeeper. In all cases, the result of applying a patch needs to be validated before pushing it to Zookeeper.
* The Profiler configuration can only be pushed when pushing all configurations at once. Attempting to push just the Profiler configuration alone, fails silently.
* Added unit tests to validate that a 'bad' patch cannot be applied.
* Added a good number of unit tests to ensure invalid configurations cannot be pushed under all circumstances (parser, enrichment, indexing, profiler, "push all").
* Added a test to ensure that the Profiler configuration can be pushed independently.
## Manual Testing
1. Launch Full Dev.
1. Environment definitions that will be used later.
```
export METRON_HOME=/usr/metron/0.4.2
export ZOOKEEPER=node1:2181
```
1. Dump all existing configurations.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP
```
1. Dump just the Profiler configuration. There should be none.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
```
1. Create a Profiler configuration on disk.
```
$ cat $METRON_HOME/config/profiler.json
{
"profiles": [
{
"profile": "hello-world",
"onlyif": "exists(ip_src_addr)",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
```
1. Push the Profiler configuration.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c PROFILER -i $METRON_HOME/config
```
1. Dump just the Profiler configuration. It should be defined now.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
```
1. Create a patch for the Profiler configuration.
```
$ cat profile.patch
[
{
"op": "add",
"path": "/profiles/0/profile",
"value": "changed-profile-name"
}
]
```
1. Apply the patch.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c PROFILER -pf profile.patch
```
1. Dump the Profiler configuration and ensure that the name of the Profile was changed.
```
[root@node1 ~]# $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
PROFILER Config: profiler
{
"profiles" : [ {
"profile" : "changed-profile-name",
"onlyif" : "exists(ip_src_addr)",
"foreach" : "ip_src_addr",
"init" : {
"count" : "0"
},
"update" : {
"count" : "count + 1"
},
"result" : "count"
} ]
}
```
1. Create a patch that would make the Profiler configuration invalid.
```
[root@node1 ~]# cat bad.patch
[
{
"op": "add",
"path": "/profiles/0/invalid",
"value": "22"
}
]
```
1. Apply the bad patch. The script should terminate with an error.
```
[root@node1 ~]# $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c PROFILER -pf bad.patch
2017-11-28 17:37:17 ERROR ConfigurationManager:314 - Unable to apply patch to Zookeeper config
java.lang.RuntimeException: Unable to load {
"profiles" : [ {
"profile" : "changed-profile-name",
"onlyif" : "exists(ip_src_addr)",
"foreach" : "ip_src_addr",
"init" : {
"count" : "0"
},
"update" : {
"count" : "count + 1"
},
"result" : "count",
"invalid" : "22"
} ]
}
at org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
at org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
at org.apache.metron.common.configuration.ConfigurationsUtils.applyConfigPatchToZookeeper(ConfigurationsUtils.java:621)
at org.apache.metron.common.cli.ConfigurationManager.patch(ConfigurationManager.java:307)
at org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:285)
at org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:244)
at org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:360)
Caused by: org.apache.metron.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "invalid" (class org.apache.metron.common.configuration.profiler.ProfileConfig), not marked as ignorable (8 known properties: "init", "expires", "groupBy", "profile", "foreach", "result", "onlyif", "update"])
at [Source: {
"profiles" : [ {
"profile" : "changed-profile-name",
"onlyif" : "exists(ip_src_addr)",
"foreach" : "ip_src_addr",
"init" : {
"count" : "0"
},
"update" : {
"count" : "count + 1"
},
"result" : "count",
"invalid" : "22"
} ]
...
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1336
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/851.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #851
----
----
> Patching Can Result in Bad Configuration
> ----------------------------------------
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can be invalid. The resulting configuration should be validated so that a patch can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the Profiler yields a broken Profiler configuration.
> {code}
> [
> {
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global', PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
> }
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global', PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global', PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain: org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
> at org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
> at org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249)
> at org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
> at org.apache.metron.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
> at org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
> at org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> at org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> at org.apache.metron.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3807)
> at org.apache.metron.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2797)
> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:79)
> at org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:66)
> ... 8 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)