You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/01/12 14:32:30 UTC
[GitHub] incubator-metron pull request #415: METRON-652: Extract indexing config from...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/415
METRON-652: Extract indexing config from enrichment config
Currently, the indexing configuration is bound and coupled with the sensor enrichment configuration. This was done historically because indexing and enrichment were part of the same topology. When the topologies were separated, the configurations were never separated, leaving a confusing section about indexes in the middle of the enrichment configuration. This effort will separate out the configuration.
Because of the configuration's simplicity, we are treating the config as just a `Map<String, Object>` and supporting the two existing configurations:
* `batchSize`
* `index`
Because this change is fairly deep, a few non-obvious things were also updated for consistency and necessity:
* The Stellar Management functions `ENRICHMENT_SET_BATCH` and `ENRICHMENT_SET_INDEX` functions have been renamed to `INDEXING_SET_BATCH` and `INDEXING_SET_INDEX`
* Support in `CONFIG_GET` and `CONFIG_PUT` for retrieving/storing indexing configurations was added.
* Support for indexing configurations were added to the zookeeper utility function
* The indexing RPM was updated to include the new configs for the shipped sensors.
I have *not* updated the management pack to create a new panel for managing indexing configs. That will be part of a follow-on JIRA (METRON-653).
I have done smoke testing on quickdev, but will include more comprehensive testing explanation in the comments. I feel that this has the possibility of causing regressions, so I want to be explicit in testing methodology.
Furthermore, I plan on creating a wiki section about about migrations, because this will necessitate migrating configs when upgrading from 0.3.0 to the next version.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-652
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/415.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #415
----
commit 1d03bc57235af2c1b90c901e7895f3c037a054c2
Author: cstella <ce...@gmail.com>
Date: 2017-01-10T14:25:33Z
initial update
commit 4c9227cb618a62bfea63b147c32054bfeadf7d08
Author: cstella <ce...@gmail.com>
Date: 2017-01-10T15:48:33Z
Updating testing components.
commit 3681edb32dca2876ecb2c0895c58a392ce558ada
Author: cstella <ce...@gmail.com>
Date: 2017-01-10T20:32:47Z
second round of changes.
commit facb6518b27fd19fd5c3f40671b57b84013c4076
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T15:03:14Z
Merge branch 'master' into METRON-652
commit ec91c279af078c94c4d762bd8ed166a18fa78da3
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T15:03:40Z
Merge branch 'master' into METRON-652
commit ed98341e904575f4d1e89533440cb920ee7a6e3d
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T17:54:59Z
updating.
commit 375e0260ce2d12e38e66e8ca05740bf6f71e348c
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T21:10:16Z
Updating documentation.
commit 3736294bfe9bb4f3f6c3862af300da49960da81f
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T21:15:24Z
Updating RPM.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #415: METRON-652: Extract indexing config from...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/415#discussion_r96253871
--- Diff: metron-platform/metron-enrichment/README.md ---
@@ -29,6 +29,16 @@ There are two types of configurations at the moment, `global` and
See the "[Global Configuration](../metron-common)" section.
+##Sensor Indexing Configuration
--- End diff --
Should this be in the metron-indexing README.md?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #415: METRON-652: Extract indexing config from...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/415
METRON-652: Extract indexing config from enrichment config
Currently, the indexing configuration is bound and coupled with the sensor enrichment configuration. This was done historically because indexing and enrichment were part of the same topology. When the topologies were separated, the configurations were never separated, leaving a confusing section about indexes in the middle of the enrichment configuration. This effort will separate out the configuration.
Because of the configuration's simplicity, we are treating the config as just a `Map<String, Object>` and supporting the two existing configurations:
* `batchSize`
* `index`
Because this change is fairly deep, a few non-obvious things were also updated for consistency and necessity:
* The Stellar Management functions `ENRICHMENT_SET_BATCH` and `ENRICHMENT_SET_INDEX` functions have been renamed to `INDEXING_SET_BATCH` and `INDEXING_SET_INDEX`
* Support in `CONFIG_GET` and `CONFIG_PUT` for retrieving/storing indexing configurations was added.
* Support for indexing configurations were added to the zookeeper utility function
* The indexing RPM was updated to include the new configs for the shipped sensors.
I have *not* updated the management pack to create a new panel for managing indexing configs. That will be part of a follow-on JIRA (METRON-653).
I have done smoke testing on quickdev, but will include more comprehensive testing explanation in the comments. I feel that this has the possibility of causing regressions, so I want to be explicit in testing methodology.
Furthermore, I plan on creating a wiki section about about migrations, because this will necessitate migrating configs when upgrading from 0.3.0 to the next version.
EDIT: Please review, but do NOT commit this PR until METRON-653 is complete and reviewed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-652
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/415.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #415
----
commit 1d03bc57235af2c1b90c901e7895f3c037a054c2
Author: cstella <ce...@gmail.com>
Date: 2017-01-10T14:25:33Z
initial update
commit 4c9227cb618a62bfea63b147c32054bfeadf7d08
Author: cstella <ce...@gmail.com>
Date: 2017-01-10T15:48:33Z
Updating testing components.
commit 3681edb32dca2876ecb2c0895c58a392ce558ada
Author: cstella <ce...@gmail.com>
Date: 2017-01-10T20:32:47Z
second round of changes.
commit facb6518b27fd19fd5c3f40671b57b84013c4076
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T15:03:14Z
Merge branch 'master' into METRON-652
commit ec91c279af078c94c4d762bd8ed166a18fa78da3
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T15:03:40Z
Merge branch 'master' into METRON-652
commit ed98341e904575f4d1e89533440cb920ee7a6e3d
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T17:54:59Z
updating.
commit 375e0260ce2d12e38e66e8ca05740bf6f71e348c
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T21:10:16Z
Updating documentation.
commit 3736294bfe9bb4f3f6c3862af300da49960da81f
Author: cstella <ce...@gmail.com>
Date: 2017-01-11T21:15:24Z
Updating RPM.
commit 3834178117f8af23ebeff95d69f4eaba965b617c
Author: cstella <ce...@gmail.com>
Date: 2017-01-16T16:11:35Z
Whoops, wrong README.md
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Just a side note: with the rpms and docker everything, it is more difficult than ever to have a handle on all the places you need to consider for changes, I don't know if you want to discuss on the list but there should be a dev. guide entry or something.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Hi @cestella,
Can you give me a little detail about the consequences of not exposing the indexing config to Ambari? It seems like a regression to me, I recall we could deploy non-default configs prior to this PR.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@ottobackwards yeah, agreed. It's a fairly complex situation at the moment and not documented very well. That might be worth a discussion on the dev list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@dlyle65535 I think that's a great point and thanks for clarifying. Looking at that file, it appears that ambari is explicitly providing the ability to modify the global config and certain flux topology properties files, but not a screen to manage the sensor-specific configuration content (i.e. at present parsers and enrichment). Please correct if I have had a reading comprehension SNAFU ;) If so, since we just added a new set of configs under the same config directory and provided hooks for `zk_load_utils.sh` to know how to load and get them, it shouldn't be any different than the existing configs.
That being said, it is a good point you make. Let me restate it and you tell me if I have the gist. If people change the configs on an m-pack installed cluster via the CLI (via `zk_load_utils.sh`) or the soon-to-be-committed management UI, ambari will revert those changes on service restart, correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #415: METRON-652: Extract indexing config from...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/415
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@mmiklavc - doesn't Enrichment Master's config do the same thing for enrichment.properties?
File(format("{metron_config_path}/enrichment.properties"),
content=Template("enrichment.properties.j2"),
owner=params.metron_user,
group=params.metron_group
)`
And Indexing Master handles elasticsearch.properties (and global.json)?
File("{0}/global.json".format(params.metron_zookeeper_config_path),
owner=params.metron_user,
content=InlineTemplate(params.global_json_template)
)
File("{0}/elasticsearch.properties".format(params.metron_zookeeper_config_path + '/..'),
owner=params.metron_user,
content=InlineTemplate(params.global_properties_template))`
That was my intention anyway. Am I mistaken?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Thanks @cestella, makes total sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
## Preliminaries
Since I will use the squid topology to pass data through in a controlled
way, we must install squid and generate one point of data:
* `yum install -y squid`
* `service squid start`
* `squidclient http://www.yahoo.com`
Also, set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.3.0`
## Free Up Space on the virtual machine
First, let's free up some headroom on the virtual machine. If you are running this on a
multinode cluster, you would not have to do this.
* Kill monit via `service monit stop`
* Kill tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print $2}');do kill -9 $i;done`
* Kill existing parser topologies via
* `storm kill snort`
* `storm kill bro`
* Kill flume via `for i in $(ps -ef | grep flume | awk '{print $2}');do kill -9 $i;done`
* Kill yaf via `for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9 $i;done`
* Kill bro via `for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9 $i;done`
## Deploy the squid parser
* Create the squid kafka topic: `/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic squid --partitions 1 --replication-factor 1`
* Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid`
### Test Case 1: Adjusting batch sizes
* Delete any squid index that currently exists (if any do).
* Create a file at `$METRON_HOME/config/zookeeper/indexing/squid.json` with the following contents:
```
{
"index" : "squid",
"batchSize" : 5
}
```
* Send 4 data points through and ensure that there are no data points in the index:
* `cat /var/log/squid/access.log /var/log/squid/access.log /var/log/squid/access.log /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*" 2> /dev/null| grep "full_hostname" | wc -l` should yield `0`
* Send a final data point through and ensure that we have 5 data points:
* `cat /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*" 2> /dev/null| grep "full_hostname" | wc -l` should yield `5`
### Test Case 2: Update configs from the CLI
* Edit the file at `$METRON_HOME/config/zookeeper/indexing/squid.json` to the following contents:
```
{
"index" : "squid",
"batchSize" : 10
}
```
* Push the configs: `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z node1:2181`
* Dump the configs and verify the squid indexing config is correct: `$METRON_HOME/bin/zk_load_configs.sh -m DUMP -z node1:2181`
### Test Case 3: Stellar Management Functions
* Execute the following in the stellar shell:
```
Stellar, Go!
Please note that functions are loading lazily in the background and will
be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300,
es.date.format=yyyy.MM.dd.HH}
[Stellar]>>> # Grab the indexing config
[Stellar]>>> squid_config := CONFIG_GET('INDEXING', 'squid', true)
Functions loaded, you may refer to functions now...
[Stellar]>>> # Update the index and batch size
[Stellar]>>> squid_config := INDEXING_SET_BATCH( INDEXING_SET_INDEX(squid_config, 'squid'), 1)
[Stellar]>>> # Push the config to zookeeper
[Stellar]>>> CONFIG_PUT('INDEXING', squid_config, 'squid')
[Stellar]>>> # Grab the updated config from zookeeper
[Stellar]>>> CONFIG_GET('INDEXING', 'squid')
{
"index" : "squid",
"batchSize" : 1
}
```
* Confirm that the dump command from `$METRON_HOME/bin/zk_load_configs.sh -m DUMP -z node1:2181` contains the config with batch size of `1`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@merrimanr , @cestella - wrt management of configs from Ambari, the answer is kinda. Sensible default configurations are pushed with certain user-specified changes that allow the system to function. A complete list can be found looking at metron-env.xml.
There is an EXTREMELY important side-effect to this that must not be forgotten. Any configuration that has even partial management by Ambari must not be modified outside of Ambari if one expects those changes to survive a service restart. Ambari will detect a change to the file and overwrite it.
So, I don't know for sure, but I suspect this PR will introduce breaking changes to the MPack install which will be corrected during the work on METRON-653.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Isn't this a duplicate of [METRON-470](https://issues.apache.org/jira/browse/METRON-470)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/415
What is the regression we're talking about? Are we talking about loading the indexing configs initially with the Mpack or managing the configs in Ambari?
I don't believe we are currently managing parser or enrichment configs in Ambari and indexing configs would also fall into that category since they are part of enrichment configs now. Or is that incorrect?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/415
OK, found this for the global config in metron-env.xml
```
<property>
<name>global-json</name>
<display-name>global.json template</display-name>
<description>This is the jinja template for global.json file</description>
<value>
{
"es.clustername": "{{ es_cluster_name }}",
"es.ip": "{{ es_url }}",
"es.date.format": "yyyy.MM.dd.HH"
}
</value>
<value-attributes>
<type>content</type>
</value-attributes>
</property>
```
This is referenced in params_linux.py
```
global_json_template = config['configurations']['metron-env']['global-json']
```
And then it's used by metron_service.py to lay down the config using jinja templates (edited for brevity):
```
def init_config():
...
Execute(ambari_format(
"{metron_home}/bin/zk_load_configs.sh --mode PUSH -i {metron_zookeeper_config_path} -z {zookeeper_quorum}"),
path=ambari_format("{java_home}/bin")
)
...
def load_global_config(params):
...
File("{0}/global.json".format(params.metron_zookeeper_config_path),
owner=params.metron_user,
content=InlineTemplate(params.global_json_template)
)
...
init_config()
```
So yes, if you change global.json external to Ambari, ie in the Metron install config directory, Ambari will rewrite what's in the local FS, and follow up with a load to ZK. As best I can tell, this is _only_ applicable to the global config, not the individual topology json configs. Those are unpacked on install via Ambari performing an RPM install, but not actually managed on an ongoing basis. The way you get into hot water here is if you've chosen to manage Ambari configs in a different directory than where Ambari believes they're located, due to it using zk_load_configs.sh under the hood. Even then, this is parameterized via `metron_zookeeper_config_path`. I can't recall if that path is absolute, relative, or both. But in metron-env it's defaulted to `config/zookeeper`
Hope this clarifies some things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@dlyle65535 It definitely is a regression. I didn't make it clear enough (I will do so now), but I very much do not want this PR to be committed before the management pack PR is committed.
The consequences of not managing in ambari is that you won't be able to adjust the default configs, which does not stop data from flowing through, but are sub-optimal (the index name defaults to the sensor name and the batch size defaults to 1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Some yes and some no. I'm a bit bleary-eyed today, so I'm not sure if this is the complete list- here's my current understanding. Ambari actively (will overwrite) manages global.json, enrichment.properties and elasticsearch.properties. It passively (will lay down the default from the rpms) manages the others.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@cestella Yeah, if we do a CONFIG_PUT via the Stellar REPL without updating the local file system config copy, that's going to cause syncing problems.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@dlyle65535 It definitely does that - I was referring to our json config mainly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #415: METRON-652: Extract indexing config from...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/415#discussion_r96261134
--- Diff: metron-platform/metron-enrichment/README.md ---
@@ -29,6 +29,16 @@ There are two types of configurations at the moment, `global` and
See the "[Global Configuration](../metron-common)" section.
+##Sensor Indexing Configuration
--- End diff --
Yes, good catch. Corrected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Ok, thanks @dlyle65535 I don't think it's a regression (since it doesn't appear to be any different behavior-wise between when it existed in enrichment vs in the new configs), but I do think that we should get a broader discussion of ambari management in light of the new management UI that @merrimanr submitted and, for that matter, the stellar REPL's `CONFIG_GET` and `CONFIG_PUT` functions. I'll kick off a dev list discussion to try to figure out what to do about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #415: METRON-652: Extract indexing config from...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/415
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Looks good @cestella . +1 pending Travis
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@merrimanr I was under the impression that there was a panel to manage the enrichment configurations per sensor in ambari, but if that's not the case and it's just initial load, then I don't see a regression. @dlyle65535 does this sound right to you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
@JonZeolla Yes it is! Whoops, my bad. I guess my JIRA search-fu isn't as good as I thought.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---