You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/10/10 06:57:03 UTC
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/795
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts
## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. The REST API, on the other hand, polls zookeeper every time a request is made to retrieve a configuration, a poll model. Rather than do this, it would be better to share an abstraction between the two pieces of infrastructure to enable the REST API to be backed by a TreeCache, which will prevent zookeeper from being abused and migrate it to a push model.
Testing instruction to come.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/795.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #795
----
commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T19:03:32Z
Abstracting zookeeper substantially.
commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:10:57Z
Updating tests to use the new API.
commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:15:51Z
Missed a couple of tests.
commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:32:41Z
moved the cache into commons.
commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T21:15:20Z
Updating test.
commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T23:41:44Z
Refactored tests to mock the configurations cache
commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:13:41Z
fixing tests.
commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:54:57Z
Updating some weird bugs.
commit 24df486a0c0d254f14f298ba8fb09e14152a2bf8
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T01:47:05Z
updating config test.
commit e6d48efd03bd848ca4eefd11c4277b2148550330
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:42:48Z
Updating log4j properties to debug travis.
commit 3e320d63512a860f571ee3fc0a8b41fa98b7558d
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:59:00Z
updating with better logging
commit 09019a52a4c689cdb71b512643c6f381dec114cf
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:17:20Z
reducing logging.
commit 9948df08bd2dc7a11e10bf800fe71427ba482b44
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:41:32Z
Removing debugging
commit 9134b81a5e1d5200e698131b0f66fc3ea015c56b
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:45:41Z
Better error debugging...
commit bbe87c5f1e29cb362f0dac90eb33f537285c1b27
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:47:21Z
Warn level.
commit 105127ed32003f0defdc7a7bed1f7357a69128ab
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:55:34Z
Small error in getTypes()
commit 475853cb72ab891ead4a360ca37c0e53035ba183
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:36:11Z
Updating configurations util
commit a0ce71be196b6c8c068a2f4ee4697af5a4e8d0f1
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:45:35Z
Updating integration tests.
commit c3c3c36e664c9e9216e04c72a06005a37d2a2652
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T06:21:18Z
Fixing tests.
----
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146020850
--- Diff: metron-platform/metron-zookeeper/src/main/java/org/apache/metron/zookeeper/ZKCache.java ---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class ZKCache implements AutoCloseable{
+ private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final int DEFAULT_CLIENT_SLEEP_MS = 1000;
+ public static final int DEFAULT_MAX_RETRIES = 3;
+
+
+
+
+ public static class Builder {
+ private Optional<CuratorFramework> client = Optional.empty();
+ private boolean ownClient = false;
+ private List<TreeCacheListener> listener = new ArrayList<>();
+ private String zkRoot;
+
+ public Builder() { }
+
+ public Builder withClient(CuratorFramework client) {
+ this.client = Optional.ofNullable(client);
+ ownClient = false;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.empty()));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl, RetryPolicy retryPolicy) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.ofNullable(retryPolicy)));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withListener(TreeCacheListener listener) {
+ this.listener.add(listener);
+ return this;
+ }
+
+ public Builder withRoot(String zkRoot) {
+ this.zkRoot = zkRoot;
+ return this;
+ }
+
+ public ZKCache build() {
+ if(!client.isPresent()) {
+ throw new IllegalArgumentException("Zookeeper client must be specified.");
+ }
+ if(listener.isEmpty()) {
+ LOG.warn("Zookeeper listener is null or empty, which is very likely an error.");
+ }
+ if(zkRoot == null) {
+ throw new IllegalArgumentException("Zookeeper root must not be null.");
+ }
+ return new ZKCache(client.get(), listener, zkRoot, ownClient);
+ }
+
+ }
+
+ private CuratorFramework client;
+ private List<TreeCacheListener> listeners;
+ private TreeCache cache;
+ private String zkRoot;
+ private boolean ownClient = false;
+
+ private ZKCache(CuratorFramework client, List<TreeCacheListener> listeners, String zkRoot, boolean ownClient) {
--- End diff --
That helps a lot for ZkCache.
Can you do the same for the other new classes? I think that includes ConfigurationsCache, ConfigurationsUpdater and SimpleEventListener, if I am not mistaken.
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146031435
--- Diff: metron-platform/metron-zookeeper/src/main/java/org/apache/metron/zookeeper/ZKCache.java ---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class ZKCache implements AutoCloseable{
+ private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final int DEFAULT_CLIENT_SLEEP_MS = 1000;
+ public static final int DEFAULT_MAX_RETRIES = 3;
+
+
+
+
+ public static class Builder {
+ private Optional<CuratorFramework> client = Optional.empty();
+ private boolean ownClient = false;
+ private List<TreeCacheListener> listener = new ArrayList<>();
+ private String zkRoot;
+
+ public Builder() { }
+
+ public Builder withClient(CuratorFramework client) {
+ this.client = Optional.ofNullable(client);
+ ownClient = false;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.empty()));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl, RetryPolicy retryPolicy) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.ofNullable(retryPolicy)));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withListener(TreeCacheListener listener) {
+ this.listener.add(listener);
+ return this;
+ }
+
+ public Builder withRoot(String zkRoot) {
+ this.zkRoot = zkRoot;
+ return this;
+ }
+
+ public ZKCache build() {
+ if(!client.isPresent()) {
+ throw new IllegalArgumentException("Zookeeper client must be specified.");
+ }
+ if(listener.isEmpty()) {
+ LOG.warn("Zookeeper listener is null or empty, which is very likely an error.");
+ }
+ if(zkRoot == null) {
+ throw new IllegalArgumentException("Zookeeper root must not be null.");
+ }
+ return new ZKCache(client.get(), listener, zkRoot, ownClient);
+ }
+
+ }
+
+ private CuratorFramework client;
+ private List<TreeCacheListener> listeners;
+ private TreeCache cache;
+ private String zkRoot;
+ private boolean ownClient = false;
+
+ private ZKCache(CuratorFramework client, List<TreeCacheListener> listeners, String zkRoot, boolean ownClient) {
--- End diff --
Sure thing.
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/795
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts
## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. The REST API, on the other hand, polls zookeeper every time a request is made to retrieve a configuration, a poll model. Rather than do this, it would be better to share an abstraction between the two pieces of infrastructure to enable the REST API to be backed by a TreeCache, which will prevent zookeeper from being abused and migrate it to a push model.
Testing instruction to come.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/795.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #795
----
commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T19:03:32Z
Abstracting zookeeper substantially.
commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:10:57Z
Updating tests to use the new API.
commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:15:51Z
Missed a couple of tests.
commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:32:41Z
moved the cache into commons.
commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T21:15:20Z
Updating test.
commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T23:41:44Z
Refactored tests to mock the configurations cache
commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:13:41Z
fixing tests.
commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:54:57Z
Updating some weird bugs.
commit 24df486a0c0d254f14f298ba8fb09e14152a2bf8
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T01:47:05Z
updating config test.
commit e6d48efd03bd848ca4eefd11c4277b2148550330
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:42:48Z
Updating log4j properties to debug travis.
commit 3e320d63512a860f571ee3fc0a8b41fa98b7558d
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:59:00Z
updating with better logging
commit 09019a52a4c689cdb71b512643c6f381dec114cf
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:17:20Z
reducing logging.
commit 9948df08bd2dc7a11e10bf800fe71427ba482b44
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:41:32Z
Removing debugging
commit 9134b81a5e1d5200e698131b0f66fc3ea015c56b
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:45:41Z
Better error debugging...
commit bbe87c5f1e29cb362f0dac90eb33f537285c1b27
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:47:21Z
Warn level.
commit 105127ed32003f0defdc7a7bed1f7357a69128ab
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:55:34Z
Small error in getTypes()
commit 475853cb72ab891ead4a360ca37c0e53035ba183
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:36:11Z
Updating configurations util
commit a0ce71be196b6c8c068a2f4ee4697af5a4e8d0f1
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:45:35Z
Updating integration tests.
commit c3c3c36e664c9e9216e04c72a06005a37d2a2652
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T06:21:18Z
Fixing tests.
commit a13c24a8daaf303704dc1313387b04f415cb236f
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T14:54:59Z
Updating infrastructure to be more spring-friendly.
commit 316eb5e24f6e46cd7d4364532834a460eeeac898
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T15:45:56Z
Fixed test.
commit e476cbaf5600f9428edffb25cd570eea90c8d957
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T16:16:45Z
Updating test to not be so chatty.
commit 0cb5bbcdfe1e17212620c752c200b78fe7ca6106
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T20:48:00Z
Found another classpath bug...
commit 2eb9d69f53cc4897a5112b89d9269dd40544d875
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T22:37:25Z
Refactoring to handle some asynchronicity
commit ddbd409acc19becf0bd5b193dcb602c273dade40
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T13:34:28Z
Asynchronicity leads to some unit test drama. We now need to retry to get the right return code.
commit 1968f392873ab57a3556d3f1bfbf7a57145806af
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T15:04:15Z
Longer time for retries.
----
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/795
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/795
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146020712
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/IndexingUpdater.java ---
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.IndexingConfigurations;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class IndexingUpdater extends ConfigurationsUpdater<IndexingConfigurations> {
+ public IndexingUpdater(Reloadable reloadable, Supplier<IndexingConfigurations> configSupplier) {
+ super(reloadable, configSupplier);
+ }
+
+ @Override
+ public Class<IndexingConfigurations> getConfigurationClass() {
+ return IndexingConfigurations.class;
+ }
+
+ @Override
+ public void forceUpdate(CuratorFramework client) {
+ try {
+ ConfigurationsUtils.updateSensorIndexingConfigsFromZookeeper(getConfigurations(), client);
+ }
+ catch (KeeperException.NoNodeException nne) {
+ LOG.warn("No current indexing configs in zookeeper, but the cache should load lazily...");
+ }
+ catch (Exception e) {
+ LOG.warn("Unable to load indexing configs from zookeeper, but the cache should load lazily...", e);
+ }
+ }
+
+ @Override
+ public IndexingConfigurations defaultConfigurations() {
+ return new IndexingConfigurations();
+ }
+
+ @Override
+ public void delete(CuratorFramework client, String path, byte[] data) throws IOException {
+ String name = path.substring(path.lastIndexOf("/") + 1);
+ if (path.startsWith(ConfigurationType.INDEXING.getZookeeperRoot())) {
--- End diff --
Yeah, I'll do that. It's a good catch, thanks @nickwallen
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/795
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts
## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. The REST API, on the other hand, polls zookeeper every time a request is made to retrieve a configuration, a poll model. Rather than do this, it would be better to share an abstraction between the two pieces of infrastructure to enable the REST API to be backed by a TreeCache, which will prevent zookeeper from being abused and migrate it to a push model.
Testing instruction to come.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/795.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #795
----
commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T19:03:32Z
Abstracting zookeeper substantially.
commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:10:57Z
Updating tests to use the new API.
commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:15:51Z
Missed a couple of tests.
commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:32:41Z
moved the cache into commons.
commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T21:15:20Z
Updating test.
commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T23:41:44Z
Refactored tests to mock the configurations cache
commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:13:41Z
fixing tests.
commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:54:57Z
Updating some weird bugs.
commit 24df486a0c0d254f14f298ba8fb09e14152a2bf8
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T01:47:05Z
updating config test.
commit e6d48efd03bd848ca4eefd11c4277b2148550330
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:42:48Z
Updating log4j properties to debug travis.
commit 3e320d63512a860f571ee3fc0a8b41fa98b7558d
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:59:00Z
updating with better logging
commit 09019a52a4c689cdb71b512643c6f381dec114cf
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:17:20Z
reducing logging.
commit 9948df08bd2dc7a11e10bf800fe71427ba482b44
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:41:32Z
Removing debugging
commit 9134b81a5e1d5200e698131b0f66fc3ea015c56b
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:45:41Z
Better error debugging...
commit bbe87c5f1e29cb362f0dac90eb33f537285c1b27
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:47:21Z
Warn level.
commit 105127ed32003f0defdc7a7bed1f7357a69128ab
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:55:34Z
Small error in getTypes()
commit 475853cb72ab891ead4a360ca37c0e53035ba183
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:36:11Z
Updating configurations util
commit a0ce71be196b6c8c068a2f4ee4697af5a4e8d0f1
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:45:35Z
Updating integration tests.
commit c3c3c36e664c9e9216e04c72a06005a37d2a2652
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T06:21:18Z
Fixing tests.
commit a13c24a8daaf303704dc1313387b04f415cb236f
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T14:54:59Z
Updating infrastructure to be more spring-friendly.
commit 316eb5e24f6e46cd7d4364532834a460eeeac898
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T15:45:56Z
Fixed test.
commit e476cbaf5600f9428edffb25cd570eea90c8d957
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T16:16:45Z
Updating test to not be so chatty.
commit 0cb5bbcdfe1e17212620c752c200b78fe7ca6106
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T20:48:00Z
Found another classpath bug...
commit 2eb9d69f53cc4897a5112b89d9269dd40544d875
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T22:37:25Z
Refactoring to handle some asynchronicity
commit ddbd409acc19becf0bd5b193dcb602c273dade40
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T13:34:28Z
Asynchronicity leads to some unit test drama. We now need to retry to get the right return code.
commit 1968f392873ab57a3556d3f1bfbf7a57145806af
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T15:04:15Z
Longer time for retries.
----
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/795
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts
## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. The REST API, on the other hand, polls zookeeper every time a request is made to retrieve a configuration, a poll model. Rather than do this, it would be better to share an abstraction between the two pieces of infrastructure to enable the REST API to be backed by a TreeCache, which will prevent zookeeper from being abused and migrate it to a push model.
Testing instruction to come.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/795.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #795
----
commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T19:03:32Z
Abstracting zookeeper substantially.
commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:10:57Z
Updating tests to use the new API.
commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:15:51Z
Missed a couple of tests.
commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:32:41Z
moved the cache into commons.
commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T21:15:20Z
Updating test.
commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T23:41:44Z
Refactored tests to mock the configurations cache
commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:13:41Z
fixing tests.
commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:54:57Z
Updating some weird bugs.
commit 24df486a0c0d254f14f298ba8fb09e14152a2bf8
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T01:47:05Z
updating config test.
commit e6d48efd03bd848ca4eefd11c4277b2148550330
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:42:48Z
Updating log4j properties to debug travis.
commit 3e320d63512a860f571ee3fc0a8b41fa98b7558d
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:59:00Z
updating with better logging
commit 09019a52a4c689cdb71b512643c6f381dec114cf
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:17:20Z
reducing logging.
commit 9948df08bd2dc7a11e10bf800fe71427ba482b44
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:41:32Z
Removing debugging
commit 9134b81a5e1d5200e698131b0f66fc3ea015c56b
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:45:41Z
Better error debugging...
commit bbe87c5f1e29cb362f0dac90eb33f537285c1b27
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:47:21Z
Warn level.
commit 105127ed32003f0defdc7a7bed1f7357a69128ab
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:55:34Z
Small error in getTypes()
commit 475853cb72ab891ead4a360ca37c0e53035ba183
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:36:11Z
Updating configurations util
commit a0ce71be196b6c8c068a2f4ee4697af5a4e8d0f1
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:45:35Z
Updating integration tests.
commit c3c3c36e664c9e9216e04c72a06005a37d2a2652
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T06:21:18Z
Fixing tests.
commit a13c24a8daaf303704dc1313387b04f415cb236f
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T14:54:59Z
Updating infrastructure to be more spring-friendly.
commit 316eb5e24f6e46cd7d4364532834a460eeeac898
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T15:45:56Z
Fixed test.
commit e476cbaf5600f9428edffb25cd570eea90c8d957
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T16:16:45Z
Updating test to not be so chatty.
commit 0cb5bbcdfe1e17212620c752c200b78fe7ca6106
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T20:48:00Z
Found another classpath bug...
commit 2eb9d69f53cc4897a5112b89d9269dd40544d875
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T22:37:25Z
Refactoring to handle some asynchronicity
commit ddbd409acc19becf0bd5b193dcb602c273dade40
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T13:34:28Z
Asynchronicity leads to some unit test drama. We now need to retry to get the right return code.
commit 1968f392873ab57a3556d3f1bfbf7a57145806af
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T15:04:15Z
Longer time for retries.
----
---
[GitHub] metron issue #795: METRON-1241: Enable the REST API to use a cache for the z...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/795
I'm going to close and reopen this a few times to see if the instability in the tests are worked out.
---
[GitHub] metron issue #795: METRON-1241: Enable the REST API to use a cache for the z...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/795
Alright, great reviews. I think I have reacted to the comments; let me know if I missed anything.
---
[GitHub] metron issue #795: METRON-1241: Enable the REST API to use a cache for the z...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/795
I tested this thoroughly and everything works as expected. I made one small comment that I feel is optional. This is an awesome PR. +1
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/795
---
[GitHub] metron issue #795: METRON-1241: Enable the REST API to use a cache for the z...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/795
+1 It looks great. Thanks!
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/795
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts
## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. The REST API, on the other hand, polls zookeeper every time a request is made to retrieve a configuration, a poll model. Rather than do this, it would be better to share an abstraction between the two pieces of infrastructure to enable the REST API to be backed by a TreeCache, which will prevent zookeeper from being abused and migrate it to a push model.
Testing instruction to come.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/795.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #795
----
commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T19:03:32Z
Abstracting zookeeper substantially.
commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:10:57Z
Updating tests to use the new API.
commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:15:51Z
Missed a couple of tests.
commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T20:32:41Z
moved the cache into commons.
commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T21:15:20Z
Updating test.
commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella <ce...@gmail.com>
Date: 2017-10-09T23:41:44Z
Refactored tests to mock the configurations cache
commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:13:41Z
fixing tests.
commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T00:54:57Z
Updating some weird bugs.
commit 24df486a0c0d254f14f298ba8fb09e14152a2bf8
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T01:47:05Z
updating config test.
commit e6d48efd03bd848ca4eefd11c4277b2148550330
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:42:48Z
Updating log4j properties to debug travis.
commit 3e320d63512a860f571ee3fc0a8b41fa98b7558d
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T02:59:00Z
updating with better logging
commit 09019a52a4c689cdb71b512643c6f381dec114cf
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:17:20Z
reducing logging.
commit 9948df08bd2dc7a11e10bf800fe71427ba482b44
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:41:32Z
Removing debugging
commit 9134b81a5e1d5200e698131b0f66fc3ea015c56b
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:45:41Z
Better error debugging...
commit bbe87c5f1e29cb362f0dac90eb33f537285c1b27
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:47:21Z
Warn level.
commit 105127ed32003f0defdc7a7bed1f7357a69128ab
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T03:55:34Z
Small error in getTypes()
commit 475853cb72ab891ead4a360ca37c0e53035ba183
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:36:11Z
Updating configurations util
commit a0ce71be196b6c8c068a2f4ee4697af5a4e8d0f1
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T04:45:35Z
Updating integration tests.
commit c3c3c36e664c9e9216e04c72a06005a37d2a2652
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T06:21:18Z
Fixing tests.
commit a13c24a8daaf303704dc1313387b04f415cb236f
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T14:54:59Z
Updating infrastructure to be more spring-friendly.
commit 316eb5e24f6e46cd7d4364532834a460eeeac898
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T15:45:56Z
Fixed test.
commit e476cbaf5600f9428edffb25cd570eea90c8d957
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T16:16:45Z
Updating test to not be so chatty.
commit 0cb5bbcdfe1e17212620c752c200b78fe7ca6106
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T20:48:00Z
Found another classpath bug...
commit 2eb9d69f53cc4897a5112b89d9269dd40544d875
Author: cstella <ce...@gmail.com>
Date: 2017-10-10T22:37:25Z
Refactoring to handle some asynchronicity
commit ddbd409acc19becf0bd5b193dcb602c273dade40
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T13:34:28Z
Asynchronicity leads to some unit test drama. We now need to retry to get the right return code.
commit 1968f392873ab57a3556d3f1bfbf7a57145806af
Author: cstella <ce...@gmail.com>
Date: 2017-10-11T15:04:15Z
Longer time for retries.
----
---
[GitHub] metron issue #795: METRON-1241: Enable the REST API to use a cache for the z...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/795
TESTING PLAN
# Preliminaries
Setup some environment variables for convenience:
* `export METRON_HOME=/usr/metron/0.4.1`
* `export ZOOKEEPER=node1:2181`
* `export BROKERLIST=node1:6667`
## Adjust the Profiler Period Duration
* Adjust the profiler config by adjusting the "Period Duration" to 1
minute in Ambari Profiler config section.
## Create data generator
* Open `~/rand_gen.py` and paste the following:
```
#!/usr/bin/python
import random
import sys
import time
def main():
mu = float(sys.argv[1])
sigma = float(sys.argv[2])
freq_s = int(sys.argv[3])
while True:
out = '{ "value" : ' + str(random.gauss(mu, sigma)) + ' }'
print out
sys.stdout.flush()
time.sleep(freq_s)
if __name__ == '__main__':
main()
```
This will generate random JSON maps with a numeric field called `value`
## Create Parser
* Create a new parser called `dummy` by editing `$METRON_HOME/config/zookeeper/parsers/dummy.json`:
```
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"dummy"
}
```
* Create the dummy kafka topic:
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic dummy --partitions 1 --replication-factor 1`
# Test Cases
## Base Case
First, let's ensure that data flows through with no enrichments or
profiler.
* Push the configs via `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z $ZOOKEEPER`
* Start the parser via `$METRON_HOME/bin/start_parser_topology.sh -z $ZOOKEEPER -s dummy`
* Send some synthetic data through (stop this after a few seconds):
`python ~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic dummy`
* Validate data has been written to the index:
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "source:type", "value" ]
}
'
```
## Modify Parser
Now we'll modify the parser and see if our modifications show up.
* Delete the dummy indices:
`curl -XDELETE "http://localhost:9200/dummy*"`
* Modify `$METRON_HOME/config/zookeeper/parsers/dummy.json` like so:
```
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"dummy",
"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "parser_my_name", "parser_value_abs" ]
,"config" : {
"parser_my_name" : "TO_UPPER('casey')",
"parser_value_abs" : "ABS(value)"
}
}
]
}
```
* Push the configs via `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z $ZOOKEEPER`
* Send some synthetic data through (stop this after a few seconds):
`python ~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic dummy`
* Validate data has been written to the index with the new fields, `parser_my_name` and `parser_value_abs`:
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "source:type", "value", "parser_my_name", "parser_value_abs" ]
}
'
```
## Modify Enrichment
* Delete the dummy indices:
`curl -XDELETE "http://localhost:9200/dummy*"`
* Modify `$METRON_HOME/config/zookeeper/enrichments/dummy.json` like so:
```
{
"enrichment" : {
"fieldMap": {
"stellar" : {
"config" : {
"enr_my_name" : "TO_UPPER('casey')",
"enr_value_abs" : "ABS(value)"
}
}
}
}
}
```
* Push the configs via `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z $ZOOKEEPER`
* Send some synthetic data through (stop this after a few seconds):
`python ~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic dummy`
* Validate data has been written to the index with the new fields, `enr_my_name` and `enr_value_abs`:
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "source:type", "value", "parser_my_name", "parser_value_abs", "enr_my_name", "enr_value_abs"]
}
'
```
## Modify Indexing
We'll change the name of the index now and make sure data is flowing
through
* Delete the dummy indices:
`curl -XDELETE "http://localhost:9200/dummy*"`
* Modify `$METRON_HOME/config/zookeeper/indexing/dummy.json` like so:
```
{
"hdfs" : {
"index": "smarty",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "smarty",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "yaf",
"batchSize": 5,
"enabled" : false
}
}
```
* Push the configs via `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z $ZOOKEEPER`
* Send some synthetic data through (stop this after a few seconds):
`python ~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic dummy`
* Validate data has been written to the new index name:
```
curl -XPOST 'http://localhost:9200/smarty*/_search?pretty' -d '
{
"_source" : [ "source:type", "value", "parser_my_name", "parser_value_abs", "enr_my_name", "enr_value_abs"]
}
'
```
## Modify Profiler
* Modify `$METRON_HOME/config/zookeeper/profiler.json` like so:
```
{
"profiles": [
{
"profile": "stat",
"foreach": "'global'",
"onlyif": "true",
"init" : {
},
"update": {
"s": "STATS_ADD(s, value)"
},
"result": "s"
}
]
}
```
* Push the configs via `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z $ZOOKEEPER`
* Send some synthetic data through (let this go for some time in another window):
`python ~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic dummy`
* Execute the following and ensure the number out is neither `null` nor `NaN`:
```
sleep 128 && echo "STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('from 5 minutes ago'))))" | $METRON_HOME/bin/stellar -z node1:2181 -na
```
# REST Test
## Parser Configurations
* Ensure that you can retrieve all the parser configs by running the following and ensuring the default sensors exist:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/parser/config' | python -m json.tool | grep sensorTopic`
* You should see the default sensors as well as our sensor, `dummy`:
* `jsonMap`
* `squid`
* `websphere`,
* `snort`
* `asa`
* `bro`
* `yaf`
* `dummy`
* Delete the `websphere` parser sensor via:
`curl -u user:password -X DELETE --header 'Accept: */*' 'http://node1:8082/api/v1/sensor/parser/config/websphere'`
* Now, list the parser configs and ensure `websphere` is not in the list:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/parser/config' | python -m json.tool | grep sensorTopic`
* Add back the websphere parser:
```
curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
"sensorTopic":"websphere",
"parserConfig":
{
"grokPath":"/patterns/websphere",
"patternLabel":"WEBSPHERE",
"timestampField":"timestamp_string",
"dateFormat":"yyyy MMM dd HH:mm:ss"
}
}' 'http://node1:8082/api/v1/sensor/parser/config'
```
* Now, list the parser configs and ensure `websphere` is in the list:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/parser/config' | python -m json.tool | grep sensorTopic`
## Enrichment Configurations
* Ensure that you can retrieve all the enrichment configs by running the following and ensuring the default sensors exist:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/enrichment/config' | python -m json.tool | grep "^ \"" | awk -F: '{print $1}' | sed 's/ //g'`
* You should see the default sensors as well as our sensor, `dummy`:
* `asa`
* `bro`
* `dummy`
* `snort`
* `websphere`
* `yaf`
* Delete the `websphere` enrichment via:
`curl -u user:password -X DELETE --header 'Accept: */*' 'http://node1:8082/api/v1/sensor/enrichment/config/websphere'`
* Now, list the enrichment configs and ensure `websphere` is not in the list:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/enrichment/config' | python -m json.tool | grep "^ \"" | awk -F: '{print $1}' | sed 's/ //g'`
* Add back the websphere enrichment:
```
curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"enrichment": {
"fieldMap": {
"geo": [
"ip_src_addr"
],
"host": [
"ip_src_addr"
]
},
"fieldToTypeMap": {
"ip_src_addr": [
"playful_classification"
]
}
}
}' 'http://node1:8082/api/v1/sensor/enrichment/config/websphere'
```
* Now, list the enrichment configs and ensure `websphere` is in the list:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/enrichment/config' | python -m json.tool | grep "^ \"" | awk -F: '{print $1}' | sed 's/ //g'`
## Indexing Configurations
* Ensure that you can retrieve all the indexing configs by running the following and ensuring the default sensors exist:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/indexing/config' | python -m json.tool | grep "^ \"" | awk -F: '{print $1}' | sed 's/ //g'`
* You should see the default sensors as well as our sensor, `dummy`:
* `asa`
* `bro`
* `error`
* `dummy`
* `snort`
* `websphere`
* `yaf`
* Delete the `websphere` indexing config via:
`curl -u user:password -X DELETE --header 'Accept: */*' 'http://node1:8082/api/v1/sensor/indexing/config/websphere'`
* Now, list the indexing configs and ensure `websphere` is not in the list:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/indexing/config' | python -m json.tool | grep "^ \"" | awk -F: '{print $1}' | sed 's/ //g'`
* Add back the websphere indexing config:
```
curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"hdfs" : {
"index": "websphere",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "websphere",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "websphere",
"batchSize": 5,
"enabled" : true
}
}' 'http://node1:8082/api/v1/sensor/indexing/config/websphere'
```
* Now, list the indexing configs and ensure `websphere` is in the list:
`curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/sensor/indexing/config' | python -m json.tool | grep "^ \"" | awk -F: '{print $1}' | sed 's/ //g'`
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/795
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146010894
--- Diff: metron-platform/metron-zookeeper/src/main/java/org/apache/metron/zookeeper/ZKCache.java ---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class ZKCache implements AutoCloseable{
+ private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final int DEFAULT_CLIENT_SLEEP_MS = 1000;
+ public static final int DEFAULT_MAX_RETRIES = 3;
+
+
+
+
+ public static class Builder {
+ private Optional<CuratorFramework> client = Optional.empty();
+ private boolean ownClient = false;
+ private List<TreeCacheListener> listener = new ArrayList<>();
+ private String zkRoot;
+
+ public Builder() { }
+
+ public Builder withClient(CuratorFramework client) {
+ this.client = Optional.ofNullable(client);
+ ownClient = false;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.empty()));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl, RetryPolicy retryPolicy) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.ofNullable(retryPolicy)));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withListener(TreeCacheListener listener) {
+ this.listener.add(listener);
+ return this;
+ }
+
+ public Builder withRoot(String zkRoot) {
+ this.zkRoot = zkRoot;
+ return this;
+ }
+
+ public ZKCache build() {
+ if(!client.isPresent()) {
+ throw new IllegalArgumentException("Zookeeper client must be specified.");
+ }
+ if(listener.isEmpty()) {
+ LOG.warn("Zookeeper listener is null or empty, which is very likely an error.");
+ }
+ if(zkRoot == null) {
+ throw new IllegalArgumentException("Zookeeper root must not be null.");
+ }
+ return new ZKCache(client.get(), listener, zkRoot, ownClient);
+ }
+
+ }
+
+ private CuratorFramework client;
+ private List<TreeCacheListener> listeners;
+ private TreeCache cache;
+ private String zkRoot;
+ private boolean ownClient = false;
+
+ private ZKCache(CuratorFramework client, List<TreeCacheListener> listeners, String zkRoot, boolean ownClient) {
--- End diff --
I think the PR is looking great. It cleans a fair bit up quite nicely.
Can you add javadoc in the appropriate places? For instance, what is 'ownClient' and when/how do I use it? A few javadocs would make it so I don't have to trace through the code to find out.
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146017385
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/IndexingUpdater.java ---
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.IndexingConfigurations;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class IndexingUpdater extends ConfigurationsUpdater<IndexingConfigurations> {
+ public IndexingUpdater(Reloadable reloadable, Supplier<IndexingConfigurations> configSupplier) {
+ super(reloadable, configSupplier);
+ }
+
+ @Override
+ public Class<IndexingConfigurations> getConfigurationClass() {
+ return IndexingConfigurations.class;
+ }
+
+ @Override
+ public void forceUpdate(CuratorFramework client) {
+ try {
+ ConfigurationsUtils.updateSensorIndexingConfigsFromZookeeper(getConfigurations(), client);
+ }
+ catch (KeeperException.NoNodeException nne) {
+ LOG.warn("No current indexing configs in zookeeper, but the cache should load lazily...");
+ }
+ catch (Exception e) {
+ LOG.warn("Unable to load indexing configs from zookeeper, but the cache should load lazily...", e);
+ }
+ }
+
+ @Override
+ public IndexingConfigurations defaultConfigurations() {
+ return new IndexingConfigurations();
+ }
+
+ @Override
+ public void delete(CuratorFramework client, String path, byte[] data) throws IOException {
+ String name = path.substring(path.lastIndexOf("/") + 1);
+ if (path.startsWith(ConfigurationType.INDEXING.getZookeeperRoot())) {
--- End diff --
All of the `ConfigurationUpdater`s perform similar logic for `delete` and `update`. Would it make sense to pull this logic up so that it can be shared?
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146019541
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends ConfigurationsUpdater<ProfilerConfigurations> {
+ public ProfilerUpdater(Reloadable reloadable, Supplier<ProfilerConfigurations> configSupplier) {
+ super(reloadable, configSupplier);
+ }
+
+ @Override
+ public Class<ProfilerConfigurations> getConfigurationClass() {
+ return ProfilerConfigurations.class;
+ }
+
+ private ProfilerConfig readFromZookeeper(CuratorFramework client) throws Exception {
+ byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+ return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), ProfilerConfig.class);
+ }
+
+ @Override
+ public void forceUpdate(CuratorFramework client) {
+ try {
+ ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), client);
+ }
+ catch (KeeperException.NoNodeException nne) {
+ LOG.warn("No current global configs in zookeeper, but the cache should load lazily...");
+ }
+ catch(Exception e) {
+ LOG.warn("Unable to load global configs from zookeeper, but the cache should load lazily...", e);
+ }
+ try {
+ ProfilerConfig config = readFromZookeeper(client);
+ if(config != null) {
+ getConfigurations().updateProfilerConfig(config);
+ }
--- End diff --
Hmm, looks-like just a difference in what is offered in `ConfigurationUtils`. Seems necessary unless we were to unpack some of the logic from `ConfigurationUtils`. But diving into `ConfigurationUtils` seems more effort than it is worth for this change.
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146021528
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends ConfigurationsUpdater<ProfilerConfigurations> {
+ public ProfilerUpdater(Reloadable reloadable, Supplier<ProfilerConfigurations> configSupplier) {
+ super(reloadable, configSupplier);
+ }
+
+ @Override
+ public Class<ProfilerConfigurations> getConfigurationClass() {
+ return ProfilerConfigurations.class;
+ }
+
+ private ProfilerConfig readFromZookeeper(CuratorFramework client) throws Exception {
+ byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+ return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), ProfilerConfig.class);
+ }
+
+ @Override
+ public void forceUpdate(CuratorFramework client) {
+ try {
+ ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), client);
+ }
+ catch (KeeperException.NoNodeException nne) {
+ LOG.warn("No current global configs in zookeeper, but the cache should load lazily...");
+ }
+ catch(Exception e) {
+ LOG.warn("Unable to load global configs from zookeeper, but the cache should load lazily...", e);
+ }
+ try {
+ ProfilerConfig config = readFromZookeeper(client);
+ if(config != null) {
+ getConfigurations().updateProfilerConfig(config);
+ }
--- End diff --
@cestella Saw your response only after my last post. Yes, makes complete sense. Thanks
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146018459
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends ConfigurationsUpdater<ProfilerConfigurations> {
+ public ProfilerUpdater(Reloadable reloadable, Supplier<ProfilerConfigurations> configSupplier) {
+ super(reloadable, configSupplier);
+ }
+
+ @Override
+ public Class<ProfilerConfigurations> getConfigurationClass() {
+ return ProfilerConfigurations.class;
+ }
+
+ private ProfilerConfig readFromZookeeper(CuratorFramework client) throws Exception {
+ byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+ return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), ProfilerConfig.class);
+ }
+
+ @Override
+ public void forceUpdate(CuratorFramework client) {
+ try {
+ ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), client);
+ }
+ catch (KeeperException.NoNodeException nne) {
+ LOG.warn("No current global configs in zookeeper, but the cache should load lazily...");
+ }
+ catch(Exception e) {
+ LOG.warn("Unable to load global configs from zookeeper, but the cache should load lazily...", e);
+ }
+ try {
+ ProfilerConfig config = readFromZookeeper(client);
+ if(config != null) {
+ getConfigurations().updateProfilerConfig(config);
+ }
--- End diff --
The `ProfilerUpdater` is the only `ConfigurationsUpdater` implementation that has this additional logic in lines 61-73. Looks fishy. Why is it special?
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146021013
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends ConfigurationsUpdater<ProfilerConfigurations> {
+ public ProfilerUpdater(Reloadable reloadable, Supplier<ProfilerConfigurations> configSupplier) {
+ super(reloadable, configSupplier);
+ }
+
+ @Override
+ public Class<ProfilerConfigurations> getConfigurationClass() {
+ return ProfilerConfigurations.class;
+ }
+
+ private ProfilerConfig readFromZookeeper(CuratorFramework client) throws Exception {
+ byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+ return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), ProfilerConfig.class);
+ }
+
+ @Override
+ public void forceUpdate(CuratorFramework client) {
+ try {
+ ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), client);
+ }
+ catch (KeeperException.NoNodeException nne) {
+ LOG.warn("No current global configs in zookeeper, but the cache should load lazily...");
+ }
+ catch(Exception e) {
+ LOG.warn("Unable to load global configs from zookeeper, but the cache should load lazily...", e);
+ }
+ try {
+ ProfilerConfig config = readFromZookeeper(client);
+ if(config != null) {
+ getConfigurations().updateProfilerConfig(config);
+ }
--- End diff --
So to summarize, that was me answering my own question. Ignore this. :)
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/795
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146012465
--- Diff: metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java ---
@@ -179,6 +181,8 @@ public void test() throws Exception {
sensorParserConfig.setParserClassName("org.apache.metron.parsers.bro.BasicBroParser");
sensorParserConfig.setSensorTopic("broTest");
sensorParserConfigService.save(sensorParserConfig);
+ //we must wait for the config to find its way into the config.
+ Thread.sleep(500);
--- End diff --
Yep, will change.
---
[GitHub] metron issue #795: METRON-1241: Enable the REST API to use a cache for the z...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/795
Alright, changes included. Did I miss anything @nickwallen ?
---
[GitHub] metron issue #795: METRON-1241: Enable the REST API to use a cache for the z...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/795
Looks ok. I also ran it in my personal travis and the tests stabilized with the latest commit.
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r145992573
--- Diff: metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java ---
@@ -179,6 +181,8 @@ public void test() throws Exception {
sensorParserConfig.setParserClassName("org.apache.metron.parsers.bro.BasicBroParser");
sensorParserConfig.setSensorTopic("broTest");
sensorParserConfigService.save(sensorParserConfig);
+ //we must wait for the config to find its way into the config.
+ Thread.sleep(500);
--- End diff --
Would it make sense to use assertEventually here instead of Thread.sleep?
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146012300
--- Diff: metron-platform/metron-zookeeper/src/main/java/org/apache/metron/zookeeper/ZKCache.java ---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class ZKCache implements AutoCloseable{
+ private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final int DEFAULT_CLIENT_SLEEP_MS = 1000;
+ public static final int DEFAULT_MAX_RETRIES = 3;
+
+
+
+
+ public static class Builder {
+ private Optional<CuratorFramework> client = Optional.empty();
+ private boolean ownClient = false;
+ private List<TreeCacheListener> listener = new ArrayList<>();
+ private String zkRoot;
+
+ public Builder() { }
+
+ public Builder withClient(CuratorFramework client) {
+ this.client = Optional.ofNullable(client);
+ ownClient = false;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.empty()));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withClient(String zookeeperUrl, RetryPolicy retryPolicy) {
+ this.client = Optional.ofNullable(createClient(zookeeperUrl, Optional.ofNullable(retryPolicy)));
+ ownClient = true;
+ return this;
+ }
+
+ public Builder withListener(TreeCacheListener listener) {
+ this.listener.add(listener);
+ return this;
+ }
+
+ public Builder withRoot(String zkRoot) {
+ this.zkRoot = zkRoot;
+ return this;
+ }
+
+ public ZKCache build() {
+ if(!client.isPresent()) {
+ throw new IllegalArgumentException("Zookeeper client must be specified.");
+ }
+ if(listener.isEmpty()) {
+ LOG.warn("Zookeeper listener is null or empty, which is very likely an error.");
+ }
+ if(zkRoot == null) {
+ throw new IllegalArgumentException("Zookeeper root must not be null.");
+ }
+ return new ZKCache(client.get(), listener, zkRoot, ownClient);
+ }
+
+ }
+
+ private CuratorFramework client;
+ private List<TreeCacheListener> listeners;
+ private TreeCache cache;
+ private String zkRoot;
+ private boolean ownClient = false;
+
+ private ZKCache(CuratorFramework client, List<TreeCacheListener> listeners, String zkRoot, boolean ownClient) {
--- End diff --
good idea!
---
[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/795#discussion_r146020428
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends ConfigurationsUpdater<ProfilerConfigurations> {
+ public ProfilerUpdater(Reloadable reloadable, Supplier<ProfilerConfigurations> configSupplier) {
+ super(reloadable, configSupplier);
+ }
+
+ @Override
+ public Class<ProfilerConfigurations> getConfigurationClass() {
+ return ProfilerConfigurations.class;
+ }
+
+ private ProfilerConfig readFromZookeeper(CuratorFramework client) throws Exception {
+ byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+ return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), ProfilerConfig.class);
+ }
+
+ @Override
+ public void forceUpdate(CuratorFramework client) {
+ try {
+ ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), client);
+ }
+ catch (KeeperException.NoNodeException nne) {
+ LOG.warn("No current global configs in zookeeper, but the cache should load lazily...");
+ }
+ catch(Exception e) {
+ LOG.warn("Unable to load global configs from zookeeper, but the cache should load lazily...", e);
+ }
+ try {
+ ProfilerConfig config = readFromZookeeper(client);
+ if(config != null) {
+ getConfigurations().updateProfilerConfig(config);
+ }
--- End diff --
This is because the Profiler doesn't have a call in ConfigurationsUtils, unlike the rest of our fundamental types (e.g. `update{Parser,Enrichment,etc}ConfigsFromZookeeper`). When calling those, an update to the global config is called and then the configs are updated. This is done here and it's why it's a bit different. I thought about moving this abstraction to `ConfigurationsUtils`, but I felt that class was already so onerous and needed to be fundamentally refactored so deeply that I didn't have the heart to create a `updateProfilerConfigsFromZookeeper` method.
---