You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2018/02/07 21:49:34 UTC
[GitHub] metron pull request #931: METRON-1449 Set Zookeeper URL for Stellar Running ...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/931
METRON-1449 Set Zookeeper URL for Stellar Running in Zeppelin Notebook
## Changes
* This enhances the Stellar interpreter for Zeppelin to allow the user to set a `zookeeper.url` property.
* If the property is defined, a Zk client connection is accessible by the Stellar functions executed within Zeppelin. If no `zookeeper.url` is defined, then the behavior remains the same as before.
* Multiple lines of Stellar can now be executed within the same block within a Zeppelin notebook.
* Updates to the README simplify installation of the interpreter in Zeppelin.
## Testing
1. Follow the README to build the interpreter and install it in Zeppelin.
1. Launch a Zookeeper instance.
1. Load up some basic configuration. Create the standard Metron settings using `zkCli`. It would look something like the following.
```
create /metron ""
create /metron/topology ""
create /metron/topology/global "{}"
create /metron/topology/parsers "{}"
create /metron/topology/enrichments "{}"
create /metron/topology/indexing "{}"
```
1. Set the zookeeper URL. Go to the Interpreters > Stellar page and define a `zookeeper.url` property.
```
zookeeper.url = localhost:2181
```
1. Add the following dependencies to the interpreter so that we can access the metron-management functions.
Yes, this is ugly. We need to fix some of our dependencies in the metron-management project (and others.)
| artifact | exclude |
|-------------------------------------------|----------------------------------|
| org.apache.metron:metron-management:0.4.3 | |
| org.apache.metron:metron-common:0.4.3 | |
| io.thekraken:grok:0.1.0 | org.apache.commons:commons-lang3 |
| org.apache.commons:commons-lang3:3.2 | |
1. Save the interpreter changes, then open a notebook and execute the following.
```
CONFIG_GET("GLOBAL")
```
An empty set of globals should be returned from Zookeeper.
1. Run multiple expressions in a single Zeppelin code block.
## Known Problems
1. There is a problem that occurs with the Zk cache for some of the metron-management functions. Calling `CONFIG_GET` will only ever return the first value that it gets from Zk. If you change the globals via a `CONFIG_PUT`, the new values will NOT be reflected by calling `CONFIG_GET` until you restart the interpreter.
This behavior differs from the CLI REPL. I have been unable to determine why this is. I do not see anything in stellar-zeppelin that would cause this and am a little suspect of dependency issues in metron-management.
Have an idea what the problem might be? Let me know!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1449
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/931.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #931
----
commit 9cc831fc7176c2a6674c204791659a5c4ac24f52
Author: Nick Allen <ni...@...>
Date: 2018-01-16T16:34:33Z
METRON-1449 Set Zookeeper URL for Stellar Running in Zeppelin Notebook
----
---
[GitHub] metron issue #931: METRON-1449 Set Zookeeper URL for Stellar Running in Zepp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/931
The last two commits are refactorings I performed trying to track down what I describe in the PR description as a "Known Problem". I need to pull each of those out as separate PRs.
---
[GitHub] metron pull request #931: METRON-1449 Set Zookeeper URL for Stellar Running ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/931
---
[GitHub] metron issue #931: METRON-1449 Set Zookeeper URL for Stellar Running in Zepp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/931
This is now ready @ottobackwards . I reverted the two additional commits and submitted those as #982 and #983 .
---
[GitHub] metron issue #931: METRON-1449 Set Zookeeper URL for Stellar Running in Zepp...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/931
@nickwallen is this set for review? Or are you working on pulling the commits out?
---
[GitHub] metron issue #931: METRON-1449 Set Zookeeper URL for Stellar Running in Zepp...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/931
Thanks for the review @ottobackwards !
---
[GitHub] metron issue #931: METRON-1449 Set Zookeeper URL for Stellar Running in Zepp...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/931
@nickwallen would you like me to commit this? Or would you rather?
---
[GitHub] metron issue #931: METRON-1449 Set Zookeeper URL for Stellar Running in Zepp...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/931
Ran the steps with local zeppelin to full_dev node. Works as described. I have found that I needed to reinstall zeppelin, since I had stellar installed already, and just removing it from the configs and /interpreters was not enough to clear it out.
+1 . Very nice
---