You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by jtstorck <gi...@git.apache.org> on 2016/11/09 19:36:34 UTC
[GitHub] nifi pull request #1193: NIFI-2957 ZooKeeper Migration Toolkit
GitHub user jtstorck opened a pull request:
https://github.com/apache/nifi/pull/1193
NIFI-2957 ZooKeeper Migration Toolkit
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
Reads from and writes to a Zookeeper that is open or secured via user/password digest or SASL
Supports persisting of data obtained from Zookeeper to a file or standard out as JSON
Supports sending of Zookeeper data to Zookeeper from a file or standard out as JSON
Does not allow data obtained from a source Zookeeper to be written back to the same Zookeeper connect string and path
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jtstorck/nifi NIFI-2957
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1193.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1193
----
commit 7b7bec120164bb2eb8fb7b0018ad5b5fa4fed6c7
Author: Jeff Storck <jt...@gmail.com>
Date: 2016-10-27T16:21:20Z
NIFI-2957 ZooKeeper Migration Toolkit
Reads from and writes to a Zookeeper that is open or secured via user/password digest or SASL
Supports persisting of data obtained from Zookeeper to a file or standard out as JSON
Supports sending of Zookeeper data to Zookeeper from a file or standard out as JSON
Does not allow data obtained from a source Zookeeper to be written back to the same Zookeeper connect string and path
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on the issue:
https://github.com/apache/nifi/pull/1193
@brosander New commit on the PR addresses the CNFE when connecting to ZK, and added some documentation to the code in ZooKeeperMigrator that writes to Zookeeper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1193#discussion_r87499434
--- Diff: nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/resources/logback.groovy ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import ch.qos.logback.classic.encoder.PatternLayoutEncoder
--- End diff --
I like the dsl config option here but can we put it in the classpath/ folder of the toolkit assembly? That should get picked up still be editable without recompile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1193
@jtstorck having both logback and log4j on the classpath is causing slf4j to complain:
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1193
@jtstorck I'm getting logging messages in stdout when I don't specify a file:
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2016-11-10 15:53:36,446 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator Persisting data from source ZooKeeper: ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
[
{
"connectString": "zoo1",
"path": "/node"
},
{
"path": "/node",
"data": [
99,
111,
110,
116,
101,
110,
116
],
"stat": {
"czxid": 9,
"mzxid": 9,
"ctime": 1478792367151,
"mtime": 1478792367151,
"version": 0,
"cversion": 0,
"aversion": 0,
"ephemeralOwner": 0,
"dataLength": 7,
"numChildren": 0,
"pzxid": 9
},
"acls": [
{
"perms": 31,
"id": {
"scheme": "world",
"id": "anyone"
}
}
],
"ephemeralOwner": 0
}
]2016-11-10 15:53:36,496 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator 1 nodes read from ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1193#discussion_r87508185
--- Diff: nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/resources/logback.groovy ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import ch.qos.logback.classic.encoder.PatternLayoutEncoder
--- End diff --
Yes, I'll look into that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on the issue:
https://github.com/apache/nifi/pull/1193
Also removed tests that specify a port for Zookeeper in favor of tests that check the connect string and path parsing from the command line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1193
With the new build I'm seeing errors running though the steps [here](https://github.com/jtstorck/docker-zk-krb#run-the-zk-migrator-to-read-the-protected-nodes-for-client-from-zookeeper).
```
2016-11-11 04:13:03,655 WARN [main-SendThread(zk-kerberos:2181)] org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at org.apache.zookeeper.Login.<init>(Login.java:44) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:226) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:131) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:949) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_111]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_111]
... 5 common frames omitted
2016-11-11 04:13:03,713 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator Persisting data from source ZooKeeper: ZooKeeperEndpointConfig{connectString=zk-kerberos:2181, path=/}
Exception in thread "main" java.lang.RuntimeException: unable to perform operation: KeeperErrorCode = ConnectionLoss for /
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:145)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.getNode(ZooKeeperMigrator.java:188)
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.readZooKeeper(ZooKeeperMigrator.java:95)
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:137)
```
Unfortunately it looks like Zookeeper has [hard log4j dependencies](https://github.com/apache/zookeeper/blob/release-3.4.6/src/java/main/org/apache/zookeeper/Login.java#L44).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on the issue:
https://github.com/apache/nifi/pull/1193
@brosander Updated the logging config to lessen the logging framework's output at startup and added a config to the assembly's classpath dir.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on the issue:
https://github.com/apache/nifi/pull/1193
@brosander Code has been updated to allow sending data to the root of a Zookeeper server. Also, you shouldn't have to specify the port when sending data if Zookeeper is running on the default port.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/1193
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1193
@jtstorck I'm having some problems trying to save a path (/node) and then restore at the same path in a new zookeeper instance.
Here's what I'm getting (with a teardown and reinstantiation of the zookeeper container between receive and send commands) :
```
zk-migrator.sh -z zoo1/node -r -f /out/out.json
zk-migrator.sh -z zoo1:2181/ -s -f /out/out.json
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2016-11-10 16:14:26,590 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator Source data was obtained from ZooKeeper: ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
2016-11-10 16:14:26,621 INFO [ForkJoinPool.commonPool-worker-1] o.a.n.t.zkmigrator.ZooKeeperMigrator transformed original node DataStatAclNode{path=/node, acls=[31,s{'world,'anyone}
], ephemeralOwner=0} to DataStatAclNode{path=//node, acls=[31,s{'world,'anyone}
], ephemeralOwner=0}
Exception in thread "main" java.lang.RuntimeException: unable to perform operation: java.lang.IllegalArgumentException: Invalid path string "//node" caused by empty node name specified @1
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:145)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Invalid path string "//node" caused by empty node name specified @1
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.writeZooKeeper(ZooKeeperMigrator.java:171)
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:139)
Caused by: java.lang.IllegalArgumentException: Invalid path string "//node" caused by empty node name specified @1
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:99)
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.ensureNodeExists(ZooKeeperMigrator.java:214)
at org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.lambda$null$5(ZooKeeperMigrator.java:159)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---