You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by anmolnar <gi...@git.apache.org> on 2018/03/10 16:54:12 UTC
[GitHub] zookeeper pull request #487: ZOOKEEPER-2994 Tool required to recover log and...
GitHub user anmolnar opened a pull request:
https://github.com/apache/zookeeper/pull/487
ZOOKEEPER-2994 Tool required to recover log and snapshot entries with CRC errors
In the even that the zookeeper transaction log become corrupted and fail CRC checks (preventing startup) we should have a mechanism to get the cluster running again.
Previously we achieved this by loading the broken transaction log with a modified version of ZK with disabled CRC check and forced it to snapshot.
It has proven that once you end up with the corrupt txn log there is no way to recover except manually modifying the crc check. That's basically why the tool is needed.
It's called TxnLogToolkit, a new console application similar to LogFormatter and SnapshotFormatter, but it's intentionally separated to keep backward compatibility in the existing tools.
This PR contains TXN log tool only.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anmolnar/zookeeper ZOOKEEPER-2994
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zookeeper/pull/487.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #487
----
commit 2936846d3b778fd9410910ec16659079a715e96b
Author: Andor Molnar <an...@...>
Date: 2018-03-07T15:33:23Z
ZOOKEEPER-2994. Added new tool TxnLogTool for txn log file recovery
commit 240fdb6f4841c66d76378b296a27ef3c373d2e9c
Author: Andor Molnar <an...@...>
Date: 2018-03-09T13:55:03Z
ZOOKEEPER-2994. Refactor FileTxnLog's padding logic to separate class for reusability
commit b6fe47da610cba46ba52b3870c3a72b9b63ea895
Author: Andor Molnar <an...@...>
Date: 2018-03-10T10:55:28Z
ZOOKEEPER-2994. Added padding, tool renamed to TxnLogToolkit,
interactive mode, etc.
----
---
[GitHub] zookeeper issue #487: ZOOKEEPER-2994 Tool required to recover log and snapsh...
Posted by phunt <gi...@git.apache.org>.
Github user phunt commented on the issue:
https://github.com/apache/zookeeper/pull/487
+1 Thanks @anmolnar this looks good. Please consider backporting to 3.4 (separate jira).
Also in future please don't include any changed files from the toplevel docs directory (html/pdf files) as these are regenerated during commit.
---
[GitHub] zookeeper issue #487: ZOOKEEPER-2994 Tool required to recover log and snapsh...
Posted by nkalmar <gi...@git.apache.org>.
Github user nkalmar commented on the issue:
https://github.com/apache/zookeeper/pull/487
I used your updated documentation, and managed to recover a corrupted log file:
bin/zkTxnLogToolkit.sh -d ~/workspace/zookeeper/standalone/version-2/log.1
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
4/9/18 3:13:19 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/9/18 3:15:21 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x2 closeSession null
4/9/18 3:17:41 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:18:13 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x4 closeSession null
EOF reached after 4 txns.
Corrupted log.1 file
bin/zkTxnLogToolkit.sh -d ~/workspace/zookeeper/standalone/version-2/log.1
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
CRC ERROR - 4/10/18 5:12:11 AM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/10/18 5:12:11 AM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/9/18 3:15:21 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x2 closeSession null
CRC ERROR - 4/9/18 3:17:41 PM CEST session 0x10044aa44aaaaaa cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:17:41 PM CEST session 0x10044aa44aaaaaa cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:18:13 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x4 closeSession null
EOF reached after 4 txns.
bin/zkTxnLogToolkit.sh -r ~/workspace/zookeeper/standalone/version-2/log.1
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
CRC ERROR - 4/10/18 5:12:11 AM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
Would you like to fix it (Yes/No/Abort) ? Y
EOF reached after 4 txns.
Recovery file /Users/norbertkalmar/workspace/zookeeper/standalone/version-2/log.1.fixed has been written with 1 fixed CRC error(s)
bin/zkTxnLogToolkit.sh -d ~/workspace/zookeeper/standalone/version-2/log.1.fixed
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
4/9/18 3:13:19 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/9/18 3:15:21 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x2 closeSession null
4/9/18 3:17:41 PM CEST session 0x10044aa44aaaaaa cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:18:13 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x4 closeSession null
EOF reached after 4 txns.
LGTM!
---
[GitHub] zookeeper issue #487: ZOOKEEPER-2994 Tool required to recover log and snapsh...
Posted by nkalmar <gi...@git.apache.org>.
Github user nkalmar commented on the issue:
https://github.com/apache/zookeeper/pull/487
Useful addition, +1
As @phunt pointed out, docs in zookeeperAdmin.xml could be updated.
---
[GitHub] zookeeper pull request #487: ZOOKEEPER-2994 Tool required to recover log and...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zookeeper/pull/487
---
[GitHub] zookeeper issue #487: ZOOKEEPER-2994 Tool required to recover log and snapsh...
Posted by anmolnar <gi...@git.apache.org>.
Github user anmolnar commented on the issue:
https://github.com/apache/zookeeper/pull/487
@phunt I added documentation to ZookeeperAdmin docs.
@nkalmar thanks!
---
[GitHub] zookeeper issue #487: ZOOKEEPER-2994 Tool required to recover log and snapsh...
Posted by phunt <gi...@git.apache.org>.
Github user phunt commented on the issue:
https://github.com/apache/zookeeper/pull/487
Looks promising - doesn't seem very useful (and potentially dangerous) without docs - perhaps add a troubleshooting or recovery section here?
http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_dataFileManagement
The jira was original for 3.5+, I think this would be great to get into 3.4+.
---