You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@joshua.apache.org by mj...@apache.org on 2016/04/05 14:44:20 UTC
[33/50] incubator-joshua-site git commit: updated packedGrammar
documentation for 6.0.5 release
updated packedGrammar documentation for 6.0.5 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/0652f03d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/0652f03d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/0652f03d
Branch: refs/heads/asf-site
Commit: 0652f03da9134c8ebedf5e4062764a2f5b275fdb
Parents: b403216
Author: Felix Hieber <fh...@amazon.com>
Authored: Thu Oct 22 14:13:57 2015 +0200
Committer: Felix Hieber <fh...@amazon.com>
Committed: Thu Oct 22 14:13:57 2015 +0200
----------------------------------------------------------------------
6.0/packing.md | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/0652f03d/6.0/packing.md
----------------------------------------------------------------------
diff --git a/6.0/packing.md b/6.0/packing.md
index 8189c66..8d84004 100644
--- a/6.0/packing.md
+++ b/6.0/packing.md
@@ -18,12 +18,12 @@ The script can be found at
example usage. You can then add it to a Joshua config file, simply
replacing a `tm` path to the compressed text-file format with a path
to the packed grammar directory (Joshua will automatically detect that
-it is packed.
+it is packed, since a packed grammar is a directory).
-Packing the grammar requires first sorting it, which can take quite a
-bit of temporary space.
+Packing the grammar requires first sorting it by the rules source side,
+which can take quite a bit of temporary space.
-*CAVEAT*: You may run into problems packing very large hiero
+*CAVEAT*: You may run into problems packing very very large Hiero
grammars. Email the support list if you do.
### Examples
@@ -31,7 +31,7 @@ bit of temporary space.
A Hiero grammar, using the compressed text file version:
tm = hiero -owner pt -maxspan 20 -path grammar.filtered.gz
-
+
Pack it:
$JOSHUA/scripts/support/grammar-packer.pl grammar.filtered.gz grammar.packed
@@ -51,3 +51,24 @@ You have a different temp file location:
Update the config file line:
tm = hiero -owner pt -maxspan 20 -path grammar.packed
+
+### Using multiple packed grammars (Joshua 6.0.5)
+
+Packed grammars serialize their vocabularies which prevented the use of multiple
+packed grammars during decoding. With Joshua 6.0.5, it is possible to use multiple packed grammars during decoding if they have the same serialized vocabulary.
+This is achieved by packing these grammars jointly using a revised packing CLI.
+
+To pack multiple grammars:
+
+ $JOSHUA/scripts/support/grammar-packer.pl grammar1.filtered.gz grammar2.filtered.gz [...] grammar1.packed grammar2.packed [...]
+
+This will produce two packed grammars with the same vocabulary. To use them in the decoder, put this in your ```joshua.config```:
+
+ tm = hiero -owner pt -maxspan 20 -path grammar1.packed
+ tm = hiero -owner pt2 -maxspan 20 -path grammar2.packed
+
+Note the different owners.
+If you are trying to load multiple packed grammars that do not have the same
+vocabulary, the decoder will throw a RuntimeException at loading time:
+
+ Exception in thread "main" java.lang.RuntimeException: Trying to load multiple packed grammars with different vocabularies! Have you packed them jointly?