You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@joshua.apache.org by mj...@apache.org on 2016/09/13 21:49:34 UTC
[1/4] incubator-joshua-site git commit: Hid old documentation,
pointed to wiki
Repository: incubator-joshua-site
Updated Branches:
refs/heads/asf-site 7b2565663 -> 22be73aba
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/thrax.html
----------------------------------------------------------------------
diff --git a/6/thrax.html b/6/thrax.html
deleted file mode 100644
index dd5e841..0000000
--- a/6/thrax.html
+++ /dev/null
@@ -1,199 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Grammar extraction with Thrax</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Grammar extraction with Thrax</h2>
- </div>
-
- <div class="blog-post">
-
- <p>One day, this will hold Thrax documentation, including how to use Thrax, how to do grammar
-filtering, and details on the configuration file options. It will also include details about our
-experience setting up and maintaining Hadoop cluster installations, knowledge wrought of hard-fought
-sweat and tears.</p>
-
-<p>In the meantime, please bother <a href="http://cs.jhu.edu/~jonny/">Jonny Weese</a> if there is something you
-need to do that you don\u2019t understand. You might also be able to dig up some information <a href="http://cs.jhu.edu/~jonny/thrax/">on the old
-Thrax page</a>.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/tms.html
----------------------------------------------------------------------
diff --git a/6/tms.html b/6/tms.html
deleted file mode 100644
index f77fb26..0000000
--- a/6/tms.html
+++ /dev/null
@@ -1,312 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Building Translation Models</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Building Translation Models</h2>
- </div>
-
- <div class="blog-post">
-
- <h1 id="build-a-translation-model">Build a translation model</h1>
-
-<p>Extracting a grammar from a large amount of data is a multi-step process. The first requirement is parallel data. The Europarl, Call Home, and Fisher corpora all contain parallel translations of Spanish and English sentences.</p>
-
-<p>We will copy (or symlink) the parallel source text files in a subdirectory called <code class="highlighter-rouge">input/</code>.</p>
-
-<p>Then, we concatenate all the training files on each side. The pipeline script normally does tokenization and normalization, but in this instance we have a custom tokenizer we need to apply to the source side, so we have to do it manually and then skip that step using the <code class="highlighter-rouge">pipeline.pl</code> option <code class="highlighter-rouge">--first-step alignment</code>.</p>
-
-<ul>
- <li>
- <p>to tokenize the English data, do</p>
-
- <table>
- <tbody>
- <tr>
- <td>cat callhome.en europarl.en fisher.en > all.en</td>
- <td>$JOSHUA/scripts/training/normalize-punctuation.pl en</td>
- <td>$JOSHUA/scripts/training/penn-treebank-tokenizer.perl</td>
- <td>$JOSHUA/scripts/lowercase.perl > all.norm.tok.lc.en</td>
- </tr>
- </tbody>
- </table>
- </li>
-</ul>
-
-<p>The same can be done for the Spanish side of the input data:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>cat callhome.es europarl.es fisher.es > all.es | $JOSHUA/scripts/training/normalize-punctuation.pl es | $JOSHUA/scripts/training/penn-treebank-tokenizer.perl | $JOSHUA/scripts/lowercase.perl > all.norm.tok.lc.es
-</code></pre>
-</div>
-
-<p>By the way, an alternative tokenizer is a Twitter tokenizer found in the <a href="http://github.com/vandurme/jerboa">Jerboa</a> project.</p>
-
-<p>The final step in the training data preparation is to remove all examples in which either of the language sides is a blank line.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>paste all.norm.tok.lc.es all.norm.tok.lc.en | grep -Pv "^\t|\t$" \
- | ./splittabs.pl all.norm.tok.lc.noblanks.es all.norm.tok.lc.noblanks.en
-</code></pre>
-</div>
-
-<p>contents of <code class="highlighter-rouge">splittabls.pl</code> by Matt Post:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
-
-<span class="c1"># splits on tab, printing respective chunks to the list of files given</span>
-<span class="c1"># as script arguments</span>
-
-<span class="k">use</span> <span class="nv">FileHandle</span><span class="p">;</span>
-
-<span class="k">my</span> <span class="nv">@fh</span><span class="p">;</span>
-<span class="vg">$|</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="c1"># don't buffer output</span>
-
-<span class="k">if</span> <span class="p">(</span><span class="nv">@ARGV</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
- <span class="k">print</span> <span class="s">"Usage: splittabs.pl < tabbed-file\n"</span><span class="p">;</span>
- <span class="nb">exit</span><span class="p">;</span>
-<span class="p">}</span>
-
-<span class="k">my</span> <span class="nv">@fh</span> <span class="o">=</span> <span class="nb">map</span> <span class="p">{</span> <span class="nv">get_filehandle</span><span class="p">(</span><span class="nv">$_</span><span class="p">)</span> <span class="p">}</span> <span class="nv">@ARGV</span><span class="p">;</span>
-<span class="nv">@ARGV</span> <span class="o">=</span> <span class="p">();</span>
-
-<span class="k">while</span> <span class="p">(</span><span class="k">my</span> <span class="nv">$line</span> <span class="o">=</span> <span class="o"><></span><span class="p">)</span> <span class="p">{</span>
- <span class="nb">chomp</span><span class="p">(</span><span class="nv">$line</span><span class="p">);</span>
- <span class="k">my</span> <span class="p">(</span><span class="nv">@fields</span><span class="p">)</span> <span class="o">=</span> <span class="nb">split</span><span class="p">(</span><span class="sr">/\t/</span><span class="p">,</span><span class="nv">$line</span><span class="p">,</span><span class="nb">scalar</span> <span class="nv">@fh</span><span class="p">);</span>
-
- <span class="nb">map</span> <span class="p">{</span> <span class="k">print</span> <span class="p">{</span><span class="nv">$fh</span><span class="p">[</span><span class="nv">$_</span><span class="p">]}</span> <span class="s">"$fields[$_]\n"</span> <span class="p">}</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="nv">$#fields</span><span class="p">);</span>
-<span class="p">}</span>
-
-<span class="k">sub </span><span class="nf">get_filehandle</span> <span class="p">{</span>
- <span class="k">my</span> <span class="nv">$file</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
-
- <span class="k">if</span> <span class="p">(</span><span class="nv">$file</span> <span class="ow">eq</span> <span class="s">"-"</span><span class="p">)</span> <span class="p">{</span>
- <span class="k">return</span> <span class="o">*</span><span class="bp">STDOUT</span><span class="p">;</span>
- <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
- <span class="nb">local</span> <span class="o">*</span><span class="nv">FH</span><span class="p">;</span>
- <span class="nb">open</span> <span class="nv">FH</span><span class="p">,</span> <span class="s">">$file"</span> <span class="ow">or</span> <span class="nb">die</span> <span class="s">"can't open '$file' for writing"</span><span class="p">;</span>
- <span class="k">return</span> <span class="o">*</span><span class="nv">FH</span><span class="p">;</span>
- <span class="p">}</span>
-<span class="p">}</span>
-</code></pre>
-</div>
-
-<p>Now we can run the pipeline to extract the grammar. Run the following script:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
-
-<span class="c"># this creates a grammar</span>
-
-<span class="c"># NEED:</span>
-<span class="c"># pair</span>
-<span class="c"># type</span>
-
-<span class="nb">set</span> -u
-
-<span class="nv">pair</span><span class="o">=</span>es-en
-<span class="nb">type</span><span class="o">=</span>hiero
-
-<span class="c">#. ~/.bashrc</span>
-
-<span class="c">#basedir=$(pwd)</span>
-
-<span class="nv">dir</span><span class="o">=</span>grammar-<span class="nv">$pair</span>-<span class="nv">$type</span>
-
-<span class="o">[[</span> ! -d <span class="nv">$dir</span> <span class="o">]]</span> <span class="o">&&</span> mkdir -p <span class="nv">$dir</span>
-<span class="nb">cd</span> <span class="nv">$dir</span>
-
-<span class="nb">source</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="nv">$pair</span> | cut -d- -f 1<span class="k">)</span>
-<span class="nv">target</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="nv">$pair</span> | cut -d- -f 2<span class="k">)</span>
-
-<span class="nv">$JOSHUA</span>/scripts/training/pipeline.pl <span class="se">\</span>
- --source <span class="nv">$source</span> <span class="se">\</span>
- --target <span class="nv">$target</span> <span class="se">\</span>
- --corpus /home/hltcoe/lorland/expts/scale12/model1/input/all.norm.tok.lc.noblanks <span class="se">\</span>
- --type <span class="nv">$type</span> <span class="se">\</span>
- --joshua-mem 100g <span class="se">\</span>
- --no-prepare <span class="se">\</span>
- --first-step align <span class="se">\</span>
- --last-step thrax <span class="se">\</span>
- --hadoop <span class="nv">$HADOOP</span> <span class="se">\</span>
- --threads 8 <span class="se">\</span>
-</code></pre>
-</div>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/tutorial.html
----------------------------------------------------------------------
diff --git a/6/tutorial.html b/6/tutorial.html
deleted file mode 100644
index 6302461..0000000
--- a/6/tutorial.html
+++ /dev/null
@@ -1,407 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Pipeline tutorial</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Pipeline tutorial</h2>
- </div>
-
- <div class="blog-post">
-
- <p>This document will walk you through using the pipeline in a variety of scenarios. Once you\u2019ve gained a
-sense for how the pipeline works, you can consult the <a href="pipeline.html">pipeline page</a> for a number of
-other options available in the pipeline.</p>
-
-<h2 id="download-and-setup">Download and Setup</h2>
-
-<p>Download and install Joshua as described on the <a href="index.html">quick start page</a>, installing it under
-<code class="highlighter-rouge">~/code/</code>. Once you\u2019ve done that, you should make sure you have the following environment variable set:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>export JOSHUA=$HOME/code/joshua-v6.0.5
-export JAVA_HOME=/usr/java/default
-</code></pre>
-</div>
-
-<p>If you have a Hadoop installation, make sure you\u2019ve set <code class="highlighter-rouge">$HADOOP</code> to point to it. For example, if the <code class="highlighter-rouge">hadoop</code> command is in <code class="highlighter-rouge">/usr/bin</code>,
-you should type</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>export HADOOP=/usr
-</code></pre>
-</div>
-
-<p>Joshua will find the binary and use it to submit to your hadoop cluster. If you don\u2019t have one, just
-make sure that HADOOP is unset, and Joshua will roll one out for you and run it in
-<a href="https://hadoop.apache.org/docs/r1.2.1/single_node_setup.html">standalone mode</a>. </p>
-
-<h2 id="a-basic-pipeline-run">A basic pipeline run</h2>
-
-<p>For today\u2019s experiments, we\u2019ll be building a Spanish\u2013English system using data included in the
-<a href="/data/fisher-callhome-corpus/">Fisher and CALLHOME translation corpus</a>. This
-data was collected by translating transcribed speech from previous LDC releases.</p>
-
-<p>Download the data and install it somewhere:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>cd ~/data
-wget --no-check -O fisher-callhome-corpus.zip https://github.com/joshua-decoder/fisher-callhome-corpus/archive/master.zip
-unzip fisher-callhome-corpus.zip
-</code></pre>
-</div>
-
-<p>Then define the environment variable <code class="highlighter-rouge">$FISHER</code> to point to it:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>cd ~/data/fisher-callhome-corpus-master
-export FISHER=$(pwd)
-</code></pre>
-</div>
-
-<h3 id="preparing-the-data">Preparing the data</h3>
-
-<p>Inside the tarball is the Fisher and CALLHOME Spanish\u2013English data, which includes Kaldi-provided
-ASR output and English translations on the Fisher and CALLHOME dataset transcriptions. Because of
-licensing restrictions, we cannot distribute the Spanish transcripts, but if you have an LDC site
-license, a script is provided to build them. You can type:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>./bin/build_fisher.sh /export/common/data/corpora/LDC/LDC2010T04
-</code></pre>
-</div>
-
-<p>Where the first argument is the path to your LDC data release. This will create the files in <code class="highlighter-rouge">corpus/ldc</code>.</p>
-
-<p>In <code class="highlighter-rouge">$FISHER/corpus</code>, there are a set of parallel directories for LDC transcripts (<code class="highlighter-rouge">ldc</code>), ASR output
-(<code class="highlighter-rouge">asr</code>), oracle ASR output (<code class="highlighter-rouge">oracle</code>), and ASR lattice output (<code class="highlighter-rouge">plf</code>). The files look like this:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ ls corpus/ldc
-callhome_devtest.en fisher_dev2.en.2 fisher_dev.en.2 fisher_test.en.2
-callhome_evltest.en fisher_dev2.en.3 fisher_dev.en.3 fisher_test.en.3
-callhome_train.en fisher_dev2.es fisher_dev.es fisher_test.es
-fisher_dev2.en.0 fisher_dev.en.0 fisher_test.en.0 fisher_train.en
-fisher_dev2.en.1 fisher_dev.en.1 fisher_test.en.1 fisher_train.es
-</code></pre>
-</div>
-
-<p>If you don\u2019t have the LDC transcripts, you can use the data in <code class="highlighter-rouge">corpus/asr</code> instead. We will now use
-this data to build our own Spanish\u2013English model using Joshua\u2019s pipeline.</p>
-
-<h3 id="run-the-pipeline">Run the pipeline</h3>
-
-<p>Create an experiments directory for containing your first experiment. <em>Note: it\u2019s important that
-this <strong>not</strong> be inside your <code class="highlighter-rouge">$JOSHUA</code> directory</em>.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>mkdir ~/expts/joshua
-cd ~/expts/joshua
-</code></pre>
-</div>
-
-<p>We will now create the baseline run, using a particular directory structure for experiments that
-will allow us to take advantage of scripts provided with Joshua for displaying the results of many
-related experiments. Because this can take quite some time to run, we are going to reduce the model
-by quite a bit by
-restriction: Joshua will only use sentences in the training sets with ten or fewer words on either
-side (Spanish or English):</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>cd ~/expts/joshua
-$JOSHUA/bin/pipeline.pl \
- --rundir 1 \
- --readme "Baseline Hiero run" \
- --source es \
- --target en \
- --type hiero \
- --corpus $FISHER/corpus/ldc/fisher_train \
- --tune $FISHER/corpus/ldc/fisher_dev \
- --test $FISHER/corpus/ldc/fisher_dev2 \
- --maxlen 10 \
- --lm-order 3
-</code></pre>
-</div>
-
-<p>This will start the pipeline building a Spanish\u2013English translation system constructed from the
-training data and a dictionary, tuned against dev, and tested against devtest. It will use the
-default values for most of the pipeline: <a href="https://code.google.com/p/giza-pp/">GIZA++</a> for alignment,
-KenLM\u2019s <code class="highlighter-rouge">lmplz</code> for building the language model, Z-MERT for tuning, KenLM with left-state
-minimization for representing LM state in the decoder, and so on. We change the order of the n-gram
-model to 3 (from its default of 5) because there is not enough data to build a 5-gram LM.</p>
-
-<p>A few notes:</p>
-
-<ul>
- <li>
- <p>This will likely take many hours to run, especially if you don\u2019t have a Hadoop cluster.</p>
- </li>
- <li>
- <p>If you are running on Mac OS X, KenLM\u2019s <code class="highlighter-rouge">lmplz</code> will not build due to the absence of static
-libraries. In that case, you should add the flag <code class="highlighter-rouge">--lm-gen srilm</code> (recommended, if SRILM is
-installed) or <code class="highlighter-rouge">--lm-gen berkeleylm</code>.</p>
- </li>
-</ul>
-
-<h3 id="variations">Variations</h3>
-
-<p>Once that is finished, you will have a baseline model. From there, you might wish to try variations
-of the baseline model. Here are some examples of what you could vary:</p>
-
-<ul>
- <li>
- <p>Build an SAMT model (<code class="highlighter-rouge">--type samt</code>), GKHM model (<code class="highlighter-rouge">--type ghkm</code>), or phrasal ITG model (<code class="highlighter-rouge">--type phrasal</code>) </p>
- </li>
- <li>
- <p>Use the Berkeley aligner instead of GIZA++ (<code class="highlighter-rouge">--aligner berkeley</code>)</p>
- </li>
- <li>
- <p>Build the language model with BerkeleyLM (<code class="highlighter-rouge">--lm-gen srilm</code>) instead of KenLM (the default)</p>
- </li>
- <li>
- <p>Change the order of the LM from the default of 5 (<code class="highlighter-rouge">--lm-order 4</code>)</p>
- </li>
- <li>
- <p>Tune with MIRA instead of MERT (<code class="highlighter-rouge">--tuner mira</code>). This requires that Moses is installed.</p>
- </li>
- <li>
- <p>Decode with a wider beam (<code class="highlighter-rouge">--joshua-args '-pop-limit 200'</code>) (the default is 100)</p>
- </li>
- <li>
- <p>Add the provided BN-EN dictionary to the training data (add another <code class="highlighter-rouge">--corpus</code> line, e.g., <code class="highlighter-rouge">--corpus $FISHER/bn-en/dict.bn-en</code>)</p>
- </li>
-</ul>
-
-<p>To do this, we will create new runs that partially reuse the results of previous runs. This is
-possible by doing two things: (1) incrementing the run directory and providing an updated README
-note; (2) telling the pipeline which of the many steps of the pipeline to begin at; and (3)
-providing the needed dependencies.</p>
-
-<h1 id="a-second-run">A second run</h1>
-
-<p>Let\u2019s begin by changing the tuner, to see what effect that has. To do so, we change the run
-directory, tell the pipeline to start at the tuning step, and provide the needed dependencies:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/pipeline.pl \
- --rundir 2 \
- --readme "Tuning with MIRA" \
- --source bn \
- --target en \
- --corpus $FISHER/bn-en/tok/training.bn-en \
- --tune $FISHER/bn-en/tok/dev.bn-en \
- --test $FISHER/bn-en/tok/devtest.bn-en \
- --first-step tune \
- --tuner mira \
- --grammar 1/grammar.gz \
- --no-corpus-lm \
- --lmfile 1/lm.gz
-</code></pre>
-</div>
-
-<p>Here, we have essentially the same invocation, but we have told the pipeline to use a different
- MIRA, to start with tuning, and have provided it with the language model file and grammar it needs
- to execute the tuning step. </p>
-
-<p>Note that we have also told it not to build a language model. This is necessary because the
- pipeline always builds an LM on the target side of the training data, if provided, but we are
- supplying the language model that was already built. We could equivalently have removed the
- <code class="highlighter-rouge">--corpus</code> line.</p>
-
-<h2 id="changing-the-model-type">Changing the model type</h2>
-
-<p>Let\u2019s compare the Hiero model we\u2019ve already built to an SAMT model. We have to reextract the
-grammar, but can reuse the alignments and the language model:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/pipeline.pl \
- --rundir 3 \
- --readme "Baseline SAMT model" \
- --source bn \
- --target en \
- --corpus $FISHER/bn-en/tok/training.bn-en \
- --tune $FISHER/bn-en/tok/dev.bn-en \
- --test $FISHER/bn-en/tok/devtest.bn-en \
- --alignment 1/alignments/training.align \
- --first-step parse \
- --no-corpus-lm \
- --lmfile 1/lm.gz
-</code></pre>
-</div>
-
-<p>See <a href="pipeline.html#steps">the pipeline script page</a> for a list of all the steps.</p>
-
-<h2 id="analyzing-the-results">Analyzing the results</h2>
-
-<p>We now have three runs, in subdirectories 1, 2, and 3. We can display summary results from them
-using the <code class="highlighter-rouge">$JOSHUA/scripts/training/summarize.pl</code> script.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/whats-new.html
----------------------------------------------------------------------
diff --git a/6/whats-new.html b/6/whats-new.html
deleted file mode 100644
index 0f7e961..0000000
--- a/6/whats-new.html
+++ /dev/null
@@ -1,200 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | What's New</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>What's New</h2>
- </div>
-
- <div class="blog-post">
-
- <p>Joshua 6.0 introduces a number of new features and improvements.</p>
-
-<ul>
- <li>A new phrase-based decoder that is as fast as Moses</li>
- <li>Significantly faster hierarchical decoding</li>
- <li>Support for class-based language modeling</li>
- <li>Reflection-based loading of feature functions for super-easy
-development of new features</li>
-</ul>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/zmert.html
----------------------------------------------------------------------
diff --git a/6/zmert.html b/6/zmert.html
deleted file mode 100644
index f9a7333..0000000
--- a/6/zmert.html
+++ /dev/null
@@ -1,274 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Z-MERT</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Z-MERT</h2>
- </div>
-
- <div class="blog-post">
-
- <p>This document describes how to manually run the ZMERT module. ZMERT is Joshua\u2019s minimum error-rate
-training module, written by Omar F. Zaidan. It is easily adapted to drop in different decoders, and
-was also written so as to work with different objective functions (other than BLEU).</p>
-
-<p>((Section (1) in <code class="highlighter-rouge">$JOSHUA/examples/ZMERT/README_ZMERT.txt</code> is an expanded version of this section))</p>
-
-<p>Z-MERT, can be used by launching the driver program (<code class="highlighter-rouge">ZMERT.java</code>), which expects a config file as
-its main argument. This config file can be used to specify any subset of Z-MERT\u2019s 20-some
-parameters. For a full list of those parameters, and their default values, run ZMERT with a single
--h argument as follows:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>java -cp $JOSHUA/bin joshua.zmert.ZMERT -h
-</code></pre>
-</div>
-
-<p>So what does a Z-MERT config file look like?</p>
-
-<p>Examine the file <code class="highlighter-rouge">examples/ZMERT/ZMERT_config_ex2.txt</code>. You will find that it
-specifies the following \u201cmain\u201d MERT parameters:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>(*) -dir dirPrefix: working directory
-(*) -s sourceFile: source sentences (foreign sentences) of the MERT dataset
-(*) -r refFile: target sentences (reference translations) of the MERT dataset
-(*) -rps refsPerSen: number of reference translations per sentence
-(*) -p paramsFile: file containing parameter names, initial values, and ranges
-(*) -maxIt maxMERTIts: maximum number of MERT iterations
-(*) -ipi initsPerIt: number of intermediate initial points per iteration
-(*) -cmd commandFile: name of file containing commands to run the decoder
-(*) -decOut decoderOutFile: name of the output file produced by the decoder
-(*) -dcfg decConfigFile: name of decoder config file
-(*) -N N: size of N-best list (per sentence) generated in each MERT iteration
-(*) -v verbosity: output verbosity level (0-2; higher value => more verbose)
-(*) -seed seed: seed used to initialize the random number generator
-</code></pre>
-</div>
-
-<p>(Note that the <code class="highlighter-rouge">-s</code> parameter is only used if Z-MERT is running Joshua as an
- internal decoder. If Joshua is run as an external decoder, as is the case in
- this README, then this parameter is ignored.)</p>
-
-<p>To test Z-MERT on the 100-sentence test set of example2, provide this config
-file to Z-MERT as follows:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>java -cp bin joshua.zmert.ZMERT -maxMem 500 examples/ZMERT/ZMERT_config_ex2.txt > examples/ZMERT/ZMERT_example/ZMERT.out
-</code></pre>
-</div>
-
-<p>This will run Z-MERT for a couple of iterations on the data from the example2
-folder. (Notice that we have made copies of the source and reference files
-from example2 and renamed them as src.txt and ref.* in the MERT_example folder,
-just to have all the files needed by Z-MERT in one place.) Once the Z-MERT run
-is complete, you should be able to inspect the log file to see what kinds of
-things it did. If everything goes well, the run should take a few minutes, of
-which more than 95% is time spent by Z-MERT waiting on Joshua to finish
-decoding the sentences (once per iteration).</p>
-
-<p>The output file you get should be equivalent to <code class="highlighter-rouge">ZMERT.out.verbosity1</code>. If you
-rerun the experiment with the verbosity (-v) argument set to 2 instead of 1,
-the output file you get should be equivalent to <code class="highlighter-rouge">ZMERT.out.verbosity2</code>, which has
-more interesting details about what Z-MERT does.</p>
-
-<p>Notice the additional <code class="highlighter-rouge">-maxMem</code> argument. It tells Z-MERT that it should not
-persist to use up memory while the decoder is running (during which time Z-MERT
-would be idle). The 500 tells Z-MERT that it can only use a maximum of 500 MB.
-For more details on this issue, see section (4) in Z-MERT\u2019s README.</p>
-
-<p>A quick note about Z-MERT\u2019s interaction with the decoder. If you examine the
-file <code class="highlighter-rouge">decoder_command_ex2.txt</code>, which is provided as the commandFile (<code class="highlighter-rouge">-cmd</code>)
-argument in Z-MERT\u2019s config file, you\u2019ll find it contains the command one would
-use to run the decoder. Z-MERT launches the commandFile as an external
-process, and assumes that it will launch the decoder to produce translations.
-(Make sure that commandFile is executable.) After launching this external
-process, Z-MERT waits for it to finish, then uses the resulting output file for
-parameter tuning (in addition to the output files from previous iterations).
-The command file here only has a single command, but your command file could
-have multiple lines. Just make sure the command file itself is executable.</p>
-
-<p>Notice that the Z-MERT arguments <code class="highlighter-rouge">configFile</code> and <code class="highlighter-rouge">decoderOutFile</code> (<code class="highlighter-rouge">-cfg</code> and
-<code class="highlighter-rouge">-decOut</code>) must match the two Joshua arguments in the commandFile\u2019s (<code class="highlighter-rouge">-cmd</code>) single
-command. Also, the Z-MERT argument for N must match the value for <code class="highlighter-rouge">top_n</code> in
-Joshua\u2019s config file, indicated by the Z-MERT argument configFile (<code class="highlighter-rouge">-cfg</code>).</p>
-
-<p>For more details on Z-MERT, refer to <code class="highlighter-rouge">$JOSHUA/examples/ZMERT/README_ZMERT.txt</code></p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
[2/4] incubator-joshua-site git commit: Hid old documentation,
pointed to wiki
Posted by mj...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/packing.html
----------------------------------------------------------------------
diff --git a/6/packing.html b/6/packing.html
deleted file mode 100644
index 647dd68..0000000
--- a/6/packing.html
+++ /dev/null
@@ -1,277 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Grammar Packing</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Grammar Packing</h2>
- </div>
-
- <div class="blog-post">
-
- <p>Grammar packing refers to the process of taking a textual grammar
-output by <a href="thrax.html">Thrax</a> (or Moses, for phrase-based models) and
-efficiently encoding it so that it can be loaded
-<a href="https://aclweb.org/anthology/W/W12/W12-3134.pdf">very quickly</a> \u2014
-packing the grammar results in significantly faster load times for
-very large grammars. Packing is done automatically by the
-<a href="pipeline.html">Joshua pipeline</a>, but you can also run the packer
-manually.</p>
-
-<p>The script can be found at
-<code class="highlighter-rouge">$JOSHUA/scripts/support/grammar-packer.pl</code>. See that script for
-example usage. You can then add it to a Joshua config file, simply
-replacing a <code class="highlighter-rouge">tm</code> path to the compressed text-file format with a path
-to the packed grammar directory (Joshua will automatically detect that
-it is packed, since a packed grammar is a directory).</p>
-
-<p>Packing the grammar requires first sorting it by the rules source side,
-which can take quite a bit of temporary space.</p>
-
-<p><em>CAVEAT</em>: You may run into problems packing very very large Hiero
- grammars. Email the support list if you do.</p>
-
-<h3 id="examples">Examples</h3>
-
-<p>A Hiero grammar, using the compressed text file version:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>tm = hiero -owner pt -maxspan 20 -path grammar.filtered.gz
-</code></pre>
-</div>
-
-<p>Pack it:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/scripts/support/grammar-packer.pl grammar.filtered.gz grammar.packed
-</code></pre>
-</div>
-
-<p>Pack a really big grammar:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/scripts/support/grammar-packer.pl -m 30g grammar.filtered.gz grammar.packed
-</code></pre>
-</div>
-
-<p>Be a little more verbose:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/scripts/support/grammar-packer.pl -m 30g grammar.filtered.gz grammar.packed
-</code></pre>
-</div>
-
-<p>You have a different temp file location:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/scripts/support/grammar-packer.pl -T /local grammar.filtered.gz grammar.packed
-</code></pre>
-</div>
-
-<p>Update the config file line:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>tm = hiero -owner pt -maxspan 20 -path grammar.packed
-</code></pre>
-</div>
-
-<h3 id="using-multiple-packed-grammars-joshua-605">Using multiple packed grammars (Joshua 6.0.5)</h3>
-
-<p>Packed grammars serialize their vocabularies which prevented the use of multiple
-packed grammars during decoding. With Joshua 6.0.5, it is possible to use multiple packed grammars during decoding if they have the same serialized vocabulary.
-This is achieved by packing these grammars jointly using a revised packing CLI.</p>
-
-<p>To pack multiple grammars:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/scripts/support/grammar-packer.pl grammar1.filtered.gz grammar2.filtered.gz [...] grammar1.packed grammar2.packed [...]
-</code></pre>
-</div>
-
-<p>This will produce two packed grammars with the same vocabulary. To use them in the decoder, put this in your <code class="highlighter-rouge">joshua.config</code>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>tm = hiero -owner pt -maxspan 20 -path grammar1.packed
-tm = hiero -owner pt2 -maxspan 20 -path grammar2.packed
-</code></pre>
-</div>
-
-<p>Note the different owners.
-If you are trying to load multiple packed grammars that do not have the same
-vocabulary, the decoder will throw a RuntimeException at loading time:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>Exception in thread "main" java.lang.RuntimeException: Trying to load multiple packed grammars with different vocabularies! Have you packed them jointly?
-</code></pre>
-</div>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/pipeline.html
----------------------------------------------------------------------
diff --git a/6/pipeline.html b/6/pipeline.html
deleted file mode 100644
index f10f3fa..0000000
--- a/6/pipeline.html
+++ /dev/null
@@ -1,966 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | The Joshua Pipeline</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>The Joshua Pipeline</h2>
- </div>
-
- <div class="blog-post">
-
- <p><em>Please note that the Joshua 6.0.3 included some big changes to directory organization of the
- pipeline\u2019s files.</em></p>
-
-<p>This page describes the Joshua pipeline script, which manages the complexity of training and
-evaluating machine translation systems. The pipeline eases the pain of two related tasks in
-statistical machine translation (SMT) research:</p>
-
-<ul>
- <li>
- <p>Training SMT systems involves a complicated process of interacting steps that are
-time-consuming and prone to failure.</p>
- </li>
- <li>
- <p>Developing and testing new techniques requires varying parameters at different points in the
-pipeline. Earlier results (which are often expensive) need not be recomputed.</p>
- </li>
-</ul>
-
-<p>To facilitate these tasks, the pipeline script:</p>
-
-<ul>
- <li>
- <p>Runs the complete SMT pipeline, from corpus normalization and tokenization, through alignment,
-model building, tuning, test-set decoding, and evaluation.</p>
- </li>
- <li>
- <p>Caches the results of intermediate steps (using robust SHA-1 checksums on dependencies), so the
-pipeline can be debugged or shared across similar runs while doing away with time spent
-recomputing expensive steps.</p>
- </li>
- <li>
- <p>Allows you to jump into and out of the pipeline at a set of predefined places (e.g., the alignment
-stage), so long as you provide the missing dependencies.</p>
- </li>
-</ul>
-
-<p>The Joshua pipeline script is designed in the spirit of Moses\u2019 <code class="highlighter-rouge">train-model.pl</code>, and shares
-(and has borrowed) many of its features. It is not as extensive as Moses\u2019
-<a href="http://www.statmt.org/moses/?n=FactoredTraining.EMS">Experiment Management System</a>, which allows
-the user to define arbitrary execution dependency graphs. However, it is significantly simpler to
-use, allowing many systems to be built with a single command (that may run for days or weeks).</p>
-
-<h2 id="dependencies">Dependencies</h2>
-
-<p>The pipeline has no <em>required</em> external dependencies. However, it has support for a number of
-external packages, some of which are included with Joshua.</p>
-
-<ul>
- <li>
- <p><a href="http://code.google.com/p/giza-pp/">GIZA++</a> (included)</p>
-
- <p>GIZA++ is the default aligner. It is included with Joshua, and should compile successfully when
-you typed <code class="highlighter-rouge">ant</code> from the Joshua root directory. It is not required because you can use the
-(included) Berkeley aligner (<code class="highlighter-rouge">--aligner berkeley</code>). We have recently also provided support
-for the <a href="http://code.google.com/p/jacana-xy/wiki/JacanaXY">Jacana-XY aligner</a> (<code class="highlighter-rouge">--aligner
-jacana</code>). </p>
- </li>
- <li>
- <p><a href="http://hadoop.apache.org/">Hadoop</a> (included)</p>
-
- <p>The pipeline uses the <a href="thrax.html">Thrax grammar extractor</a>, which is built on Hadoop. If you
-have a Hadoop installation, simply ensure that the <code class="highlighter-rouge">$HADOOP</code> environment variable is defined, and
-the pipeline will use it automatically at the grammar extraction step. If you are going to
-attempt to extract very large grammars, it is best to have a good-sized Hadoop installation.</p>
-
- <p>(If you do not have a Hadoop installation, you might consider setting one up. Hadoop can be
-installed in a
-<a href="http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#PseudoDistributed">\u201cpseudo-distributed\u201d</a>
-mode that allows it to use just a few machines or a number of processors on a single machine.
-The main issue is to ensure that there are a lot of independent physical disks, since in our
-experience Hadoop starts to exhibit lots of hard-to-trace problems if there is too much demand on
-the disks.)</p>
-
- <p>If you don\u2019t have a Hadoop installation, there are still no worries. The pipeline will unroll a
-standalone installation and use it to extract your grammar. This behavior will be triggered if
-<code class="highlighter-rouge">$HADOOP</code> is undefined.</p>
- </li>
- <li>
- <p><a href="http://statmt.org/moses/">Moses</a> (not included). Moses is needed
-if you wish to use its \u2018kbmira\u2019 tuner (\u2013tuner kbmira), or if you
-wish to build phrase-based models.</p>
- </li>
- <li>
- <p><a href="http://www.speech.sri.com/projects/srilm/">SRILM</a> (not included; not needed; not recommended)</p>
-
- <p>By default, the pipeline uses the included <a href="https://kheafield.com/code/kenlm/">KenLM</a> for
-building (and also querying) language models. Joshua also includes a Java program from the
-<a href="http://code.google.com/p/berkeleylm/">Berkeley LM</a> package that contains code for constructing a
-Kneser-Ney-smoothed language model in ARPA format from the target side of your training data.<br />
-There is no need to use SRILM, but if you do wish to use it, you need to do the following:</p>
-
- <ol>
- <li>Install SRILM and set the <code class="highlighter-rouge">$SRILM</code> environment variable to point to its installed location.</li>
- <li>Add the <code class="highlighter-rouge">--lm-gen srilm</code> flag to your pipeline invocation.</li>
- </ol>
-
- <p>More information on this is available in the <a href="#lm">LM building section of the pipeline</a>. SRILM
-is not used for representing language models during decoding (and in fact is not supported,
-having been supplanted by <a href="http://kheafield.com/code/kenlm/">KenLM</a> (the default) and
-BerkeleyLM).</p>
- </li>
-</ul>
-
-<p>After installing any dependencies, follow the brief instructions on
-the <a href="install.html">installation page</a>, and then you are ready to build
-models. </p>
-
-<h2 id="a-basic-pipeline-run">A basic pipeline run</h2>
-
-<p>The pipeline takes a set of inputs (training, tuning, and test data), and creates a set of
-intermediate files in the <em>run directory</em>. By default, the run directory is the current directory,
-but it can be changed with the <code class="highlighter-rouge">--rundir</code> parameter.</p>
-
-<p>For this quick start, we will be working with the example that can be found in
-<code class="highlighter-rouge">$JOSHUA/examples/training</code>. This example contains 1,000 sentences of Urdu-English data (the full
-dataset is available as part of the
-<a href="/indian-parallel-corpora/">Indian languages parallel corpora</a> with
-100-sentence tuning and test sets with four references each.</p>
-
-<p>Running the pipeline requires two main steps: data preparation and invocation.</p>
-
-<ol>
- <li>
- <p>Prepare your data. The pipeline script needs to be told where to find the raw training, tuning,
-and test data. A good convention is to place these files in an input/ subdirectory of your run\u2019s
-working directory (NOTE: do not use <code class="highlighter-rouge">data/</code>, since a directory of that name is created and used
-by the pipeline itself for storing processed files). The expected format (for each of training,
-tuning, and test) is a pair of files that share a common path prefix and are distinguished by
-their extension, e.g.,</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>input/
- train.SOURCE
- train.TARGET
- tune.SOURCE
- tune.TARGET
- test.SOURCE
- test.TARGET
-</code></pre>
- </div>
-
- <p>These files should be parallel at the sentence level (with one sentence per line), should be in
-UTF-8, and should be untokenized (tokenization occurs in the pipeline). SOURCE and TARGET denote
-variables that should be replaced with the actual target and source language abbreviations (e.g.,
-\u201cur\u201d and \u201cen\u201d).</p>
- </li>
- <li>
- <p>Run the pipeline. The following is the minimal invocation to run the complete pipeline:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/pipeline.pl \
- --rundir . \
- --type hiero \
- --corpus input/train \
- --tune input/tune \
- --test input/devtest \
- --source SOURCE \
- --target TARGET
-</code></pre>
- </div>
-
- <p>The <code class="highlighter-rouge">--corpus</code>, <code class="highlighter-rouge">--tune</code>, and <code class="highlighter-rouge">--test</code> flags define file prefixes that are concatened with the
-language extensions given by <code class="highlighter-rouge">--target</code> and <code class="highlighter-rouge">--source</code> (with a \u201c.\u201d in between). Note the
-correspondences with the files defined in the first step above. The prefixes can be either
-absolute or relative pathnames. This particular invocation assumes that a subdirectory <code class="highlighter-rouge">input/</code>
-exists in the current directory, that you are translating from a language identified \u201cur\u201d
-extension to a language identified by the \u201cen\u201d extension, that the training data can be found at
-<code class="highlighter-rouge">input/train.en</code> and <code class="highlighter-rouge">input/train.ur</code>, and so on.</p>
- </li>
-</ol>
-
-<p><em>Don\u2019t</em> run the pipeline directly from <code class="highlighter-rouge">$JOSHUA</code>, or, for that matter, in any directory with lots of other files.
-This can cause problems because the pipeline creates lots of files under <code class="highlighter-rouge">--rundir</code> that can clobber existing files.
-You should run experiments in a clean directory.
-For example, if you have Joshua installed in <code class="highlighter-rouge">$HOME/code/joshua</code>, manage your runs in a different location, such as <code class="highlighter-rouge">$HOME/expts/joshua</code>.</p>
-
-<p>Assuming no problems arise, this command will run the complete pipeline in about 20 minutes,
-producing BLEU scores at the end. As it runs, you will see output that looks like the following:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>[train-copy-en] rebuilding...
- dep=/Users/post/code/joshua/test/pipeline/input/train.en
- dep=data/train/train.en.gz [NOT FOUND]
- cmd=cat /Users/post/code/joshua/test/pipeline/input/train.en | gzip -9n > data/train/train.en.gz
- took 0 seconds (0s)
-[train-copy-ur] rebuilding...
- dep=/Users/post/code/joshua/test/pipeline/input/train.ur
- dep=data/train/train.ur.gz [NOT FOUND]
- cmd=cat /Users/post/code/joshua/test/pipeline/input/train.ur | gzip -9n > data/train/train.ur.gz
- took 0 seconds (0s)
-...
-</code></pre>
-</div>
-
-<p>And in the current directory, you will see the following files (among
-other files, including intermediate files
-generated by the individual sub-steps).</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>data/
- train/
- corpus.ur
- corpus.en
- thrax-input-file
- tune/
- corpus.ur -> tune.tok.lc.ur
- corpus.en -> tune.tok.lc.en
- grammar.filtered.gz
- grammar.glue
- test/
- corpus.ur -> test.tok.lc.ur
- corpus.en -> test.tok.lc.en
- grammar.filtered.gz
- grammar.glue
-alignments/
- 0/
- [giza/berkeley aligner output files]
- 1/
- ...
- training.align
-thrax-hiero.conf
-thrax.log
-grammar.gz
-lm.gz
-tune/
- decoder_command
- model/
- [model files]
- params.txt
- joshua.log
- mert.log
- joshua.config.final
- final-bleu
-test/
- model/
- [model files]
- output
- final-bleu
-</code></pre>
-</div>
-
-<p>These files will be described in more detail in subsequent sections of this tutorial.</p>
-
-<p>Another useful flag is the <code class="highlighter-rouge">--rundir DIR</code> flag, which chdir()s to the specified directory before
-running the pipeline. By default the rundir is the current directory. Changing it can be useful
-for organizing related pipeline runs. In fact, we highly recommend
-that you organize your runs using consecutive integers, also taking a
-minute to pass a short note with the <code class="highlighter-rouge">--readme</code> flag, which allows you
-to quickly generate reports on <a href="#managing">groups of related experiments</a>.
-Relative paths specified to other flags (e.g., to <code class="highlighter-rouge">--corpus</code>
-or <code class="highlighter-rouge">--lmfile</code>) are relative to the directory the pipeline was called <em>from</em>, not the rundir itself
-(unless they happen to be the same, of course).</p>
-
-<p>The complete pipeline comprises many tens of small steps, which can be grouped together into a set
-of traditional pipeline tasks:</p>
-
-<ol>
- <li><a href="#prep">Data preparation</a></li>
- <li><a href="#alignment">Alignment</a></li>
- <li><a href="#parsing">Parsing</a> (syntax-based grammars only)</li>
- <li><a href="#tm">Grammar extraction</a></li>
- <li><a href="#lm">Language model building</a></li>
- <li><a href="#tuning">Tuning</a></li>
- <li><a href="#testing">Testing</a></li>
- <li><a href="#analysis">Analysis</a></li>
-</ol>
-
-<p>These steps are discussed below, after a few intervening sections about high-level details of the
-pipeline.</p>
-
-<h2 id="a-idmanaging--managing-groups-of-experiments"><a id="managing"></a> Managing groups of experiments</h2>
-
-<p>The real utility of the pipeline comes when you use it to manage groups of experiments. Typically,
-there is a held-out test set, and we want to vary a number of training parameters to determine what
-effect this has on BLEU scores or some other metric. Joshua comes with a script
-<code class="highlighter-rouge">$JOSHUA/scripts/training/summarize.pl</code> that collects information from a group of runs and reports
-them to you. This script works so long as you organize your runs as follows:</p>
-
-<ol>
- <li>
- <p>Your runs should be grouped together in a root directory, which I\u2019ll call <code class="highlighter-rouge">$EXPDIR</code>.</p>
- </li>
- <li>
- <p>For comparison purposes, the runs should all be evaluated on the same test set.</p>
- </li>
- <li>
- <p>Each run in the run group should be in its own numbered directory, shown with the files used by
-the summarize script:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>$RUNDIR/
- 1/
- README.txt
- test/
- final-bleu
- final-times
- [other files]
- 2/
- README.txt
- test/
- final-bleu
- final-times
- [other files]
- ...
-</code></pre>
- </div>
- </li>
-</ol>
-
-<p>You can get such directories using the <code class="highlighter-rouge">--rundir N</code> flag to the pipeline. </p>
-
-<p>Run directories can build off each other. For example, <code class="highlighter-rouge">1/</code> might contain a complete baseline
-run. If you wanted to just change the tuner, you don\u2019t need to rerun the aligner and model builder,
-so you can reuse the results by supplying the second run with the information it needs that was
-computed in step 1:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/pipeline.pl \
- --first-step tune \
- --grammar 1/grammar.gz \
- ...
-</code></pre>
-</div>
-
-<p>More details are below.</p>
-
-<h2 id="grammar-options">Grammar options</h2>
-
-<p>Hierarchical Joshua can extract three types of grammars: Hiero
-grammars, GHKM, and SAMT grammars. As described on the
-<a href="file-formats.html">file formats page</a>, all of them are encoded into
-the same file format, but they differ in terms of the richness of
-their nonterminal sets.</p>
-
-<p>Hiero grammars make use of a single nonterminals, and are extracted by computing phrases from
-word-based alignments and then subtracting out phrase differences. More detail can be found in
-<a href="http://www.mitpressjournals.org/doi/abs/10.1162/coli.2007.33.2.201">Chiang (2007) [PDF]</a>.
-<a href="http://www.isi.edu/%7Emarcu/papers/cr_ghkm_naacl04.pdf">GHKM</a> (new with 5.0) and
-<a href="http://www.cs.cmu.edu/~zollmann/samt/">SAMT</a> grammars make use of a source- or target-side parse
-tree on the training data, differing in the way they extract rules using these trees: GHKM extracts
-synchronous tree substitution grammar rules rooted in a subset of the tree constituents, whereas
-SAMT projects constituent labels down onto phrases. SAMT grammars are usually many times larger and
-are much slower to decode with, but sometimes increase BLEU score. Both grammar formats are
-extracted with the <a href="thrax.html">Thrax software</a>.</p>
-
-<p>By default, the Joshua pipeline extract a Hiero grammar, but this can be altered with the <code class="highlighter-rouge">--type
-(ghkm|samt)</code> flag. For GHKM grammars, the default is to use
-<a href="http://www-nlp.stanford.edu/~mgalley/software/stanford-ghkm-latest.tar.gz">Michel Galley\u2019s extractor</a>,
-but you can also use Moses\u2019 extractor with <code class="highlighter-rouge">--ghkm-extractor moses</code>. Galley\u2019s extractor only outputs
-two features, so the scores tend to be significantly lower than that of Moses\u2019.</p>
-
-<p>Joshua (new in version 6) also includes an unlexicalized phrase-based
-decoder. Building a phrase-based model requires you to have Moses
-installed, since its <code class="highlighter-rouge">train-model.perl</code> script is used to extract the
-phrase table. You can enable this by defining the <code class="highlighter-rouge">$MOSES</code> environment
-variable and then specifying <code class="highlighter-rouge">--type phrase</code>.</p>
-
-<h2 id="other-high-level-options">Other high-level options</h2>
-
-<p>The following command-line arguments control run-time behavior of multiple steps:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">--threads N</code> (1)</p>
-
- <p>This enables multithreaded operation for a number of steps: alignment (with GIZA, max two
-threads), parsing, and decoding (any number of threads)</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--jobs N</code> (1)</p>
-
- <p>This enables parallel operation over a cluster using the qsub command. This feature is not
-well-documented at this point, but you will likely want to edit the file
-<code class="highlighter-rouge">$JOSHUA/scripts/training/parallelize/LocalConfig.pm</code> to setup your qsub environment, and may also
-want to pass specific qsub commands via the <code class="highlighter-rouge">--qsub-args "ARGS"</code>
-command. We suggest you stick to the standard Joshua model that
-tries to use as many cores as are available with the <code class="highlighter-rouge">--threads N</code> option.</p>
- </li>
-</ul>
-
-<h2 id="restarting-failed-runs">Restarting failed runs</h2>
-
-<p>If the pipeline dies, you can restart it with the same command you used the first time. If you
-rerun the pipeline with the exact same invocation as the previous run (or an overlapping
-configuration \u2013 one that causes the same set of behaviors), you will see slightly different
-output compared to what we saw above:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>[train-copy-en] cached, skipping...
-[train-copy-ur] cached, skipping...
-...
-</code></pre>
-</div>
-
-<p>This indicates that the caching module has discovered that the step was already computed and thus
-did not need to be rerun. This feature is quite useful for restarting pipeline runs that have
-crashed due to bugs, memory limitations, hardware failures, and the myriad other problems that
-plague MT researchers across the world.</p>
-
-<p>Often, a command will die because it was parameterized incorrectly. For example, perhaps the
-decoder ran out of memory. This allows you to adjust the parameter (e.g., <code class="highlighter-rouge">--joshua-mem</code>) and rerun
-the script. Of course, if you change one of the parameters a step depends on, it will trigger a
-rerun, which in turn might trigger further downstream reruns.</p>
-
-<h2 id="a-idsteps--skipping-steps-quitting-early"><a id="steps"></a> Skipping steps, quitting early</h2>
-
-<p>You will also find it useful to start the pipeline somewhere other than data preparation (for
-example, if you have already-processed data and an alignment, and want to begin with building a
-grammar) or to end it prematurely (if, say, you don\u2019t have a test set and just want to tune a
-model). This can be accomplished with the <code class="highlighter-rouge">--first-step</code> and <code class="highlighter-rouge">--last-step</code> flags, which take as
-argument a case-insensitive version of the following steps:</p>
-
-<ul>
- <li>
- <p><em>FIRST</em>: Data preparation. Everything begins with data preparation. This is the default first
- step, so there is no need to be explicit about it.</p>
- </li>
- <li>
- <p><em>ALIGN</em>: Alignment. You might want to start here if you want to skip data preprocessing.</p>
- </li>
- <li>
- <p><em>PARSE</em>: Parsing. This is only relevant for building SAMT grammars (<code class="highlighter-rouge">--type samt</code>), in which case
- the target side (<code class="highlighter-rouge">--target</code>) of the training data (<code class="highlighter-rouge">--corpus</code>) is parsed before building a
- grammar.</p>
- </li>
- <li>
- <p><em>THRAX</em>: Grammar extraction <a href="thrax.html">with Thrax</a>. If you jump to this step, you\u2019ll need to
- provide an aligned corpus (<code class="highlighter-rouge">--alignment</code>) along with your parallel data. </p>
- </li>
- <li>
- <p><em>TUNE</em>: Tuning. The exact tuning method is determined with <code class="highlighter-rouge">--tuner {mert,mira,pro}</code>. With this
- option, you need to specify a grammar (<code class="highlighter-rouge">--grammar</code>) or separate tune (<code class="highlighter-rouge">--tune-grammar</code>) and test
- (<code class="highlighter-rouge">--test-grammar</code>) grammars. A full grammar (<code class="highlighter-rouge">--grammar</code>) will be filtered against the relevant
- tuning or test set unless you specify <code class="highlighter-rouge">--no-filter-tm</code>. If you want a language model built from
- the target side of your training data, you\u2019ll also need to pass in the training corpus
- (<code class="highlighter-rouge">--corpus</code>). You can also specify an arbitrary number of additional language models with one or
- more <code class="highlighter-rouge">--lmfile</code> flags.</p>
- </li>
- <li>
- <p><em>TEST</em>: Testing. If you have a tuned model file, you can test new corpora by passing in a test
- corpus with references (<code class="highlighter-rouge">--test</code>). You\u2019ll need to provide a run name (<code class="highlighter-rouge">--name</code>) to store the
- results of this run, which will be placed under <code class="highlighter-rouge">test/NAME</code>. You\u2019ll also need to provide a
- Joshua configuration file (<code class="highlighter-rouge">--joshua-config</code>), one or more language models (<code class="highlighter-rouge">--lmfile</code>), and a
- grammar (<code class="highlighter-rouge">--grammar</code>); this will be filtered to the test data unless you specify
- <code class="highlighter-rouge">--no-filter-tm</code>) or unless you directly provide a filtered test grammar (<code class="highlighter-rouge">--test-grammar</code>).</p>
- </li>
- <li>
- <p><em>LAST</em>: The last step. This is the default target of <code class="highlighter-rouge">--last-step</code>.</p>
- </li>
-</ul>
-
-<p>We now discuss these steps in more detail.</p>
-
-<h3 id="a-idprep--1-data-preparation"><a id="prep"></a> 1. DATA PREPARATION</h3>
-
-<p>Data prepare involves doing the following to each of the training data (<code class="highlighter-rouge">--corpus</code>), tuning data
-(<code class="highlighter-rouge">--tune</code>), and testing data (<code class="highlighter-rouge">--test</code>). Each of these values is an absolute or relative path
-prefix. To each of these prefixes, a \u201c.\u201d is appended, followed by each of SOURCE (<code class="highlighter-rouge">--source</code>) and
-TARGET (<code class="highlighter-rouge">--target</code>), which are file extensions identifying the languages. The SOURCE and TARGET
-files must have the same number of lines. </p>
-
-<p>For tuning and test data, multiple references are handled automatically. A single reference will
-have the format TUNE.TARGET, while multiple references will have the format TUNE.TARGET.NUM, where
-NUM starts at 0 and increments for as many references as there are.</p>
-
-<p>The following processing steps are applied to each file.</p>
-
-<ol>
- <li>
- <p><strong>Copying</strong> the files into <code class="highlighter-rouge">$RUNDIR/data/TYPE</code>, where TYPE is one of \u201ctrain\u201d, \u201ctune\u201d, or \u201ctest\u201d.
-Multiple <code class="highlighter-rouge">--corpora</code> files are concatenated in the order they are specified. Multiple <code class="highlighter-rouge">--tune</code>
-and <code class="highlighter-rouge">--test</code> flags are not currently allowed.</p>
- </li>
- <li>
- <p><strong>Normalizing</strong> punctuation and text (e.g., removing extra spaces, converting special
-quotations). There are a few language-specific options that depend on the file extension
-matching the <a href="http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes">two-letter ISO 639-1</a>
-designation.</p>
- </li>
- <li>
- <p><strong>Tokenizing</strong> the data (e.g., separating out punctuation, converting brackets). Again, there
-are language-specific tokenizations for a few languages (English, German, and Greek).</p>
- </li>
- <li>
- <p>(Training only) <strong>Removing</strong> all parallel sentences with more than <code class="highlighter-rouge">--maxlen</code> tokens on either
-side. By default, MAXLEN is 50. To turn this off, specify <code class="highlighter-rouge">--maxlen 0</code>.</p>
- </li>
- <li>
- <p><strong>Lowercasing</strong>.</p>
- </li>
-</ol>
-
-<p>This creates a series of intermediate files which are saved for posterity but compressed. For
-example, you might see</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>data/
- train/
- train.en.gz
- train.tok.en.gz
- train.tok.50.en.gz
- train.tok.50.lc.en
- corpus.en -> train.tok.50.lc.en
-</code></pre>
-</div>
-
-<p>The file \u201ccorpus.LANG\u201d is a symbolic link to the last file in the chain. </p>
-
-<h2 id="alignment-a-idalignment-">2. ALIGNMENT <a id="alignment"></a></h2>
-
-<p>Alignments are between the parallel corpora at <code class="highlighter-rouge">$RUNDIR/data/train/corpus.{SOURCE,TARGET}</code>. To
-prevent the alignment tables from getting too big, the parallel corpora are grouped into files of no
-more than ALIGNER_CHUNK_SIZE blocks (controlled with a parameter below). The last block is folded
-into the penultimate block if it is too small. These chunked files are all created in a
-subdirectory of <code class="highlighter-rouge">$RUNDIR/data/train/splits</code>, named <code class="highlighter-rouge">corpus.LANG.0</code>, <code class="highlighter-rouge">corpus.LANG.1</code>, and so on.</p>
-
-<p>The pipeline parameters affecting alignment are:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">--aligner ALIGNER</code> {giza (default), berkeley, jacana}</p>
-
- <p>Which aligner to use. The default is <a href="http://code.google.com/p/giza-pp/">GIZA++</a>, but
-<a href="http://code.google.com/p/berkeleyaligner/">the Berkeley aligner</a> can be used instead. When
-using the Berkeley aligner, you\u2019ll want to pay attention to how much memory you allocate to it
-with <code class="highlighter-rouge">--aligner-mem</code> (the default is 10g).</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--aligner-chunk-size SIZE</code> (1,000,000)</p>
-
- <p>The number of sentence pairs to compute alignments over. The training data is split into blocks
-of this size, aligned separately, and then concatenated.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--alignment FILE</code></p>
-
- <p>If you have an already-computed alignment, you can pass that to the script using this flag.
-Note that, in this case, you will want to skip data preparation and alignment using
-<code class="highlighter-rouge">--first-step thrax</code> (the first step after alignment) and also to specify <code class="highlighter-rouge">--no-prepare</code> so
-as not to retokenize the data and mess with your alignments.</p>
-
- <p>The alignment file format is the standard format where 0-indexed many-many alignment pairs for a
-sentence are provided on a line, source language first, e.g.,</p>
-
- <p>0-0 0-1 1-2 1-7 \u2026</p>
-
- <p>This value is required if you start at the grammar extraction step.</p>
- </li>
-</ul>
-
-<p>When alignment is complete, the alignment file can be found at <code class="highlighter-rouge">$RUNDIR/alignments/training.align</code>.
-It is parallel to the training corpora. There are many files in the <code class="highlighter-rouge">alignments/</code> subdirectory that
-contain the output of intermediate steps.</p>
-
-<h3 id="a-idparsing--3-parsing"><a id="parsing"></a> 3. PARSING</h3>
-
-<p>To build SAMT and GHKM grammars (<code class="highlighter-rouge">--type samt</code> and <code class="highlighter-rouge">--type ghkm</code>), the target side of the
-training data must be parsed. The pipeline assumes your target side will be English, and will parse
-it for you using <a href="http://code.google.com/p/berkeleyparser/">the Berkeley parser</a>, which is included.
-If it is not the case that English is your target-side language, the target side of your training
-data (found at CORPUS.TARGET) must already be parsed in PTB format. The pipeline will notice that
-it is parsed and will not reparse it.</p>
-
-<p>Parsing is affected by both the <code class="highlighter-rouge">--threads N</code> and <code class="highlighter-rouge">--jobs N</code> options. The former runs the parser in
-multithreaded mode, while the latter distributes the runs across as cluster (and requires some
-configuration, not yet documented). The options are mutually exclusive.</p>
-
-<p>Once the parsing is complete, there will be two parsed files:</p>
-
-<ul>
- <li><code class="highlighter-rouge">$RUNDIR/data/train/corpus.en.parsed</code>: this is the mixed-case file that was parsed.</li>
- <li><code class="highlighter-rouge">$RUNDIR/data/train/corpus.parsed.en</code>: this is a leaf-lowercased version of the above file used for
-grammar extraction.</li>
-</ul>
-
-<h2 id="thrax-grammar-extraction-a-idtm-">4. THRAX (grammar extraction) <a id="tm"></a></h2>
-
-<p>The grammar extraction step takes three pieces of data: (1) the source-language training corpus, (2)
-the target-language training corpus (parsed, if an SAMT grammar is being extracted), and (3) the
-alignment file. From these, it computes a synchronous context-free grammar. If you already have a
-grammar and wish to skip this step, you can do so passing the grammar with the <code class="highlighter-rouge">--grammar
-/path/to/grammar</code> flag.</p>
-
-<p>The main variable in grammar extraction is Hadoop. If you have a Hadoop installation, simply ensure
-that the environment variable <code class="highlighter-rouge">$HADOOP</code> is defined, and Thrax will seamlessly use it. If you <em>do
-not</em> have a Hadoop installation, the pipeline will roll out out for you, running Hadoop in
-standalone mode (this mode is triggered when <code class="highlighter-rouge">$HADOOP</code> is undefined). Theoretically, any grammar
-extractable on a full Hadoop cluster should be extractable in standalone mode, if you are patient
-enough; in practice, you probably are not patient enough, and will be limited to smaller
-datasets. You may also run into problems with disk space; Hadoop uses a lot (use <code class="highlighter-rouge">--tmp
-/path/to/tmp</code> to specify an alternate place for temporary data; we suggest you use a local disk
-partition with tens or hundreds of gigabytes free, and not an NFS partition). Setting up your own
-Hadoop cluster is not too difficult a chore; in particular, you may find it helpful to install a
-<a href="http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html">pseudo-distributed version of Hadoop</a>.
-In our experience, this works fine, but you should note the following caveats:</p>
-
-<ul>
- <li>It is of crucial importance that you have enough physical disks. We have found that having too
-few, or too slow of disks, results in a whole host of seemingly unrelated issues that are hard to
-resolve, such as timeouts. </li>
- <li>NFS filesystems can cause lots of problems. You should really try to install physical disks that
-are dedicated to Hadoop scratch space.</li>
-</ul>
-
-<p>Here are some flags relevant to Hadoop and grammar extraction with Thrax:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">--hadoop /path/to/hadoop</code></p>
-
- <p>This sets the location of Hadoop (overriding the environment variable <code class="highlighter-rouge">$HADOOP</code>)</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--hadoop-mem MEM</code> (2g)</p>
-
- <p>This alters the amount of memory available to Hadoop mappers (passed via the
-<code class="highlighter-rouge">mapred.child.java.opts</code> options).</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--thrax-conf FILE</code></p>
-
- <p>Use the provided Thrax configuration file instead of the (grammar-specific) default. The Thrax
- templates are located at <code class="highlighter-rouge">$JOSHUA/scripts/training/templates/thrax-TYPE.conf</code>, where TYPE is one
- of \u201chiero\u201d or \u201csamt\u201d.</p>
- </li>
-</ul>
-
-<p>When the grammar is extracted, it is compressed and placed at <code class="highlighter-rouge">$RUNDIR/grammar.gz</code>.</p>
-
-<h2 id="a-idlm--5-language-model"><a id="lm"></a> 5. Language model</h2>
-
-<p>Before tuning can take place, a language model is needed. A language model is always built from the
-target side of the training corpus unless <code class="highlighter-rouge">--no-corpus-lm</code> is specified. In addition, you can
-provide other language models (any number of them) with the <code class="highlighter-rouge">--lmfile FILE</code> argument. Other
-arguments are as follows.</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">--lm</code> {kenlm (default), berkeleylm}</p>
-
- <p>This determines the language model code that will be used when decoding. These implementations
-are described in their respective papers (PDFs:
-<a href="http://kheafield.com/professional/avenue/kenlm.pdf">KenLM</a>,
-<a href="http://nlp.cs.berkeley.edu/pubs/Pauls-Klein_2011_LM_paper.pdf">BerkeleyLM</a>). KenLM is written in
-C++ and requires a pass through the JNI, but is recommended because it supports left-state minimization.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--lmfile FILE</code></p>
-
- <p>Specifies a pre-built language model to use when decoding. This language model can be in ARPA
-format, or in KenLM format when using KenLM or BerkeleyLM format when using that format.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--lm-gen</code> {kenlm (default), srilm, berkeleylm}, <code class="highlighter-rouge">--buildlm-mem MEM</code>, <code class="highlighter-rouge">--witten-bell</code></p>
-
- <p>At the tuning step, an LM is built from the target side of the training data (unless
-<code class="highlighter-rouge">--no-corpus-lm</code> is specified). This controls which code is used to build it. The default is a
-KenLM\u2019s <a href="http://kheafield.com/code/kenlm/estimation/">lmplz</a>, and is strongly recommended.</p>
-
- <p>If SRILM is used, it is called with the following arguments:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code> $SRILM/bin/i686-m64/ngram-count -interpolate SMOOTHING -order 5 -text TRAINING-DATA -unk -lm lm.gz
-</code></pre>
- </div>
-
- <p>Where SMOOTHING is <code class="highlighter-rouge">-kndiscount</code>, or <code class="highlighter-rouge">-wbdiscount</code> if <code class="highlighter-rouge">--witten-bell</code> is passed to the pipeline.</p>
-
- <p><a href="http://code.google.com/p/berkeleylm/source/browse/trunk/src/edu/berkeley/nlp/lm/io/MakeKneserNeyArpaFromText.java">BerkeleyLM java class</a>
-is also available. It computes a Kneser-Ney LM with a constant discounting (0.75) and no count
-thresholding. The flag <code class="highlighter-rouge">--buildlm-mem</code> can be used to control how much memory is allocated to the
-Java process. The default is \u201c2g\u201d, but you will want to increase it for larger language models.</p>
-
- <p>A language model built from the target side of the training data is placed at <code class="highlighter-rouge">$RUNDIR/lm.gz</code>. </p>
- </li>
-</ul>
-
-<h2 id="interlude-decoder-arguments">Interlude: decoder arguments</h2>
-
-<p>Running the decoder is done in both the tuning stage and the testing stage. A critical point is
-that you have to give the decoder enough memory to run. Joshua can be very memory-intensive, in
-particular when decoding with large grammars and large language models. The default amount of
-memory is 3100m, which is likely not enough (especially if you are decoding with SAMT grammar). You
-can alter the amount of memory for Joshua using the <code class="highlighter-rouge">--joshua-mem MEM</code> argument, where MEM is a Java
-memory specification (passed to its <code class="highlighter-rouge">-Xmx</code> flag).</p>
-
-<h2 id="a-idtuning--6-tuning"><a id="tuning"></a> 6. TUNING</h2>
-
-<p>Two optimizers are provided with Joshua: MERT and PRO (<code class="highlighter-rouge">--tuner {mert,pro}</code>). If Moses is
-installed, you can also use Cherry & Foster\u2019s k-best batch MIRA (<code class="highlighter-rouge">--tuner mira</code>, recommended).
-Tuning is run till convergence in the <code class="highlighter-rouge">$RUNDIR/tune</code> directory.</p>
-
-<p>When tuning is finished, each final configuration file can be found at either</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$RUNDIR/tune/joshua.config.final
-</code></pre>
-</div>
-
-<h2 id="a-idtesting--7-testing"><a id="testing"></a> 7. Testing</h2>
-
-<p>For each of the tuner runs, Joshua takes the tuner output file and decodes the test set. If you
-like, you can also apply minimum Bayes-risk decoding to the decoder output with <code class="highlighter-rouge">--mbr</code>. This
-usually yields about 0.3 - 0.5 BLEU points, but is time-consuming.</p>
-
-<p>After decoding the test set with each set of tuned weights, Joshua computes the mean BLEU score,
-writes it to <code class="highlighter-rouge">$RUNDIR/test/final-bleu</code>, and cats it. It also writes a file
-<code class="highlighter-rouge">$RUNDIR/test/final-times</code> containing a summary of runtime information. That\u2019s the end of the pipeline!</p>
-
-<p>Joshua also supports decoding further test sets. This is enabled by rerunning the pipeline with a
-number of arguments:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">--first-step TEST</code></p>
-
- <p>This tells the decoder to start at the test step.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">--joshua-config CONFIG</code></p>
-
- <p>A tuned parameter file is required. This file will be the output of some prior tuning run.
-Necessary pathnames and so on will be adjusted.</p>
- </li>
-</ul>
-
-<h2 id="a-idanalysis-8-analysis"><a id="analysis"> 8. ANALYSIS</a></h2>
-
-<p>If you have used the suggested layout, with a number of related runs all contained in a common
-directory with sequential numbers, you can use the script <code class="highlighter-rouge">$JOSHUA/scripts/training/summarize.pl</code> to
-display a summary of the mean BLEU scores from all runs, along with the text you placed in the run
-README file (using the pipeline\u2019s <code class="highlighter-rouge">--readme TEXT</code> flag).</p>
-
-<h2 id="common-use-cases-and-pitfalls">COMMON USE CASES AND PITFALLS</h2>
-
-<ul>
- <li>
- <p>If the pipeline dies at the \u201cthrax-run\u201d stage with an error like the following:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>JOB FAILED (return code 1)
-hadoop/bin/hadoop: line 47:
-/some/path/to/a/directory/hadoop/bin/hadoop-config.sh: No such file or directory
-Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FsShell
-Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
-</code></pre>
- </div>
-
- <p>This occurs if the <code class="highlighter-rouge">$HADOOP</code> environment variable is set but does not point to a working
-Hadoop installation. To fix it, make sure to unset the variable:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code># in bash
-unset HADOOP
-</code></pre>
- </div>
-
- <p>and then rerun the pipeline with the same invocation.</p>
- </li>
- <li>
- <p>Memory usage is a major consideration in decoding with Joshua and hierarchical grammars. In
-particular, SAMT grammars often require a large amount of memory. Many steps have been taken to
-reduce memory usage, including beam settings and test-set- and sentence-level filtering of
-grammars. However, memory usage can still be in the tens of gigabytes.</p>
-
- <p>To accommodate this kind of variation, the pipeline script allows you to specify both (a) the
-amount of memory used by the Joshua decoder instance and (b) the amount of memory required of
-nodes obtained by the qsub command. These are accomplished with the <code class="highlighter-rouge">--joshua-mem</code> MEM and
-<code class="highlighter-rouge">--qsub-args</code> ARGS commands. For example,</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>pipeline.pl --joshua-mem 32g --qsub-args "-l pvmem=32g -q himem.q" ...
-</code></pre>
- </div>
-
- <p>Also, should Thrax fail, it might be due to a memory restriction. By default, Thrax requests 2 GB
-from the Hadoop server. If more memory is needed, set the memory requirement with the
-<code class="highlighter-rouge">--hadoop-mem</code> in the same way as the <code class="highlighter-rouge">--joshua-mem</code> option is used.</p>
- </li>
- <li>
- <p>Other pitfalls and advice will be added as it is discovered.</p>
- </li>
-</ul>
-
-<h2 id="feedback">FEEDBACK</h2>
-
-<p>Please email joshua_support@googlegroups.com with problems or suggestions.</p>
-
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/quick-start.html
----------------------------------------------------------------------
diff --git a/6/quick-start.html b/6/quick-start.html
deleted file mode 100644
index d1b9d51..0000000
--- a/6/quick-start.html
+++ /dev/null
@@ -1,251 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Quick Start</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Quick Start</h2>
- </div>
-
- <div class="blog-post">
-
- <p>If you just want to use Joshua to translate data, the quickest way is
-to download a <a href="/language-packs/">pre-built model</a>. </p>
-
-<p>If not language pack is available, or if you have your own parallel
-data that you want to train the translation engine on, then you have
-to build your own model. This takes a bit more knowledge and effort,
-but is made easier with Joshua\u2019s <a href="pipeline.html">pipeline script</a>,
-which runs all the steps of preparing data, aligning it, and
-extracting and tuning component models. </p>
-
-<p>Detailed information about running the pipeline can be found in
-<a href="/6.0/pipeline.html">the pipeline documentation</a>, but as a quick
-start, you can build a simple Bengali\u2013English model by following
-these instructions.</p>
-
-<p><em>NOTE: We suggest you build models outside the <code class="highlighter-rouge">$JOSHUA</code> directory</em>.</p>
-
-<p>First, download the dataset:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>mkdir -p ~/models/bn-en/
-cd ~/models/bn-en
-wget -q https://github.com/joshua-decoder/indian-parallel-corpora/archive/1.0.tar.gz
-tar xzf indian-parallel-corpora-1.0.tar.gz
-ln -s indian-parallel-corpora-1.0 input
-</code></pre>
-</div>
-
-<p>Then, train and test a model</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/pipeline.pl --source bn --target en \
- --type hiero \
- --no-prepare --aligner berkeley \
- --corpus input/bn-en/tok/training.bn-en \
- --tune input/bn-en/tok/dev.bn-en \
- --test input/bn-en/tok/devtest.bn-en
-</code></pre>
-</div>
-
-<p>This will align the data with the Berkeley aligner, build a Hiero
-model, tune with MERT, decode the test sets, and reports results that
-should correspond with what you find on
-<a href="/indian-parallel-corpora/">the Indian Parallel Corpora page</a>. For
-more details, including information on the many options available with
-the pipeline script, please see <a href="pipeline.html">its documentation page</a>.</p>
-
-<p>Finally, you can export the full model as a language pack:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>./run-bundler.py \
- tune/joshua.config.final \
- language-pack-bn-en \
- --pack-tm grammar.gz
-</code></pre>
-</div>
-
-<p>(or possibly <code class="highlighter-rouge">tune/1/joshua.config.final</code> if you\u2019re using an older version of
-the pipeline).</p>
-
-<p>This will create a <a href="bundle.html">runnable model</a> in
-<code class="highlighter-rouge">language-pack-bn-en</code>. See the <code class="highlighter-rouge">README</code> file in that directory for
-information on how to run the decoder.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/server.html
----------------------------------------------------------------------
diff --git a/6/server.html b/6/server.html
deleted file mode 100644
index 07df127..0000000
--- a/6/server.html
+++ /dev/null
@@ -1,218 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Server mode</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Server mode</h2>
- </div>
-
- <div class="blog-post">
-
- <p>The Joshua decoder can be run as a TCP/IP server instead of a POSIX-style command-line tool. Clients can concurrently connect to a socket and receive a set of newline-separated outputs for a set of newline-separated inputs.</p>
-
-<p>Threading takes place both within and across requests. Threads from the decoder pool are assigned in round-robin manner across requests, preventing starvation.</p>
-
-<h1 id="invoking-the-server">Invoking the server</h1>
-
-<p>A running server is configured at invokation time. To start in server mode, run <code class="highlighter-rouge">joshua-decoder</code> with the option <code class="highlighter-rouge">-server-port [PORT]</code>. Additionally, the server can be configured in the same ways as when using the command-line-functionality.</p>
-
-<p>E.g.,</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/joshua-decoder -server-port 10101 -mark-oovs false -output-format "%s" -threads 10
-</code></pre>
-</div>
-
-<h2 id="using-the-server">Using the server</h2>
-
-<p>To test that the server is working, a set of inputs can be sent to the server from the command line. </p>
-
-<p>The server, as configured in the example above, will then respond to requests on port 10101. You can test it out with the <code class="highlighter-rouge">nc</code> utility:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>wget -qO - http://cs.jhu.edu/~post/files/pg1023.txt | head -132 | tail -11 | nc localhost 10101
-</code></pre>
-</div>
-
-<p>Since no model was loaded, this will just return the text to you as sent to the server.</p>
-
-<p>The <code class="highlighter-rouge">-server-port</code> option can also be used when creating a <a href="bundle.html">bundled configuration</a> that will be run in server mode.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
[4/4] incubator-joshua-site git commit: Hid old documentation,
pointed to wiki
Posted by mj...@apache.org.
Hid old documentation, pointed to wiki
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/22be73ab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/22be73ab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/22be73ab
Branch: refs/heads/asf-site
Commit: 22be73abaae047e5862d559be019d95eebc78983
Parents: 7b25656
Author: Matt Post <po...@cs.jhu.edu>
Authored: Tue Sep 13 23:49:28 2016 +0200
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Tue Sep 13 23:49:28 2016 +0200
----------------------------------------------------------------------
6.0/index.html | 206 +---------
6/advanced.html | 192 ----------
6/bundle.html | 297 ---------------
6/decoder.html | 671 --------------------------------
6/faq.html | 376 ------------------
6/features.html | 192 ----------
6/file-formats.html | 270 -------------
6/index.html | 210 -----------
6/install.html | 301 ---------------
6/jacana.html | 331 ----------------
6/large-lms.html | 390 -------------------
6/packing.html | 277 --------------
6/pipeline.html | 966 -----------------------------------------------
6/quick-start.html | 251 ------------
6/server.html | 218 -----------
6/thrax.html | 199 ----------
6/tms.html | 312 ---------------
6/tutorial.html | 407 --------------------
6/whats-new.html | 200 ----------
6/zmert.html | 274 --------------
20 files changed, 4 insertions(+), 6536 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6.0/index.html
----------------------------------------------------------------------
diff --git a/6.0/index.html b/6.0/index.html
index 7392541..2be3626 100644
--- a/6.0/index.html
+++ b/6.0/index.html
@@ -2,209 +2,11 @@
<html lang="en">
<head>
<meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Joshua documentation</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
+ <meta http-equiv="refresh" content="0; url=https://cwiki.apache.org/confluence/display/JOSHUA/" />
</head>
-
<body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Joshua documentation</h2>
- </div>
-
- <div class="blog-post">
-
- <p>This page contains end-user oriented documentation for the 6.0 release of
-<a href="http://joshua-decoder.org/">the Joshua decoder</a>.</p>
-
-<p>To navigate the documentation, use the links on the navigation bar to
-the left. For more detail on the decoder itself, including its command-line options, see
-<a href="decoder.html">the Joshua decoder page</a>. You can also learn more about other steps of
-<a href="pipeline.html">the Joshua MT pipeline</a>, including <a href="thrax.html">grammar extraction</a> with Thrax and
-Joshua\u2019s <a href="packing.html">efficient grammar representation</a>.</p>
-
-<p>A <a href="bundle.html">bundled configuration</a>, which is a minimal set of configuration, resource, and script files, can be created and easily transferred and shared.</p>
-
-<h2 id="development">Development</h2>
-
-<p>For developer support, please consult <a href="http://cs.jhu.edu/~post/joshua-docs">the javadoc documentation</a> and the <a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Joshua developers mailing list</a>.</p>
-
-<h2 id="support">Support</h2>
-
-<p>If you have problems or issues, you might find some help <a href="faq.html">on our answers page</a> or
-<a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_support">in the mailing list archives</a>.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
+ The Apache Joshua web pages are now being hosted on Confluence. If you are not automatically
+ redirected, please <a href="https://cwiki.apache.org/confluence/display/JOSHUA/">follow this
+ link.</a>
</body>
</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/advanced.html
----------------------------------------------------------------------
diff --git a/6/advanced.html b/6/advanced.html
deleted file mode 100644
index f4a3335..0000000
--- a/6/advanced.html
+++ /dev/null
@@ -1,192 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Advanced features</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Advanced features</h2>
- </div>
-
- <div class="blog-post">
-
-
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/bundle.html
----------------------------------------------------------------------
diff --git a/6/bundle.html b/6/bundle.html
deleted file mode 100644
index 1f0ee11..0000000
--- a/6/bundle.html
+++ /dev/null
@@ -1,297 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Building a language pack</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Building a language pack</h2>
- </div>
-
- <div class="blog-post">
-
- <p><em>The information in this page applies to Joshua 6.0.3 and greater</em>.</p>
-
-<p>Joshua distributes <a href="/language-packs">language packs</a>, which are models
-that have been trained and tuned for particular language pairs. You
-can easily create your own language pack after you have trained and
-tuned a model using the provided
-<code class="highlighter-rouge">$JOSHUA/scripts/support/run-bundler.py</code> script, which gathers files
-from a pipeline training directory and bundles them together for easy
-distribution and release.</p>
-
-<p>The script takes just two mandatory arguments in the following order:</p>
-
-<ol>
- <li>The path to the Joshua configuration file to base the bundle
-on. This file should contain the tuned weights from the tuning run, so
-you can use either the final tuned file from the tuning run
-(<code class="highlighter-rouge">tune/joshua.config.final</code>) or from the test run
-(<code class="highlighter-rouge">test/model/joshua.config</code>).</li>
- <li>The directory to place the language pack in. If this directory
-already exists, the script will die, unless you also pass <code class="highlighter-rouge">--force</code>.</li>
-</ol>
-
-<p>In addition, there are a number of other arguments that may be important.</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">--root /path/to/root</code>. If file paths in the Joshua config file are
- not absolute, you need to provide relative root. If you specify a
- tuned pipeline file (such as <code class="highlighter-rouge">tune/joshua.config.final</code> above), the
- paths should all be absolute. If you instead provide a config file
- from a previous run bundle (e.g., <code class="highlighter-rouge">test/model/joshua.config</code>), the
- bundle directory above is the relative root.</p>
- </li>
- <li>
- <p>The config file options that are used in the pipeline are likely not
-the ones you want if you release a model. For example, the tuning
-configuration file contains options that tell Joshua to output 300
-translation candidates for each sentence (<code class="highlighter-rouge">-top-n 300</code>) and to
-include lots of detail about each translation (<code class="highlighter-rouge">-output-format '%i
-||| %s ||| %f ||| %c'</code>). Because of this, you will want to tell the
-run bundler to change many of the config file options to be more
-geared towards human-readable output. The default copy-config
-options are options are <code class="highlighter-rouge">-top-n 0 -output-format %S -mark-oovs
-false</code>, which accomplishes exactly this (human readability).</p>
- </li>
- <li>
- <p>A very important issue has to do with the translation model (the
-\u201cTM\u201d, also sometimes called the grammar or phrase table). The
-translation model can be very large, so that it takes a long time to
-load and to <a href="packing.html">pack</a>. To reduce this time during model
-training, the translation model is filtered against the tuning and
-testing data in the pipeline, and these filtered models will be what
-is listed in the source config files. However, when exporting a
-model for use as a language pack, you need to export the full model
-instead of the filtered one so as to maximize your coverage on new
-test data. The <code class="highlighter-rouge">--tm</code> parameter is used to accomplish this; it takes
-an argument specifying the path to the full model. If you would
-additionally like the large model to be <a href="packing.html">packed</a> (this
-is recommended; it reformats the TM so that it can be quickly loaded
-at run time), you can use <code class="highlighter-rouge">--pack-tm</code> instead. You can only pack one
-TM (but typically there is only TM anyway). Multiple <code class="highlighter-rouge">--tm</code>
-parameters can be passed; they will replace TMs found in the config
-file in the order they are found.</p>
- </li>
-</ul>
-
-<p>Here is an example invocation for packing a hierarchical model using
-the final tuned Joshua config file:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>./run-bundler.py \
- --force --verbose \
- /path/to/rundir/tune/joshua.config.final \
- language-pack-YYYY-MM-DD \
- --root /path/to/rundir \
- --pack-tm /path/to/rundir/grammar.gz \
- --copy-config-options \
- '-top-n 1 -output-format %S -mark-oovs false' \
- --server-port 5674
-</code></pre>
-</div>
-
-<p>The copy config options tell the decoder to present just the
-single-best (<code class="highlighter-rouge">-top-n 0</code>) translated output string that has been
-heuristically capitalized (<code class="highlighter-rouge">-output-format %S</code>), to not append <code class="highlighter-rouge">_OOV</code>
-to OOVs (<code class="highlighter-rouge">-mark-oovs false</code>), and to use the translation model
-<code class="highlighter-rouge">/path/to/rundir/grammar.gz</code> as the main translation model, packing it
-before placing it in the bundle. Note that these arguments to
-<code class="highlighter-rouge">--copy-config</code> are the default, so you could leave this off entirely.
-See <a href="decoder.html">this page</a> for a longer list of decoder options.</p>
-
-<p>This command is a slight variation used for phrase-based models, which
-instead takes the test-set Joshua config (the result is the same):</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>./run-bundler.py \
- --force --verbose \
- /path/to/rundir/test/model/joshua.config \
- --root /path/to/rundir/test/model \
- language-pack-YYYY-MM-DD \
- --pack-tm /path/to/rundir/model/phrase-table.gz \
- --server-port 5674
-</code></pre>
-</div>
-
-<p>In both cases, a new directory <code class="highlighter-rouge">language-pack-YYYY-MM-DD</code> will be
-created along with a README and a number of support files.</p>
-
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/decoder.html
----------------------------------------------------------------------
diff --git a/6/decoder.html b/6/decoder.html
deleted file mode 100644
index 45d238b..0000000
--- a/6/decoder.html
+++ /dev/null
@@ -1,671 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Decoder configuration parameters</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Decoder configuration parameters</h2>
- </div>
-
- <div class="blog-post">
-
- <p>Joshua configuration parameters affect the runtime behavior of the decoder itself. This page
-describes the complete list of these parameters and describes how to invoke the decoder manually.</p>
-
-<p>To run the decoder, a convenience script is provided that loads the necessary Java libraries.
-Assuming you have set the environment variable <code class="highlighter-rouge">$JOSHUA</code> to point to the root of your installation,
-its syntax is:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/decoder [-m memory-amount] [-c config-file other-joshua-options ...]
-</code></pre>
-</div>
-
-<p>The <code class="highlighter-rouge">-m</code> argument, if present, must come first, and the memory specification is in Java format
-(e.g., 400m, 4g, 50g). Most notably, the suffixes \u201cm\u201d and \u201cg\u201d are used for \u201cmegabytes\u201d and
-\u201cgigabytes\u201d, and there cannot be a space between the number and the unit. The value of this
-argument is passed to Java itself in the invocation of the decoder, and the remaining options are
-passed to Joshua. The <code class="highlighter-rouge">-c</code> parameter has special import because it specifies the location of the
-configuration file.</p>
-
-<p>The Joshua decoder works by reading from STDIN and printing translations to STDOUT as they are
-received, according to a number of <a href="#output">output options</a>. If no run-time parameters are
-specified (e.g., no translation model), sentences are simply pushed through untranslated. Blank
-lines are similarly pushed through as blank lines, so as to maintain parallelism with the input.</p>
-
-<p>Parameters can be provided to Joshua via a configuration file and from the command
-line. Command-line arguments override values found in the configuration file. The format for
-configuration file parameters is</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>parameter = value
-</code></pre>
-</div>
-
-<p>Command-line options are specified in the following format</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>-parameter value
-</code></pre>
-</div>
-
-<p>Values are one of four types (which we list here mostly to call attention to the boolean format):</p>
-
-<ul>
- <li>STRING, an arbitrary string (no spaces)</li>
- <li>FLOAT, a floating-point value</li>
- <li>INT, an integer</li>
- <li>
- <p>BOOLEAN, a boolean value. For booleans, <code class="highlighter-rouge">true</code> evaluates to true, and all other values evaluate
-to false. For command-line options, the value may be omitted, in which case it evaluates to
-true. For example, the following are equivalent:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/decoder -mark-oovs true
-$JOSHUA/bin/decoder -mark-oovs
-</code></pre>
- </div>
- </li>
-</ul>
-
-<h2 id="joshua-configuration-file">Joshua configuration file</h2>
-
-<p>In addition to the decoder parameters described below, the configuration file contains the model
-feature weights. These weights are distinguished from runtime parameters in that they are delimited
-by a space instead of an equals sign. They take the following
-format, and by convention are placed at the end of the configuration file:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>lm_0 4.23
-tm_pt_0 -0.2
-OOVPenalty -100
-</code></pre>
-</div>
-
-<p>Joshua can make use of thousands of features, which are described in further detail in the
-<a href="features.html">feature file</a>.</p>
-
-<h2 id="joshua-decoder-parameters">Joshua decoder parameters</h2>
-
-<p>This section contains a list of the Joshua run-time parameters. An important note about the
-parameters is that they are collapsed to canonical form, in which dashes (-) and underscores (-) are
-removed and case is converted to lowercase. For example, the following parameter forms are
-equivalent (either in the configuration file or from the command line):</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="err">top-n,</span><span class="w"> </span><span class="err">topN,</span><span class="w"> </span><span class="err">top_n,</span><span class="w"> </span><span class="err">TOP_N,</span><span class="w"> </span><span class="err">t-o-p-N</span><span class="p">}</span><span class="w">
-</span><span class="p">{</span><span class="err">poplimit,</span><span class="w"> </span><span class="err">pop-limit,</span><span class="w"> </span><span class="err">pop-limit,</span><span class="w"> </span><span class="err">popLimit,PoPlImIt</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
-
-<p>This basically defines equivalence classes of parameters, and relieves you of the task of having to
-remember the exact format of each parameter.</p>
-
-<p>In what follows, we group the configuration parameters in the following groups:</p>
-
-<ul>
- <li><a href="#general">General options</a></li>
- <li><a href="#pruning">Pruning</a></li>
- <li><a href="#tm">Translation model options</a></li>
- <li><a href="#lm">Language model options</a></li>
- <li><a href="#output">Output options</a></li>
- <li><a href="#modes">Alternate modes of operation</a></li>
-</ul>
-
-<p><a id="general"></a></p>
-
-<h3 id="general-decoder-options">General decoder options</h3>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">c</code>, <code class="highlighter-rouge">config</code> \u2014 <em>NULL</em></p>
-
- <p>Specifies the configuration file from which Joshua options are loaded. This feature is unique in
- that it must be specified from the command line (obviously).</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">amortize</code> \u2014 <em>true</em></p>
-
- <p>When true, specifies that sorting of the rule lists at each trie node in the grammar should be
-delayed until the trie node is accessed. When false, all such nodes are sorted before decoding
-even begins. Setting to true results in slower per-sentence decoding, but allows the decoder to
-begin translating almost immediately (especially with large grammars).</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">server-port</code> \u2014 <em>0</em></p>
-
- <p>If set to a nonzero value, Joshua will start a multithreaded TCP/IP server on the specified
-port. Clients can connect to it directly through programming APIs or command-line tools like
-<code class="highlighter-rouge">telnet</code> or <code class="highlighter-rouge">nc</code>.</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>$ $JOSHUA/bin/decoder -m 30g -c /path/to/config/file -server-port 8723
-...
-$ cat input.txt | nc localhost 8723 > results.txt
-</code></pre>
- </div>
- </li>
- <li>
- <p><code class="highlighter-rouge">maxlen</code> \u2014 <em>200</em></p>
-
- <p>Input sentences longer than this are truncated.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">feature-function</code></p>
-
- <p>Enables a particular feature function. See the <a href="features.html">feature function page</a> for more information.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">oracle-file</code> \u2014 <em>NULL</em></p>
-
- <p>The location of a set of oracle reference translations, parallel to the input. When present,
-after producing the hypergraph by decoding the input sentence, the oracle is used to rescore the
-translation forest with a BLEU approximation in order to extract the oracle-translation from the
-forest. This is useful for obtaining an (approximation to an) upper bound on your translation
-model under particular search settings.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">default-nonterminal</code> \u2014 <em>\u201cX\u201d</em></p>
-
- <p>This is the nonterminal symbol assigned to out-of-vocabulary (OOV) items. Joshua assigns this
- label to every word of the input, in fact, so that even known words can be translated as OOVs, if
- the model prefers them. Usually, a very low weight on the <code class="highlighter-rouge">OOVPenalty</code> feature discourages their
- use unless necessary.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">goal-symbol</code> \u2014 <em>\u201cGOAL\u201d</em></p>
-
- <p>This is the symbol whose presence in the chart over the whole input span denotes a successful
- parse (translation). It should match the LHS nonterminal in your glue grammar. Internally,
- Joshua represents nonterminals enclosed in square brackets (e.g., \u201c[GOAL]\u201d), which you can
- optionally supply in the configuration file.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">true-oovs-only</code> \u2014 <em>false</em></p>
-
- <p>By default, Joshua creates an OOV entry for every word in the source sentence, regardless of
-whether it is found in the grammar. This allows every word to be pushed through untranslated
-(although potentially incurring a high cost based on the <code class="highlighter-rouge">OOVPenalty</code> feature). If this option is
-set, then only true OOVs are entered into the chart as OOVs. To determine \u201ctrue\u201d OOVs, Joshua
-examines the first level of the grammar trie for each word of the input (this isn\u2019t a perfect
-heuristic, since a word could be present only in deeper levels of the trie).</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">threads</code>, <code class="highlighter-rouge">num-parallel-decoders</code> \u2014 <em>1</em></p>
-
- <p>This determines how many simultaneous decoding threads to launch. </p>
-
- <p>Outputs are assembled in order and Joshua has to hold on to the complete target hypergraph until
-it is ready to be processed for output, so too many simultaneous threads could result in lots of
-memory usage if a long sentence results in many sentences being queued up. We have run Joshua
-with as many as 64 threads without any problems of this kind, but it\u2019s useful to keep in the back
-of your mind.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">weights-file</code> \u2014 NULL</p>
-
- <p>Weights are appended to the end of the Joshua configuration file, by convention. If you prefer to
-put them in a separate file, you can do so, and point to the file with this parameter.</p>
- </li>
-</ul>
-
-<h3 id="pruning-options-a-idpruning-">Pruning options <a id="pruning"></a></h3>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">pop-limit</code> \u2014 <em>100</em></p>
-
- <p>The number of cube-pruning hypotheses that are popped from the candidates list for each span of
-the input. Higher values result in a larger portion of the search space being explored at the
-cost of an increased search time. For exhaustive search, set <code class="highlighter-rouge">pop-limit</code> to 0.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">filter-grammar</code> \u2014 false</p>
-
- <p>Set to true, this enables dynamic sentence-level filtering. For each sentence, each grammar is
-filtered at runtime down to rules that can be applied to the sentence under consideration. This
-takes some time (which we haven\u2019t thoroughly quantified), but can result in the removal of many
-rules that are only partially applicable to the sentence.</p>
- </li>
- <li><code class="highlighter-rouge">constrain-parse</code> \u2014 <em>false</em></li>
- <li>
- <p><code class="highlighter-rouge">use_pos_labels</code> \u2014 <em>false</em></p>
-
- <p><em>These features are not documented.</em></p>
- </li>
-</ul>
-
-<h3 id="translation-model-options-a-idtm-">Translation model options <a id="tm"></a></h3>
-
-<p>Joshua supports any number of translation models. Conventionally, two are supplied: the main grammar
-containing translation rules, and the glue grammar for patching things together. Internally, Joshua
-doesn\u2019t distinguish between the roles of these grammars; they are treated differently only in that
-they typically have different span limits (the maximum input width they can be applied to).</p>
-
-<p>Grammars are instantiated with config file lines of the following form:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>tm = TYPE OWNER SPAN_LIMIT FILE
-</code></pre>
-</div>
-
-<ul>
- <li><code class="highlighter-rouge">TYPE</code> is the grammar type, which must be set to \u201cthrax\u201d. </li>
- <li><code class="highlighter-rouge">OWNER</code> is the grammar\u2019s owner, which defines the set of <a href="features.html">feature weights</a> that
-apply to the weights found in each line of the grammar (using different owners allows each grammar
-to have different sets and numbers of weights, while sharing owners allows weights to be shared
-across grammars).</li>
- <li><code class="highlighter-rouge">SPAN_LIMIT</code> is the maximum span of the input that rules from this grammar can be applied to. A
-span limit of 0 means \u201cno limit\u201d, while a span limit of -1 means that rules from this grammar must
-be anchored to the left side of the sentence (index 0).</li>
- <li><code class="highlighter-rouge">FILE</code> is the path to the file containing the grammar. If the file is a directory, it is assumed
-to be <a href="packed.html">packed</a>. Only one packed grammar can currently be used at a time.</li>
-</ul>
-
-<p>For reference, the following two translation model lines are used by the <a href="pipeline.html">pipeline</a>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>tm = thrax pt 20 /path/to/packed/grammar
-tm = thrax glue -1 /path/to/glue/grammar
-</code></pre>
-</div>
-
-<h3 id="language-model-options-a-idlm-">Language model options <a id="lm"></a></h3>
-
-<p>Joshua supports any number of language models. With Joshua 6.0, these
-are just regular feature functions:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>feature-function = LanguageModel -lm_file /path/to/lm/file -lm_order N -lm_type TYPE
-feature-function = StateMinimizingLanguageModel -lm_file /path/to/lm/file -lm_order N -lm_type TYPE
-</code></pre>
-</div>
-
-<p><code class="highlighter-rouge">LanguageModel</code> is a generic language model, supporting types \u2018kenlm\u2019
-(the default) and \u2018berkeleylm\u2019. <code class="highlighter-rouge">StateMinimizingLanguageModel</code>
-implements LM state minimization to reduce the size of context n-grams
-where appropriate
-(<a href="http://www.aclweb.org/anthology/W08-0402.pdf">Li and Khudanpur, 2008</a>;
-<a href="https://aclweb.org/anthology/N/N13/N13-1116.pdf">Heafield et al., 2013</a>). This
-is currently only supported by KenLM, so the <code class="highlighter-rouge">-lm_type</code> option is not
-available here.</p>
-
-<p>The other key/value pairs are defined as follows:</p>
-
-<ul>
- <li><code class="highlighter-rouge">lm_type</code>: one of \u201ckenlm\u201d \u201cberkeleylm\u201d</li>
- <li><code class="highlighter-rouge">lm_order</code>: the order of the language model</li>
- <li><code class="highlighter-rouge">lm_file</code>: the path to the language model file. All language model
- types support the standard ARPA format. Additionally, if the LM
- type is \u201ckenlm\u201d, this file can be compiled into KenLM\u2019s compiled
- format (using the program at <code class="highlighter-rouge">$JOSHUA/bin/build_binary</code>); if the
- the LM type is \u201cberkeleylm\u201d, it can be compiled by following the
- directions in
- <code class="highlighter-rouge">$JOSHUA/src/joshua/decoder/ff/lm/berkeley_lm/README</code>. The
- <a href="pipeline.html">pipeline</a> will automatically compile either type.</li>
-</ul>
-
-<p>For each language model, you need to specify a feature weight in the following format:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>lm_0 WEIGHT
-lm_1 WEIGHT
-...
-</code></pre>
-</div>
-
-<p>where the indices correspond to the order of the language model declaration lines.</p>
-
-<h3 id="output-options-a-idoutput-">Output options <a id="output"></a></h3>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">output-format</code> <em>New in 5.0</em></p>
-
- <p>Joshua prints a lot of information to STDERR (making this more granular is on the TODO
-list). Output to STDOUT is reserved for decoder translations, and is controlled by the</p>
-
- <ul>
- <li>
- <p><code class="highlighter-rouge">%i</code>: the sentence number (0-indexed)</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%e</code>: the source sentence</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%s</code>: the translated sentence</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%S</code>: the translated sentence, with some basic capitalization and denomralization. e.g.,</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>$ echo "� who you lookin' at , mr. ?" | $JOSHUA/bin/decoder -output-format "%S" -mark-oovs false 2> /dev/null
-�Who you lookin' at, Mr.?
-</code></pre>
- </div>
- </li>
- <li>
- <p><code class="highlighter-rouge">%t</code>: the target-side tree projection, all printed on one line (PTB style)</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%d</code>: the synchronous derivation, with each rules printed indented on their own lines</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%f</code>: the list of feature values (as name=value pairs)</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%c</code>: the model cost</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%w</code>: the weight vector (unimplemented)</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">%a</code>: the alignments between source and target words (currently broken for hierarchical mode)</p>
- </li>
- </ul>
-
- <p>The default value is</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>output-format = %i ||| %s ||| %f ||| %c
-</code></pre>
- </div>
-
- <p>i.e.,</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>input ID ||| translation ||| model scores ||| score
-</code></pre>
- </div>
- </li>
- <li>
- <p><code class="highlighter-rouge">top-n</code> \u2014 <em>300</em></p>
-
- <p>The number of translation hypotheses to output, sorted in decreasing order of model score</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">use-unique-nbest</code> \u2014 <em>true</em></p>
-
- <p>When constructing the n-best list for a sentence, skip hypotheses whose string has already been
-output.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">escape-trees</code> \u2014 <em>false</em></p>
- </li>
- <li>
- <p><code class="highlighter-rouge">include-align-index</code> \u2014 <em>false</em></p>
-
- <p>Output the source words indices that each target word aligns to.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">mark-oovs</code> \u2014 <em>false</em></p>
-
- <p>if <code class="highlighter-rouge">true</code>, this causes the text \u201c_OOV\u201d to be appended to each untranslated word in the output.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">visualize-hypergraph</code> \u2014 <em>false</em></p>
-
- <p>If set to true, a visualization of the hypergraph will be displayed, though you will have to
-explicitly include the relevant jar files. See the example usage in
-<code class="highlighter-rouge">$JOSHUA/examples/tree_visualizer/</code>, which contains a demonstration of a source sentence,
-translation, and synchronous derivation.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">dump-hypergraph</code> \u2014 \u201c\u201d</p>
-
- <p>This feature directs that the hypergraph should be written to disk for each input sentence. If
-set, the value should contain the string \u201c%d\u201d, which is replaced with the sentence number. For
-example,</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>cat input.txt | $JOSHUA/bin/decoder -dump-hypergraph hgs/%d.txt
-</code></pre>
- </div>
-
- <p>Note that the output directory must exist.</p>
-
- <p>TODO: revive the
-<a href="http://aclweb.org/aclwiki/index.php?title=Hypergraph_Format">discussion on a common hypergraph format</a>
-on the ACL Wiki and support that format.</p>
- </li>
-</ul>
-
-<h3 id="lattice-decoding">Lattice decoding</h3>
-
-<p>In addition to regular sentences, Joshua can decode weighted lattices encoded in
-<a href="http://www.statmt.org/moses/?n=Moses.WordLattices">the PLF format</a>, except that path costs should
-be listed as <b>log probabilities</b> instead of probabilities. Lattice decoding was originally
-added by Lane Schwartz and <a href="http://www.cs.cmu.edu/~cdyer/">Chris Dyer</a>.</p>
-
-<p>Joshua will automatically detect whether the input sentence is a regular sentence (the usual case)
-or a lattice. If a lattice, a feature will be activated that accumulates the cost of different
-paths through the lattice. In this case, you need to ensure that a weight for this feature is
-present in <a href="decoder.html">your model file</a>. The <a href="pipeline.html">pipeline</a> will handle this
-automatically, or if you are doing this manually, you can add the line</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>SourcePath COST
-</code></pre>
-</div>
-
-<p>to your Joshua configuration file. </p>
-
-<p>Lattices must be listed one per line.</p>
-
-<h3 id="alternate-modes-of-operation-a-idmodes-">Alternate modes of operation <a id="modes"></a></h3>
-
-<p>In addition to decoding input sentences in the standard way, Joshua supports both <em>constrained
-decoding</em> and <em>synchronous parsing</em>. In both settings, both the source and target sides are provided
-as input, and the decoder finds a derivation between them.</p>
-
-<h4 id="constrained-decoding">Constrained decoding</h4>
-
-<p>To enable constrained decoding, simply append the desired target string as part of the input, in
-the following format:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>source sentence ||| target sentence
-</code></pre>
-</div>
-
-<p>Joshua will translate the source sentence constrained to the target sentence. There are a few
-caveats:</p>
-
-<ul>
- <li>
- <p>Left-state minimization cannot be enabled for the language model</p>
- </li>
- <li>
- <p>A heuristic is used to constrain the derivation (the LM state must match against the
-input). This is not a perfect heuristic, and sometimes results in analyses that are not
-perfectly constrained to the input, but have extra words.</p>
- </li>
-</ul>
-
-<h4 id="synchronous-parsing">Synchronous parsing</h4>
-
-<p>Joshua supports synchronous parsing as a two-step sequence of monolingual parses, as described in
-Dyer (NAACL 2010) (<a href="http://www.aclweb.org/anthology/N10-1033\u200e.pdf">PDF</a>). To enable this:</p>
-
-<ul>
- <li>
- <p>Set the configuration parameter <code class="highlighter-rouge">parse = true</code>.</p>
- </li>
- <li>
- <p>Remove all language models from the input file </p>
- </li>
- <li>
- <p>Provide input in the following format:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code> source sentence ||| target sentence
-</code></pre>
- </div>
- </li>
-</ul>
-
-<p>You may also wish to display the synchronouse parse tree (<code class="highlighter-rouge">-output-format %t</code>) and the alignment
-(<code class="highlighter-rouge">-show-align-index</code>).</p>
-
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/faq.html
----------------------------------------------------------------------
diff --git a/6/faq.html b/6/faq.html
deleted file mode 100644
index 8db5143..0000000
--- a/6/faq.html
+++ /dev/null
@@ -1,376 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Frequently Asked Questions</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Frequently Asked Questions</h2>
- </div>
-
- <div class="blog-post">
-
- <p>Solutions to common problems will be posted here as we become aware of
-them. If you need help with something, please check
-<a href="https://groups.google.com/forum/#!forum/joshua_support">our support group</a>
-for a solution, or
-<a href="https://groups.google.com/forum/#!newtopic/joshua_support">post a new question</a>.</p>
-
-<h3 id="i-get-a-message-stating-no-ken-in-javalibrarypath">I get a message stating: \u201cno ken in java.library.path\u201d</h3>
-
-<p>This occurs when <a href="https://kheafield.com/code/kenlm/">KenLM</a> failed to
-build. This can occur for a number of reasons:</p>
-
-<ul>
- <li>
- <p><a href="http://www.boost.org/">Boost</a> isn\u2019t installed. Boost is
-available through most package management tools, so try that
-first. You can also build it from source.</p>
- </li>
- <li>
- <p>Boost is installed, but not in your path. The easiest solution is
-to add the boost library directory to your <code class="highlighter-rouge">$LD_LIBRARY_PATH</code>
-environment variable. You can also edit the file
-<code class="highlighter-rouge">$JOSHUA/src/joshua/decoder/ff/lm/kenlm/Makefile</code> and define
-<code class="highlighter-rouge">BOOST_ROOT</code> to point to your boost location. Then rebuild KenLM
-with the command</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>ant -f $JOSHUA/build.xml kenlm
-</code></pre>
- </div>
- </li>
- <li>
- <p>You have run into boost\u2019s weird naming of multi-threaded
-libraries. For some reason, boost libraries sometimes have a
-<code class="highlighter-rouge">-mt</code> extension applied when they are built with multi-threaded
-support. This will cause the linker to fail, since it is looking
-for, e.g., <code class="highlighter-rouge">-lboost_system</code> instead of <code class="highlighter-rouge">-lboost_system-mt</code>. Edit
-the same Makefile as above and uncomment the <code class="highlighter-rouge">BOOST_MT = -mt</code>
-line, then try to compile again with</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>ant -f $JOSHUA.build.xml kenlm
-</code></pre>
- </div>
- </li>
-</ul>
-
-<p>You may find the following reference URLs to be useful.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>https://groups.google.com/forum/#!topic/joshua_support/SiGO41tkpsw
-http://stackoverflow.com/questions/12583080/c-library-in-using-boost-library
-</code></pre>
-</div>
-
-<h3 id="how-do-i-make-joshua-produce-better-results">How do I make Joshua produce better results?</h3>
-
-<p>One way is to add a larger language model. Build on Gigaword, news
-crawl data, etc. <code class="highlighter-rouge">lmplz</code> makes it easy to build and efficient to
-represent (especially if you compress it with `build_binary). To
-include it in Joshua, there are two ways:</p>
-
-<ul>
- <li>
- <p><em>Pipeline</em>. By default, Joshua\u2019s pipeline builds a language
- model on the target side of your parallel training data. But
- Joshua can decode with any number of additional language models
- as well. So you can build a language model separately,
- presumably on much more data (since you won\u2019t be constrained
- only to one side of parallel data, which is much more scarce
- than monolingual data). Once you\u2019ve built extra language models
- and compiled them with KenLM\u2019s <code class="highlighter-rouge">build_binary</code> script, you can
- tell the pipeline to use them with any number of <code class="highlighter-rouge">--lmfile
- /path/to/lm/file</code> flags.</p>
- </li>
- <li>
- <p><em>Joshua</em> (directly).
- <a href="http://localhost:4000/6.0/file-formats.html">This file</a>
- documents the Joshua configuration file format.</p>
- </li>
-</ul>
-
-<h3 id="i-have-already-run-the-pipeline-once-how-do-i-run-it-again-skipping-the-early-stages-and-just-retuning-the-model">I have already run the pipeline once. How do I run it again, skipping the early stages and just retuning the model?</h3>
-
-<p>You would need to do this if, for example, you added a language
-model, or changed some other parameter (e.g., an improvement to the
-decoder). To do this, follow the following steps:</p>
-
-<ul>
- <li>Re-run the pipeline giving it a new <code class="highlighter-rouge">--rundir N+1</code> (where <code class="highlighter-rouge">N</code> is the last
-run, and <code class="highlighter-rouge">N+1</code> is a new, non-existent directory). </li>
- <li>Give it all the other flags that you gave before, such as the
-tuning data, testing data, source and target flags, etc. You
-don\u2019t have to give it the training data.</li>
- <li>Tell it to start at the tuning step with <code class="highlighter-rouge">--first-step TUNE</code></li>
- <li>Tell it where all of your language model files are with <code class="highlighter-rouge">--lmfile
-/path/to/lm</code> lines. You also have to tell it where the main
-language model is, which is usually <code class="highlighter-rouge">--lmfile N/lm.kenlm</code> (paths
-are relative to the directory above the run directory.</li>
- <li>Tell it where the main grammar is, e.g., <code class="highlighter-rouge">--grammar
-N/grammar.gz</code>. If the tuning and test data hasn\u2019t changed, you
-can also point it to the filtered and packed versions to save a
-little time using <code class="highlighter-rouge">--tune-grammar N/data/tune/grammar.packed</code> and
-<code class="highlighter-rouge">--test-grammar N/data/test/grammar.packed</code>, where <code class="highlighter-rouge">N</code> here again
-is the previous run (or some other run; it can be anywhere).</li>
-</ul>
-
-<p>Here\u2019s an example. Let\u2019s say you ran a full pipeline as run 1, and
-now added a new language model and want to see how it affects the
-decoder. Your first run might have been invoked like this:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/scripts/training/pipeline.pl \
- --rundir 1 \
- --readme "Baseline French--English Europarl hiero system" \
- --corpus /path/to/europarl \
- --tune /path/to/europarl/tune \
- --test /path/to/europarl/test \
- --source fr \
- --target en \
- --threads 8 \
- --joshua-mem 30g \
- --tuner mira \
- --type hiero \
- --aligner berkeley
-</code></pre>
-</div>
-
-<p>Your new run will look like this:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/scripts/training/pipeline.pl \
- --rundir 2 \
- --readme "Adding in a huge language model" \
- --tune /path/to/europarl/tune \
- --test /path/to/europarl/test \
- --source fr \
- --target en \
- --threads 8 \
- --joshua-mem 30g \
- --tuner mira \
- --type hiero \
- --aligner berkeley \
- --first-step TUNE \
- --lmfile 1/lm.kenlm \
- --lmfile /path/to/huge/new/lm \
- --tune-grammar 1/data/tune/grammar.packed \
- --test-grammar 1/data/test/grammar.packed
-</code></pre>
-</div>
-
-<p>Notice the changes: we removed the <code class="highlighter-rouge">--corpus</code> (though it would have
-been fine to have left it, it would have just been skipped),
-specified the first step, changed the run directory and README
-comments, and pointed to the grammars and <em>both</em> language model files.</p>
-
-<p>How can I enable specific feature functions?</p>
-
-<p>Let\u2019s say you created a new feature function, <code class="highlighter-rouge">OracleFeature</code>, and
-you want to enable it. You can do this in two ways. Through the
-pipeline, simply pass it the argument <code class="highlighter-rouge">--joshua-args "list of
-joshua args"</code>. These will then be passed to the decoder when it is
-invoked. You can enable your feature functions, then using
-something like</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/pipeline.pl --joshua-args '-feature-function OracleFeature'
-</code></pre>
-</div>
-
-<p>If you call the decoder directly, you can just put that line in
-the configuration file, e.g.,</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>feature-function = OracleFeature
-</code></pre>
-</div>
-
-<p>or you can pass it directly to Joshua on the command line using
-the standard notation, e.g.,</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/joshua-decoder -feature-function OracleFeature
-</code></pre>
-</div>
-
-<p>These could be stacked, e.g.,</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/joshua-decoder -feature-function OracleFeature \
- -feature-function MagicFeature \
- -feature-function MTSolverFeature \
- ...
-</code></pre>
-</div>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
[3/4] incubator-joshua-site git commit: Hid old documentation,
pointed to wiki
Posted by mj...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/features.html
----------------------------------------------------------------------
diff --git a/6/features.html b/6/features.html
deleted file mode 100644
index 6e617cf..0000000
--- a/6/features.html
+++ /dev/null
@@ -1,192 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Features</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Features</h2>
- </div>
-
- <div class="blog-post">
-
- <p>Joshua 5.0 uses a sparse feature representation to encode features internally.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/file-formats.html
----------------------------------------------------------------------
diff --git a/6/file-formats.html b/6/file-formats.html
deleted file mode 100644
index 4918253..0000000
--- a/6/file-formats.html
+++ /dev/null
@@ -1,270 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Joshua file formats</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Joshua file formats</h2>
- </div>
-
- <div class="blog-post">
-
- <p>This page describes the formats of Joshua configuration and support files.</p>
-
-<h2 id="translation-models-grammars">Translation models (grammars)</h2>
-
-<p>Joshua supports two grammar file formats: a text-based version (also used by Hiero, shared by
-<a href="">cdec</a>, and supported by <a href="">hierarchical Moses</a>), and an efficient
-<a href="packing.html">packed representation</a> developed by <a href="http://cs.jhu.edu/~juri">Juri Ganitkevich</a>.</p>
-
-<p>Grammar rules follow this format.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>[LHS] ||| SOURCE-SIDE ||| TARGET-SIDE ||| FEATURES
-</code></pre>
-</div>
-
-<p>The source and target sides contain a mixture of terminals and nonterminals. The nonterminals are
-linked across sides by indices. There is no limit to the number of paired nonterminals in the rule
-or on the nonterminal labels (Joshua supports decoding with SAMT and GHKM grammars).</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>[X] ||| el chico [X,1] ||| the boy [X,1] ||| -3.14 0 2 17
-[S] ||| el chico [VP,1] ||| the boy [VP,1] ||| -3.14 0 2 17
-[VP] ||| [NP,1] [IN,2] [VB,3] ||| [VB,3] [IN,2] [NP,1] ||| 0.0019026637 0.81322956
-</code></pre>
-</div>
-
-<p>The feature values can have optional labels, e.g.:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>[X] ||| el chico [X,1] ||| the boy [X,1] ||| lexprob=-3.14 lexicalized=1 numwords=2 count=17
-</code></pre>
-</div>
-
-<p>One file common to decoding is the glue grammar, which for hiero grammar is defined as follows:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>[GOAL] ||| <s> ||| <s> ||| 0
-[GOAL] ||| [GOAL,1] [X,2] ||| [GOAL,1] [X,2] ||| -1
-[GOAL] ||| [GOAL,1] </s> ||| [GOAL,1] </s> ||| 0
-</code></pre>
-</div>
-
-<p>Joshua\u2019s <a href="pipeline.html">pipeline</a> supports extraction of Hiero and SAMT grammars via
-<a href="thrax.html">Thrax</a> or GHKM grammars using <a href="http://www-nlp.stanford.edu/~mgalley/">Michel Galley</a>\u2019s
-GHKM extractor (included) or Moses\u2019 GHKM extractor (if Moses is installed).</p>
-
-<h2 id="language-model">Language Model</h2>
-
-<p>Joshua has two language model implementations: <a href="http://kheafield.com/code/kenlm/">KenLM</a> and
-<a href="http://berkeleylm.googlecode.com">BerkeleyLM</a>. All language model implementations support the
-standard ARPA format output by <a href="http://www.speech.sri.com/projects/srilm/">SRILM</a>. In addition,
-KenLM and BerkeleyLM support compiled formats that can be loaded more quickly and efficiently. KenLM
-is written in C++ and is supported via a JNI bridge, while BerkeleyLM is written in Java. KenLM is
-the default because of its support for left-state minimization.</p>
-
-<h3 id="compiling-for-kenlm">Compiling for KenLM</h3>
-
-<p>To compile an ARPA grammar for KenLM, use the (provided) <code class="highlighter-rouge">build-binary</code> command, located deep within
-the Joshua source code:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/bin/build_binary lm.arpa lm.kenlm
-</code></pre>
-</div>
-
-<p>This script takes the <code class="highlighter-rouge">lm.arpa</code> file and produces the compiled version in <code class="highlighter-rouge">lm.kenlm</code>.</p>
-
-<h3 id="compiling-for-berkeleylm">Compiling for BerkeleyLM</h3>
-
-<p>To compile a grammar for BerkeleyLM, type:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>java -cp $JOSHUA/lib/berkeleylm.jar -server -mxMEM edu.berkeley.nlp.lm.io.MakeLmBinaryFromArpa lm.arpa lm.berkeleylm
-</code></pre>
-</div>
-
-<p>The <code class="highlighter-rouge">lm.berkeleylm</code> file can then be listed directly in the <a href="decoder.html">Joshua configuration file</a>.</p>
-
-<h2 id="joshua-configuration-file">Joshua configuration file</h2>
-
-<p>The <a href="decoder.html">decoder page</a> documents decoder command-line and config file options.</p>
-
-<h2 id="thrax-configuration">Thrax configuration</h2>
-
-<p>See <a href="thrax.html">the thrax page</a> for more information about the Thrax configuration file.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/index.html
----------------------------------------------------------------------
diff --git a/6/index.html b/6/index.html
deleted file mode 100644
index 7392541..0000000
--- a/6/index.html
+++ /dev/null
@@ -1,210 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Joshua documentation</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Joshua documentation</h2>
- </div>
-
- <div class="blog-post">
-
- <p>This page contains end-user oriented documentation for the 6.0 release of
-<a href="http://joshua-decoder.org/">the Joshua decoder</a>.</p>
-
-<p>To navigate the documentation, use the links on the navigation bar to
-the left. For more detail on the decoder itself, including its command-line options, see
-<a href="decoder.html">the Joshua decoder page</a>. You can also learn more about other steps of
-<a href="pipeline.html">the Joshua MT pipeline</a>, including <a href="thrax.html">grammar extraction</a> with Thrax and
-Joshua\u2019s <a href="packing.html">efficient grammar representation</a>.</p>
-
-<p>A <a href="bundle.html">bundled configuration</a>, which is a minimal set of configuration, resource, and script files, can be created and easily transferred and shared.</p>
-
-<h2 id="development">Development</h2>
-
-<p>For developer support, please consult <a href="http://cs.jhu.edu/~post/joshua-docs">the javadoc documentation</a> and the <a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Joshua developers mailing list</a>.</p>
-
-<h2 id="support">Support</h2>
-
-<p>If you have problems or issues, you might find some help <a href="faq.html">on our answers page</a> or
-<a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_support">in the mailing list archives</a>.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/install.html
----------------------------------------------------------------------
diff --git a/6/install.html b/6/install.html
deleted file mode 100644
index b972e81..0000000
--- a/6/install.html
+++ /dev/null
@@ -1,301 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Installation</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Installation</h2>
- </div>
-
- <div class="blog-post">
-
- <h3 id="download-and-install">Download and install</h3>
-
-<p>To use Joshua as a standalone decoder (with <a href="/language-packs/">language packs</a>), you only need to download and install the runtime version of the decoder.
-If you also wish to build translation models from your own data, you will want to install the full version.
-See the instructions below.</p>
-
-<ol>
- <li>
- <p>Set up some basic environment variables.
-You need to define <code class="highlighter-rouge">$JAVA_HOME</code></p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>export JAVA_HOME=/path/to/java
-
-# JAVA_HOME is not very standardized. Here are some places to look:
-# OS X: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
-# Linux: export JAVA_HOME=/usr/java/default
-</code></pre>
- </div>
- </li>
- <li>
- <p>If you are installing the full version of Joshua, you also need to define <code class="highlighter-rouge">$HADOOP</code> to point to your Hadoop installation.
-(Joshua looks for the Hadoop executuble in <code class="highlighter-rouge">$HADOOP/bin/hadoop</code>)</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>export HADOOP=/usr
-</code></pre>
- </div>
-
- <p>If you don\u2019t have a Hadoop installation, <a href="pipeline.html">Joshua\u2019s pipeline</a> can install a standalone version for you.</p>
- </li>
- <li>
- <p>To install just the runtime version of Joshua, type</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>wget -q http://cs.jhu.edu/~post/files/joshua-runtime-6.0.5.tgz
-</code></pre>
- </div>
-
- <p>Then build everything</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>tar xzf joshua-runtime-6.0.5.tgz
-cd joshua-runtime-6.0.5
-
-# Add this to your init files
-export JOSHUA=$(pwd)
-
-# build everything
-ant
-</code></pre>
- </div>
- </li>
- <li>
- <p>To instead install the full version, type</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code>wget -q http://cs.jhu.edu/~post/files/joshua-6.0.5.tgz
-
-tar xzf joshua-6.0.5.tgz
-cd joshua-6.0.5
-
-# Add this to your init files
-export JOSHUA=$(pwd)
-
-# build everything
-ant
-</code></pre>
- </div>
- </li>
-</ol>
-
-<h3 id="building-new-models">Building new models</h3>
-
-<p>If you wish to build models for new language pairs from existing data (such as the <a href="http://statmt.org/wmt14/">WMT data</a>), you need to install some additional dependencies.</p>
-
-<ol>
- <li>
- <p>For learning hierarchical models, Joshua includes a tool called <a href="thrax.html">Thrax</a>, which
-is built on Hadoop. If you have a Hadoop installation, make sure that the environment variable
-<code class="highlighter-rouge">$HADOOP</code> is set and points to it. If you don\u2019t, Joshua will roll one out for you in standalone
-mode. Hadoop is only needed if you plan to build new models with Joshua.</p>
- </li>
- <li>
- <p>You will need to install Moses if either of the following applies to you:</p>
-
- <ul>
- <li>
- <p>You wish to build <a href="phrase.html">phrase-based models</a> (Joshua 6 includes a phrase-based
-decoder, but not the tools for building such a model)</p>
- </li>
- <li>
- <p>You are building your own models (phrase- or syntax-based) and wish to use Cherry & Foster\u2019s
-<a href="http://aclweb.org/anthology-new/N/N12/N12-1047v2.pdf">batch MIRA tuner</a> instead of the included
-MERT implementation, <a href="zmert.html">Z-MERT</a>. </p>
- </li>
- </ul>
-
- <p>Follow <a href="http://www.statmt.org/moses/?n=Development.GetStarted">the instructions for installing Moses
-here</a>, and then define the <code class="highlighter-rouge">$MOSES</code>
-environment variable to point to the root of the Moses installation.</p>
- </li>
-</ol>
-
-<h2 id="more-information">More information</h2>
-
-<p>For more detail on the decoder itself, including its command-line options, see
-<a href="decoder.html">the Joshua decoder page</a>. You can also learn more about other steps of
-<a href="pipeline.html">the Joshua MT pipeline</a>, including <a href="thrax.html">grammar extraction</a> with Thrax and
-Joshua\u2019s <a href="packing.html">efficient grammar representation</a>.</p>
-
-<p>If you have problems or issues, you might find some help <a href="faq.html">on our answers page</a> or
-<a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_support">in the mailing list archives</a>.</p>
-
-<p>A <a href="bundle.html">bundled configuration</a>, which is a minimal set of configuration, resource, and script files, can be created and easily transferred and shared.</p>
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/jacana.html
----------------------------------------------------------------------
diff --git a/6/jacana.html b/6/jacana.html
deleted file mode 100644
index b8f5a79..0000000
--- a/6/jacana.html
+++ /dev/null
@@ -1,331 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Alignment with Jacana</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Alignment with Jacana</h2>
- </div>
-
- <div class="blog-post">
-
- <h2 id="introduction">Introduction</h2>
-
-<p>jacana-xy is a token-based word aligner for machine translation, adapted from the original
-English-English word aligner jacana-align described in the following paper:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>A Lightweight and High Performance Monolingual Word Aligner. Xuchen Yao, Benjamin Van Durme,
-Chris Callison-Burch and Peter Clark. Proceedings of ACL 2013, short papers.
-</code></pre>
-</div>
-
-<p>It currently supports only aligning from French to English with a very limited feature set, from the
-one week hack at the <a href="http://statmt.org/mtm13">Eighth MT Marathon 2013</a>. Please feel free to check
-out the code, read to the bottom of this page, and
-<a href="http://www.cs.jhu.edu/~xuchen/">send the author an email</a> if you want to add more language pairs to
-it.</p>
-
-<h2 id="build">Build</h2>
-
-<p>jacana-xy is written in a mixture of Java and Scala. If you build from ant, you have to set up the
-environmental variables <code class="highlighter-rouge">JAVA_HOME</code> and <code class="highlighter-rouge">SCALA_HOME</code>. In my system, I have:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.26
-export SCALA_HOME=/home/xuchen/Downloads/scala-2.10.2
-</code></pre>
-</div>
-
-<p>Then type:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>ant
-</code></pre>
-</div>
-
-<p>build/lib/jacana-xy.jar will be built for you.</p>
-
-<p>If you build from Eclipse, first install scala-ide, then import the whole jacana folder as a Scala project. Eclipse should find the .project file and set up the project automatically for you.</p>
-
-<p>Demo
-scripts-align/runDemoServer.sh shows up the web demo. Direct your browser to http://localhost:8080/ and you should be able to align some sentences.</p>
-
-<p>Note: To make jacana-xy know where to look for resource files, pass the property JACANA_HOME with Java when you run it:</p>
-
-<p>java -DJACANA_HOME=/path/to/jacana -cp jacana-xy.jar \u2026\u2026</p>
-
-<p>Browser
-You can also browse one or two alignment files (*.json) with firefox opening src/web/AlignmentBrowser.html:</p>
-
-<p>Note 1: due to strict security setting for accessing local files, Chrome/IE won\u2019t work.</p>
-
-<p>Note 2: the input *.json files have to be in the same folder with AlignmentBrowser.html.</p>
-
-<p>Align
-scripts-align/alignFile.sh aligns tab-separated sentence files and outputs the output to a .json file that\u2019s accepted by the browser:</p>
-
-<p>java -DJACANA_HOME=../ -jar ../build/lib/jacana-xy.jar -src fr -tgt en -m fr-en.model -a s.txt -o s.json</p>
-
-<p>scripts-align/alignFile.sh takes GIZA++-style input files (one file containing the source sentences, and the other file the target sentences) and outputs to one .align file with dashed alignment indices (e.g. \u201c1-2 0-4\u201d):</p>
-
-<p>java -DJACANA_HOME=../ -jar ../build/lib/jacana-xy.jar -m fr-en.model -src fr -tgt en -a s1.txt -b s2.txt -o s.align</p>
-
-<p>Training
-java -DJACANA_HOME=../ -jar ../build/lib/jacana-xy.jar -r train.json -d dev.json -t test.json -m /tmp/align.model</p>
-
-<p>The aligner then would train on train.json, and report F1 values on dev.json for every 10 iterations, when the stopping criterion has reached, it will test on test.json.</p>
-
-<p>For every 10 iterations, a model file is saved to (in this example) /tmp/align.model.iter_XX.F1_XX.X. Normally what I do is to select the one with the best F1 on dev.json, then run a final test on test.json:</p>
-
-<p>java -DJACANA_HOME=../ -jar ../build/lib/jacana-xy.jar -t test.json -m /tmp/align.model.iter_XX.F1_XX.X</p>
-
-<p>In this case since the training data is missing, the aligner assumes it\u2019s a test job, then reads model file still from the -m option, and test on test.json.</p>
-
-<p>All the json files are in a format like the following (also accepted by the browser for display):</p>
-
-<p>[
- {
- \u201cid\u201d: \u201c0008\u201d,
- \u201cname\u201d: \u201cHansards.french-english.0008\u201d,
- \u201cpossibleAlign\u201d: \u201c0-0 0-1 0-2\u201d,
- \u201csource\u201d: \u201cbravo !\u201d,
- \u201csureAlign\u201d: \u201c1-3\u201d,
- \u201ctarget\u201d: \u201chear , hear !\u201d
- },
- {
- \u201cid\u201d: \u201c0009\u201d,
- \u201cname\u201d: \u201cHansards.french-english.0009\u201d,
- \u201cpossibleAlign\u201d: \u201c1-1 6-5 7-5 6-6 7-6 13-10 13-11\u201d,
- \u201csource\u201d: \u201cmonsieur le Orateur , ma question se adresse � le ministre charg� de les transports .\u201d,
- \u201csureAlign\u201d: \u201c0-0 2-1 3-2 4-3 5-4 8-7 9-8 10-9 12-10 14-11 15-12\u201d,
- \u201ctarget\u201d: \u201cMr. Speaker , my question is directed to the Minister of Transport .\u201d
- }
-]
-Where possibleAlign is not used.</p>
-
-<p>The stopping criterion is to run up to 300 iterations or when the objective difference between two iterations is less than 0.001, whichever happens first. Currently they are hard-coded. If you need to be flexible on this, send me an email!</p>
-
-<p>Support More Languages
-To add support to more languages, you need:</p>
-
-<p>labelled word alignment (in the download there\u2019s already French-English under alignment-data/fr-en; I also have Chinese-English and Arabic-English; let me know if you have more). Usually 100 labelled sentence pairs would be enough
-implement some feature functions for this language pair
-To add more features, you need to implement the following interface:</p>
-
-<p>edu.jhu.jacana.align.feature.AlignFeature</p>
-
-<p>and override the following function:</p>
-
-<p>addPhraseBasedFeature</p>
-
-<p>For instance, a simple feature that checks whether the two words are translations in wiktionary for the French-English alignment task has the function implemented as:</p>
-
-<p>def addPhraseBasedFeature(pair: AlignPair, ins:AlignFeatureVector, i:Int, srcSpan:Int, j:Int, tgtSpan:Int,
- currState:Int, featureAlphabet: Alphabet){
- if (j == -1) {
- } else {
- val srcTokens = pair.srcTokens.slice(i, i+srcSpan).mkString(\u201c \u201c)
- val tgtTokens = pair.tgtTokens.slice(j, j+tgtSpan).mkString(\u201c \u201c)</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>if (WiktionaryMultilingual.exists(srcTokens, tgtTokens)) {
- ins.addFeature("InWiktionary", NONE_STATE, currState, 1.0, srcSpan, featureAlphabet)
-}
-</code></pre>
-</div>
-
-<p>} <br />
-}
-This is a more general function that also deals with phrase alignment. But it is suggested to implement it just for token alignment as currently the phrase alignment part is very slow to train (60x slower than token alignment).</p>
-
-<p>Some other language-independent and English-only features are implemented under the package edu.jhu.jacana.align.feature, for instance:</p>
-
-<p>StringSimilarityAlignFeature: various string similarity measures</p>
-
-<p>PositionalAlignFeature: features based on relative sentence positions</p>
-
-<p>DistortionAlignFeature: Markovian (state transition) features</p>
-
-<p>When you add features for more languages, just create a new package like the one for French-English:</p>
-
-<p>edu.jhu.jacana.align.feature.fr_en</p>
-
-<p>and start coding!</p>
-
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/22be73ab/6/large-lms.html
----------------------------------------------------------------------
diff --git a/6/large-lms.html b/6/large-lms.html
deleted file mode 100644
index edf4878..0000000
--- a/6/large-lms.html
+++ /dev/null
@@ -1,390 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="description" content="">
- <meta name="author" content="">
- <link rel="icon" href="../../favicon.ico">
-
- <title>Joshua Documentation | Building large LMs with SRILM</title>
-
- <!-- Bootstrap core CSS -->
- <link href="/dist/css/bootstrap.min.css" rel="stylesheet">
-
- <!-- Custom styles for this template -->
- <link href="/joshua6.css" rel="stylesheet">
- </head>
-
- <body>
-
- <div class="blog-masthead">
- <div class="container">
- <nav class="blog-nav">
- <!-- <a class="blog-nav-item active" href="#">Joshua</a> -->
- <a class="blog-nav-item" href="/">Joshua</a>
- <!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
- <a class="blog-nav-item" href="/language-packs/">Language packs</a>
- <a class="blog-nav-item" href="/data/">Datasets</a>
- <a class="blog-nav-item" href="/support/">Support</a>
- <a class="blog-nav-item" href="/contributors.html">Contributors</a>
- </nav>
- </div>
- </div>
-
- <div class="container">
-
- <div class="row">
-
- <div class="col-sm-2">
- <div class="sidebar-module">
- <!-- <h4>About</h4> -->
- <center>
- <img src="/images/joshua-logo-small.png" />
- <p>Joshua machine translation toolkit</p>
- </center>
- </div>
- <hr>
- <center>
- <a href="/releases/current/" target="_blank"><button class="button">Download Joshua 6.0.5</button></a>
- <br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
- <p>Released November 5, 2015</p>
- </center>
- <hr>
- <!-- <div class="sidebar-module"> -->
- <!-- <span id="download"> -->
- <!-- <a href="http://joshua-decoder.org/downloads/joshua-6.0.tgz">Download</a> -->
- <!-- </span> -->
- <!-- </div> -->
- <div class="sidebar-module">
- <h4>Using Joshua</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/install.html">Installation</a></li>
- <li><a href="/6.0/quick-start.html">Quick Start</a></li>
- </ol>
- </div>
- <hr>
- <div class="sidebar-module">
- <h4>Building new models</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/pipeline.html">Pipeline</a></li>
- <li><a href="/6.0/tutorial.html">Tutorial</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
- </ol>
- </div>
-<!--
- <div class="sidebar-module">
- <h4>Phrase-based</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/phrase.html">Training</a></li>
- </ol>
- </div>
--->
- <hr>
- <div class="sidebar-module">
- <h4>Advanced</h4>
- <ol class="list-unstyled">
- <li><a href="/6.0/bundle.html">Building language packs</a></li>
- <li><a href="/6.0/decoder.html">Decoder options</a></li>
- <li><a href="/6.0/file-formats.html">File formats</a></li>
- <li><a href="/6.0/packing.html">Packing TMs</a></li>
- <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
- </ol>
- </div>
-
- <hr>
- <div class="sidebar-module">
- <h4>Developer</h4>
- <ol class="list-unstyled">
- <li><a href="https://github.com/joshua-decoder/joshua">Github</a></li>
- <li><a href="http://cs.jhu.edu/~post/joshua-docs">Javadoc</a></li>
- <li><a href="https://groups.google.com/forum/?fromgroups#!forum/joshua_developers">Mailing list</a></li>
- </ol>
- </div>
-
- </div><!-- /.blog-sidebar -->
-
-
- <div class="col-sm-8 blog-main">
-
-
- <div class="blog-title">
- <h2>Building large LMs with SRILM</h2>
- </div>
-
- <div class="blog-post">
-
- <p>The following is a tutorial for building a large language model from the
-English Gigaword Fifth Edition corpus
-<a href="http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2011T07">LDC2011T07</a>
-using SRILM. English text is provided from seven different sources.</p>
-
-<h3 id="step-0-clean-up-the-corpus">Step 0: Clean up the corpus</h3>
-
-<p>The Gigaword corpus has to be stripped of all SGML tags and tokenized.
-Instructions for performing those steps are not included in this
-documentation. A description of this process can be found in a paper
-called <a href="https://akbcwekex2012.files.wordpress.com/2012/05/28_paper.pdf">\u201cAnnotated
-Gigaword\u201d</a>.</p>
-
-<p>The Joshua package ships with a script that converts all alphabetical
-characters to their lowercase equivalent. The script is located at
-<code class="highlighter-rouge">$JOSHUA/scripts/lowercase.perl</code>.</p>
-
-<p>Make a directory structure as follows:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>gigaword/
-\u251c\u2500\u2500 corpus/
-\u2502�� \u251c\u2500\u2500 afp_eng/
-\u2502�� \u2502�� \u251c\u2500\u2500 afp_eng_199405.lc.gz
-\u2502�� \u2502�� \u251c\u2500\u2500 afp_eng_199406.lc.gz
-\u2502�� \u2502�� \u251c\u2500\u2500 ...
-\u2502�� \u2502�� \u2514\u2500\u2500 counts/
-\u2502�� \u251c\u2500\u2500 apw_eng/
-\u2502�� \u2502�� \u251c\u2500\u2500 apw_eng_199411.lc.gz
-\u2502�� \u2502�� \u251c\u2500\u2500 apw_eng_199412.lc.gz
-\u2502�� \u2502�� \u251c\u2500\u2500 ...
-\u2502�� \u2502�� \u2514\u2500\u2500 counts/
-\u2502�� \u251c\u2500\u2500 cna_eng/
-\u2502�� \u2502�� \u251c\u2500\u2500 ...
-\u2502�� \u2502�� \u2514\u2500\u2500 counts/
-\u2502�� \u251c\u2500\u2500 ltw_eng/
-\u2502�� \u2502�� \u251c\u2500\u2500 ...
-\u2502�� \u2502�� \u2514\u2500\u2500 counts/
-\u2502�� \u251c\u2500\u2500 nyt_eng/
-\u2502�� \u2502�� \u251c\u2500\u2500 ...
-\u2502�� \u2502�� \u2514\u2500\u2500 counts/
-\u2502�� \u251c\u2500\u2500 wpb_eng/
-\u2502�� \u2502�� \u251c\u2500\u2500 ...
-\u2502�� \u2502�� \u2514\u2500\u2500 counts/
-\u2502�� \u2514\u2500\u2500 xin_eng/
-\u2502�� �� \u251c\u2500\u2500 ...
-\u2502�� �� \u2514\u2500\u2500 counts/
-\u2514\u2500\u2500 lm/
- �� \u251c\u2500\u2500 afp_eng/
- �� \u251c\u2500\u2500 apw_eng/
- �� \u251c\u2500\u2500 cna_eng/
- �� \u251c\u2500\u2500 ltw_eng/
- �� \u251c\u2500\u2500 nyt_eng/
- �� \u251c\u2500\u2500 wpb_eng/
- �� \u2514\u2500\u2500 xin_eng/
-</code></pre>
-</div>
-
-<p>The next step will be to build smaller LMs and then interpolate them into one
-file.</p>
-
-<h3 id="step-1-count-ngrams">Step 1: Count ngrams</h3>
-
-<p>Run the following script once from each source directory under the <code class="highlighter-rouge">corpus/</code>
-directory (edit it to specify the path to the <code class="highlighter-rouge">ngram-count</code> binary as well as
-the number of processors):</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
-
-<span class="nv">NGRAM_COUNT</span><span class="o">=</span><span class="nv">$SRILM_SRC</span>/bin/i686-m64/ngram-count
-<span class="nv">args</span><span class="o">=</span><span class="s2">""</span>
-
-<span class="k">for </span><span class="nb">source </span><span class="k">in</span> <span class="k">*</span>.gz; <span class="k">do
- </span><span class="nv">args</span><span class="o">=</span><span class="nv">$args</span><span class="s2">"-sort -order 5 -text </span><span class="nv">$source</span><span class="s2"> -write counts/</span><span class="nv">$source</span><span class="s2">-counts.gz "</span>
-<span class="k">done
-
-</span><span class="nb">echo</span> <span class="nv">$args</span> | xargs --max-procs<span class="o">=</span>4 -n 7 <span class="nv">$NGRAM_COUNT</span>
-</code></pre>
-</div>
-
-<p>Then move each <code class="highlighter-rouge">counts/</code> directory to the corresponding directory under
-<code class="highlighter-rouge">lm/</code>. Now that each ngram has been counted, we can make a language
-model for each of the seven sources.</p>
-
-<h3 id="step-2-make-individual-language-models">Step 2: Make individual language models</h3>
-
-<p>SRILM includes a script, called <code class="highlighter-rouge">make-big-lm</code>, for building large language
-models under resource-limited environments. The manual for this script can be
-read online
-<a href="http://www-speech.sri.com/projects/srilm/manpages/training-scripts.1.html">here</a>.
-Since the Gigaword corpus is so large, it is convenient to use <code class="highlighter-rouge">make-big-lm</code>
-even in environments with many parallel processors and a lot of memory.</p>
-
-<p>Initiate the following script from each of the source directories under the
-<code class="highlighter-rouge">lm/</code> directory (edit it to specify the path to the <code class="highlighter-rouge">make-big-lm</code> script as
-well as the pruning threshold):</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
-<span class="nb">set</span> -x
-
-<span class="nv">CMD</span><span class="o">=</span><span class="nv">$SRILM_SRC</span>/bin/make-big-lm
-<span class="nv">PRUNE_THRESHOLD</span><span class="o">=</span>1e-8
-
-<span class="nv">$CMD</span> <span class="se">\</span>
- -name gigalm <span class="sb">`</span><span class="k">for </span>k <span class="k">in </span>counts/<span class="k">*</span>.gz; <span class="k">do </span><span class="nb">echo</span> <span class="s2">" </span><span class="se">\</span><span class="s2">
- -read </span><span class="nv">$k</span><span class="s2"> "</span>; <span class="k">done</span><span class="sb">`</span> <span class="se">\</span>
- -lm lm.gz <span class="se">\</span>
- -max-per-file 100000000 <span class="se">\</span>
- -order 5 <span class="se">\</span>
- -kndiscount <span class="se">\</span>
- -interpolate <span class="se">\</span>
- -unk <span class="se">\</span>
- -prune <span class="nv">$PRUNE_THRESHOLD</span>
-</code></pre>
-</div>
-
-<p>The language model attributes chosen are the following:</p>
-
-<ul>
- <li>N-grams up to order 5</li>
- <li>Kneser-Ney smoothing</li>
- <li>N-gram probability estimates at the specified order <em>n</em> are interpolated with
-lower-order estimates</li>
- <li>include the unknown-word token as a regular word</li>
- <li>pruning N-grams based on the specified threshold</li>
-</ul>
-
-<p>Next, we will mix the models together into a single file.</p>
-
-<h3 id="step-3-mix-models-together">Step 3: Mix models together</h3>
-
-<p>Using development text, interpolation weights can determined that give highest
-weight to the source language models that have the lowest perplexity on the
-specified development set.</p>
-
-<h4 id="step-3-1-determine-interpolation-weights">Step 3-1: Determine interpolation weights</h4>
-
-<p>Initiate the following script from the <code class="highlighter-rouge">lm/</code> directory (edit it to specify the
-path to the <code class="highlighter-rouge">ngram</code> binary as well as the path to the development text file):</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
-<span class="nb">set</span> -x
-
-<span class="nv">NGRAM</span><span class="o">=</span><span class="nv">$SRILM_SRC</span>/bin/i686-m64/ngram
-<span class="nv">DEV_TEXT</span><span class="o">=</span>~mpost/expts/wmt12/runs/es-en/data/tune/tune.tok.lc.es
-
-<span class="nb">dirs</span><span class="o">=(</span> afp_eng apw_eng cna_eng ltw_eng nyt_eng wpb_eng xin_eng <span class="o">)</span>
-
-<span class="k">for </span>d <span class="k">in</span> <span class="k">${</span><span class="nv">dirs</span><span class="p">[@]</span><span class="k">}</span> ; <span class="k">do</span>
- <span class="nv">$NGRAM</span> -debug 2 -order 5 -unk -lm <span class="nv">$d</span>/lm.gz -ppl <span class="nv">$DEV_TEXT</span> > <span class="nv">$d</span>/lm.ppl ;
-<span class="k">done
-
-</span>compute-best-mix <span class="k">*</span>/lm.ppl > best-mix.ppl
-</code></pre>
-</div>
-
-<p>Take a look at the contents of <code class="highlighter-rouge">best-mix.ppl</code>. It will contain a sequence of
-values in parenthesis. These are the interpolation weights of the source
-language models in the order specified. Copy and paste the values within the
-parenthesis into the script below.</p>
-
-<h4 id="step-3-2-combine-the-models">Step 3-2: Combine the models</h4>
-
-<p>Initiate the following script from the <code class="highlighter-rouge">lm/</code> directory (edit it to specify the
-path to the <code class="highlighter-rouge">ngram</code> binary as well as the interpolation weights):</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
-<span class="nb">set</span> -x
-
-<span class="nv">NGRAM</span><span class="o">=</span><span class="nv">$SRILM_SRC</span>/bin/i686-m64/ngram
-<span class="nv">DIRS</span><span class="o">=(</span> afp_eng apw_eng cna_eng ltw_eng nyt_eng wpb_eng xin_eng <span class="o">)</span>
-<span class="nv">LAMBDAS</span><span class="o">=(</span>0.00631272 0.000647602 0.251555 0.0134726 0.348953 0.371566 0.00749238<span class="o">)</span>
-
-<span class="nv">$NGRAM</span> -order 5 -unk <span class="se">\</span>
- -lm <span class="k">${</span><span class="nv">DIRS</span><span class="p">[0]</span><span class="k">}</span>/lm.gz -lambda <span class="k">${</span><span class="nv">LAMBDAS</span><span class="p">[0]</span><span class="k">}</span> <span class="se">\</span>
- -mix-lm <span class="k">${</span><span class="nv">DIRS</span><span class="p">[1]</span><span class="k">}</span>/lm.gz <span class="se">\</span>
- -mix-lm2 <span class="k">${</span><span class="nv">DIRS</span><span class="p">[2]</span><span class="k">}</span>/lm.gz -mix-lambda2 <span class="k">${</span><span class="nv">LAMBDAS</span><span class="p">[2]</span><span class="k">}</span> <span class="se">\</span>
- -mix-lm3 <span class="k">${</span><span class="nv">DIRS</span><span class="p">[3]</span><span class="k">}</span>/lm.gz -mix-lambda3 <span class="k">${</span><span class="nv">LAMBDAS</span><span class="p">[3]</span><span class="k">}</span> <span class="se">\</span>
- -mix-lm4 <span class="k">${</span><span class="nv">DIRS</span><span class="p">[4]</span><span class="k">}</span>/lm.gz -mix-lambda4 <span class="k">${</span><span class="nv">LAMBDAS</span><span class="p">[4]</span><span class="k">}</span> <span class="se">\</span>
- -mix-lm5 <span class="k">${</span><span class="nv">DIRS</span><span class="p">[5]</span><span class="k">}</span>/lm.gz -mix-lambda5 <span class="k">${</span><span class="nv">LAMBDAS</span><span class="p">[5]</span><span class="k">}</span> <span class="se">\</span>
- -mix-lm6 <span class="k">${</span><span class="nv">DIRS</span><span class="p">[6]</span><span class="k">}</span>/lm.gz -mix-lambda6 <span class="k">${</span><span class="nv">LAMBDAS</span><span class="p">[6]</span><span class="k">}</span> <span class="se">\</span>
- -write-lm mixed_lm.gz
-</code></pre>
-</div>
-
-<p>The resulting file, <code class="highlighter-rouge">mixed_lm.gz</code> is a language model based on all the text in
-the Gigaword corpus and with some probabilities biased to the development text
-specify in step 3-1. It is in the ARPA format. The optional next step converts
-it into KenLM format.</p>
-
-<h4 id="step-3-3-convert-to-kenlm">Step 3-3: Convert to KenLM</h4>
-
-<p>The KenLM format has some speed advantages over the ARPA format. Issuing the
-following command will write a new language model file <code class="highlighter-rouge">mixed_lm-kenlm.gz</code> that
-is the <code class="highlighter-rouge">mixed_lm.gz</code> language model transformed into the KenLM format.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$JOSHUA/src/joshua/decoder/ff/lm/kenlm/build_binary mixed_lm.gz mixed_lm.kenlm
-</code></pre>
-</div>
-
-
-
- <!-- <h4 class="blog-post-title">Welcome to Joshua!</h4> -->
-
- <!-- <p>This blog post shows a few different types of content that's supported and styled with Bootstrap. Basic typography, images, and code are all supported.</p> -->
- <!-- <hr> -->
- <!-- <p>Cum sociis natoque penatibus et magnis <a href="#">dis parturient montes</a>, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.</p> -->
- <!-- <blockquote> -->
- <!-- <p>Curabitur blandit tempus porttitor. <strong>Nullam quis risus eget urna mollis</strong> ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.</p> -->
- <!-- </blockquote> -->
- <!-- <p>Etiam porta <em>sem malesuada magna</em> mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.</p> -->
- <!-- <h2>Heading</h2> -->
- <!-- <p>Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</p> -->
- <!-- <pre><code>Example code block</code></pre> -->
- <!-- <p>Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.</p> -->
- <!-- <h3>Sub-heading</h3> -->
- <!-- <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> -->
- <!-- <ul> -->
- <!-- <li>Praesent commodo cursus magna, vel scelerisque nisl consectetur et.</li> -->
- <!-- <li>Donec id elit non mi porta gravida at eget metus.</li> -->
- <!-- <li>Nulla vitae elit libero, a pharetra augue.</li> -->
- <!-- </ul> -->
- <!-- <p>Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.</p> -->
- <!-- <ol> -->
- <!-- <li>Vestibulum id ligula porta felis euismod semper.</li> -->
- <!-- <li>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.</li> -->
- <!-- <li>Maecenas sed diam eget risus varius blandit sit amet non magna.</li> -->
- <!-- </ol> -->
- <!-- <p>Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.</p> -->
- <!-- </div><\!-- /.blog-post -\-> -->
-
- </div>
-
- </div><!-- /.row -->
-
-
-
- </div><!-- /.container -->
-
- <!-- Bootstrap core JavaScript
- ================================================== -->
- <!-- Placed at the end of the document so the pages load faster -->
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
- <script src="../../dist/js/bootstrap.min.js"></script>
- <!-- <script src="../../assets/js/docs.min.js"></script> -->
- <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
- <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script>
- -->
-
- <!-- Start of StatCounter Code for Default Guide -->
- <script type="text/javascript">
- var sc_project=8264132;
- var sc_invisible=1;
- var sc_security="4b97fe2d";
- </script>
- <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
- <noscript>
- <div class="statcounter">
- <a title="hit counter joomla"
- href="http://statcounter.com/joomla/"
- target="_blank">
- <img class="statcounter"
- src="http://c.statcounter.com/8264132/0/4b97fe2d/1/"
- alt="hit counter joomla" />
- </a>
- </div>
- </noscript>
- <!-- End of StatCounter Code for Default Guide -->
- </body>
-</html>
-