You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@joshua.apache.org by mj...@apache.org on 2016/04/05 14:39:50 UTC
[01/50] incubator-joshua-site git commit: fixed link
Repository: incubator-joshua-site
Updated Branches:
refs/heads/master [created] 9a4315b56
fixed link
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/31ba97f9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/31ba97f9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/31ba97f9
Branch: refs/heads/master
Commit: 31ba97f9e43ff7f1e7dab0d708d4c63b29d97c91
Parents: e72fbca
Author: Matt Post <po...@cs.jhu.edu>
Authored: Mon May 18 13:12:46 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Mon May 18 13:12:46 2015 -0400
----------------------------------------------------------------------
_layouts/default6.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/31ba97f9/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index ec54a0d..5c981ea 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -88,7 +88,7 @@
<div class="sidebar-module">
<h4>Advanced</h4>
<ol class="list-unstyled">
- <li><a href="/6.0/file-formats.html">Language packs</a></li>
+ <li><a href="/6.0/bundle.html">Building language packs</a></li>
<li><a href="/6.0/decoder.html">Decoder options</a></li>
<li><a href="/6.0/file-formats.html">File formats</a></li>
</ol>
[15/50] incubator-joshua-site git commit: Updated tutorial
Posted by mj...@apache.org.
Updated tutorial
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/5b80a147
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/5b80a147
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/5b80a147
Branch: refs/heads/master
Commit: 5b80a14748fab4b9ad6ee331c90e9d3926b3ae7c
Parents: dc45f50
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed Jun 10 00:10:29 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed Jun 10 00:10:29 2015 -0400
----------------------------------------------------------------------
6.0/tutorial.md | 96 +++++++++++++++++++++++++--------------------
_layouts/default6.html | 7 +---
2 files changed, 55 insertions(+), 48 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/5b80a147/6.0/tutorial.md
----------------------------------------------------------------------
diff --git a/6.0/tutorial.md b/6.0/tutorial.md
index 9a43e93..d167cdc 100644
--- a/6.0/tutorial.md
+++ b/6.0/tutorial.md
@@ -13,75 +13,87 @@ other options available in the pipeline.
Download and install Joshua as described on the [quick start page](index.html), installing it under
`~/code/`. Once you've done that, you should make sure you have the following environment variable set:
- export JOSHUA=$HOME/code/joshua-v5.0
+ export JOSHUA=$HOME/code/joshua-v{{ site.data.joshua.release_version }}
export JAVA_HOME=/usr/java/default
-If you have a Hadoop installation, make sure you've set `$HADOOP` to point to it (if not, Joshua
-will roll out a standalone cluster for you). If you'd like to use kbmira for tuning, you should also
-install Moses, and define the environment variable `$MOSES` to point to the root of its installation.
+If you have a Hadoop installation, make sure you've set `$HADOOP` to point to it. For example, if the `hadoop` command is in `/usr/bin`,
+you should type
+
+ export HADOOP=/usr
+
+Joshua will find the binary and use it to submit to your hadoop cluster. If you don't have one, just
+make sure that HADOOP is unset, and Joshua will roll one out for you and run it in
+[standalone mode](https://hadoop.apache.org/docs/r1.2.1/single_node_setup.html).
## A basic pipeline run
-For today's experiments, we'll be building a Bengali--English system using data included in the
-[Indian Languages Parallel Corpora](/indian-parallel-corpora/). This data was collected by taking
-the 100 most-popular Bengali Wikipedia pages and translating them into English using Amazon's
-[Mechanical Turk](http://www.mturk.com/). As a warning, many of these pages contain material that is
-not typically found in machine translation tutorials.
+For today's experiments, we'll be building a Spanish--English system using data included in the
+[Fisher and CALLHOME translation corpus](/data/fisher-callhome-corpus/). This
+data was collected by translating transcribed speech from previous LDC releases.
Download the data and install it somewhere:
cd ~/data
- wget -q --no-check -O indian-parallel-corpora.zip https://github.com/joshua-decoder/indian-parallel-corpora/archive/master.zip
- unzip indian-parallel-corpora.zip
+ wget --no-check -O fisher-spanish-corpus.zip https://github.com/joshua-decoder/fisher-callhome-corpus/archive/master.zip
+ unzip fisher-spanish-corpus.zip
-Then define the environment variable `$INDIAN` to point to it:
+Then define the environment variable `$FISHER` to point to it:
- cd ~/data/indian-parallel-corpora-master
- export INDIAN=$(pwd)
+ cd ~/data/fisher-spanish-corpus-master
+ export FISHER=$(pwd)
### Preparing the data
-Inside this tarball is a directory for each language pair. Within each language directory is another
-directory named `tok/`, which contains pre-tokenized and normalized versions of the data. This was
-done because the normalization scripts provided with Joshua is written in scripting languages that
-often have problems properly handling UTF-8 character sets. We will be using these tokenized
-versions, and preventing the pipeline from retokenizing using the `--no-prepare` flag.
+Inside the tarball is the Fisher and CALLHOME Spanish--English data, which includes Kaldi-provided
+ASR output and English translations on the Fisher and CALLHOME dataset transcriptions. Because of
+licensing restrictions, we cannot distribute the Spanish transcripts, but if you have an LDC site
+license, a script is provided to build them. You can type:
+
+ ./bin/build_fisher.sh /export/common/data/corpora/LDC/LDC2010T04
-In `$INDIAN/bn-en/tok`, you should see the following files:
+Where the first argument is the path to your LDC data release. This will create the files in `corpus/ldc`.
- $ ls $INDIAN/bn-en/tok
- dev.bn-en.bn devtest.bn-en.bn dict.bn-en.bn test.bn-en.en.2
- dev.bn-en.en.0 devtest.bn-en.en.0 dict.bn-en.en test.bn-en.en.3
- dev.bn-en.en.1 devtest.bn-en.en.1 test.bn-en.bn training.bn-en.bn
- dev.bn-en.en.2 devtest.bn-en.en.2 test.bn-en.en.0 training.bn-en.en
- dev.bn-en.en.3 devtest.bn-en.en.3 test.bn-en.en.1
+In `$FISHER/corpus`, there are a set of parallel directories for LDC transcripts (`ldc`), ASR output
+(`asr`), oracle ASR output (`oracle`), and ASR lattice output (`plf`). The files look like this:
-We will now use this data to test the complete pipeline with a single command.
+ $ ls corpus/ldc
+ callhome_devtest.en fisher_dev2.en.2 fisher_dev.en.2 fisher_test.en.2
+ callhome_evltest.en fisher_dev2.en.3 fisher_dev.en.3 fisher_test.en.3
+ callhome_train.en fisher_dev2.es fisher_dev.es fisher_test.es
+ fisher_dev2.en.0 fisher_dev.en.0 fisher_test.en.0 fisher_train.en
+ fisher_dev2.en.1 fisher_dev.en.1 fisher_test.en.1 fisher_train.es
+
+If you don't have the LDC transcripts, you can use the data in `corpus/asr` instead. We will now use
+this data to build our own Spanish--English model using Joshua's pipeline.
### Run the pipeline
-Create an experiments directory for containing your first experiment:
+Create an experiments directory for containing your first experiment. *Note: it's important that
+this **not** be inside your `$JOSHUA` directory*.
mkdir ~/expts/joshua
cd ~/expts/joshua
We will now create the baseline run, using a particular directory structure for experiments that
will allow us to take advantage of scripts provided with Joshua for displaying the results of many
-related experiments.
+related experiments. Because this can take quite some time to run, we are going to add a crippling
+restriction: Joshua will only use sentences in the training sets with ten or fewer words on either
+side (Spanish or English):
cd ~/expts/joshua
$JOSHUA/bin/pipeline.pl \
--rundir 1 \
--readme "Baseline Hiero run" \
- --source bn \
+ --source es \
--target en \
- --corpus $INDIAN/bn-en/tok/training.bn-en \
- --corpus $INDIAN/bn-en/tok/dict.bn-en \
- --tune $INDIAN/bn-en/tok/dev.bn-en \
- --test $INDIAN/bn-en/tok/devtest.bn-en \
+ --type hiero \
+ --corpus $FISHER/corpus/ldc/fisher_train \
+ --tune $FISHER/corpus/ldc/fisher_dev \
+ --test $FISHER/corpus/ldc/fisher_dev2 \
+ --maxlen 10 \
--lm-order 3
-This will start the pipeline building a Bengali--English translation system constructed from the
+This will start the pipeline building a Spanish--English translation system constructed from the
training data and a dictionary, tuned against dev, and tested against devtest. It will use the
default values for most of the pipeline: [GIZA++](https://code.google.com/p/giza-pp/) for alignment,
KenLM's `lmplz` for building the language model, Z-MERT for tuning, KenLM with left-state
@@ -113,7 +125,7 @@ of the baseline model. Here are some examples of what you could vary:
- Decode with a wider beam (`--joshua-args '-pop-limit 200'`) (the default is 100)
-- Add the provided BN-EN dictionary to the training data (add another `--corpus` line, e.g., `--corpus $INDIAN/bn-en/dict.bn-en`)
+- Add the provided BN-EN dictionary to the training data (add another `--corpus` line, e.g., `--corpus $FISHER/bn-en/dict.bn-en`)
To do this, we will create new runs that partially reuse the results of previous runs. This is
possible by doing two things: (1) incrementing the run directory and providing an updated README
@@ -130,9 +142,9 @@ directory, tell the pipeline to start at the tuning step, and provide the needed
--readme "Tuning with MIRA" \
--source bn \
--target en \
- --corpus $INDIAN/bn-en/tok/training.bn-en \
- --tune $INDIAN/bn-en/tok/dev.bn-en \
- --test $INDIAN/bn-en/tok/devtest.bn-en \
+ --corpus $FISHER/bn-en/tok/training.bn-en \
+ --tune $FISHER/bn-en/tok/dev.bn-en \
+ --test $FISHER/bn-en/tok/devtest.bn-en \
--first-step tune \
--tuner mira \
--grammar 1/grammar.gz \
@@ -158,9 +170,9 @@ grammar, but can reuse the alignments and the language model:
--readme "Baseline SAMT model" \
--source bn \
--target en \
- --corpus $INDIAN/bn-en/tok/training.bn-en \
- --tune $INDIAN/bn-en/tok/dev.bn-en \
- --test $INDIAN/bn-en/tok/devtest.bn-en \
+ --corpus $FISHER/bn-en/tok/training.bn-en \
+ --tune $FISHER/bn-en/tok/dev.bn-en \
+ --test $FISHER/bn-en/tok/devtest.bn-en \
--alignment 1/alignments/training.align \
--first-step parse \
--no-corpus-lm \
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/5b80a147/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index 63a8adf..3d19a7b 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -34,11 +34,6 @@
<div class="container">
- <!-- <div class="blog-header"> -->
- <!-- <h1 class="blog-title">Joshua</h1> -->
- <!-- <\!-- <p class="lead blog-description">The Joshua machine translation system</p> -\-> -->
- <!-- </div> -->
-
<div class="row">
<div class="col-sm-2">
@@ -65,7 +60,6 @@
<ol class="list-unstyled">
<li><a href="/6.0/install.html">Installation</a></li>
<li><a href="/6.0/quick-start.html">Quick Start</a></li>
- <li><a href="/6.0/faq.html">FAQ</a></li>
</ol>
</div>
<hr>
@@ -73,6 +67,7 @@
<h4>Building new models</h4>
<ol class="list-unstyled">
<li><a href="/6.0/pipeline.html">Pipeline</a></li>
+ <li><a href="/6.0/tutorial.html">Tutorial</a></li>
<li><a href="/6.0/faq.html">FAQ</a></li>
</ol>
</div>
[41/50] incubator-joshua-site git commit: added button to runtime
Posted by mj...@apache.org.
added button to runtime
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/025c8f35
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/025c8f35
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/025c8f35
Branch: refs/heads/master
Commit: 025c8f35038745fcafc51539c83c41a4d1cbe9ad
Parents: 33eedd8
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Nov 6 09:59:39 2015 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Nov 6 09:59:39 2015 -0500
----------------------------------------------------------------------
6.0/install.md | 38 +++++++++++++++++++-------------------
_layouts/default6.html | 2 ++
language-packs/index.md | 2 +-
releases/runtime/index.html | 4 ++++
4 files changed, 26 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/025c8f35/6.0/install.md
----------------------------------------------------------------------
diff --git a/6.0/install.md b/6.0/install.md
index 6d13e4d..87e0079 100644
--- a/6.0/install.md
+++ b/6.0/install.md
@@ -12,46 +12,46 @@ See the instructions below.
1. Set up some basic environment variables.
You need to define `$JAVA_HOME`
- export JAVA_HOME=/path/to/java
+ export JAVA_HOME=/path/to/java
- # JAVA_HOME is not very standardized. Here are some places to look:
- # OS X: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
- # Linux: export JAVA_HOME=/usr/java/default
+ # JAVA_HOME is not very standardized. Here are some places to look:
+ # OS X: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
+ # Linux: export JAVA_HOME=/usr/java/default
1. If you are installing the full version of Joshua, you also need to define `$HADOOP` to point to your Hadoop installation.
(Joshua looks for the Hadoop executuble in `$HADOOP/bin/hadoop`)
- export HADOOP=/usr
+ export HADOOP=/usr
If you don't have a Hadoop installation, [Joshua's pipeline](pipeline.html) can install a standalone version for you.
1. To install just the runtime version of Joshua, type
- wget -q http://cs.jhu.edu/~post/files/joshua-runtime-{{ site.data.joshua.release_version }}.tgz
+ wget -q http://cs.jhu.edu/~post/files/joshua-runtime-{{ site.data.joshua.release_version }}.tgz
Then build everything
- tar xzf joshua-runtime-{{ site.data.joshua.release_version }}.tgz
- cd joshua-runtime-{{ site.data.joshua.release_version }}
+ tar xzf joshua-runtime-{{ site.data.joshua.release_version }}.tgz
+ cd joshua-runtime-{{ site.data.joshua.release_version }}
- # Add this to your init files
- export JOSHUA=$(pwd)
+ # Add this to your init files
+ export JOSHUA=$(pwd)
- # build everything
- ant
+ # build everything
+ ant
1. To instead install the full version, type
- wget -q http://cs.jhu.edu/~post/files/joshua-{{ site.data.joshua.release_version }}.tgz
+ wget -q http://cs.jhu.edu/~post/files/joshua-{{ site.data.joshua.release_version }}.tgz
- tar xzf joshua-{{ site.data.joshua.release_version }}.tgz
- cd joshua-{{ site.data.joshua.release_version }}
+ tar xzf joshua-{{ site.data.joshua.release_version }}.tgz
+ cd joshua-{{ site.data.joshua.release_version }}
- # Add this to your init files
- export JOSHUA=$(pwd)
+ # Add this to your init files
+ export JOSHUA=$(pwd)
- # build everything
- ant
+ # build everything
+ ant
### Building new models
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/025c8f35/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index 0ca8adb..94cd42c 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -48,6 +48,8 @@
<hr>
<center>
<a href="/releases/current/" target="_blank"><button class="button">Download Joshua {{ site.data.joshua.release_version }}</button></a>
+ <br />
+ <a href="/releases/runtime/" target="_blank"><button class="button">Download Joshua (runtime only) {{ site.data.joshua.release_version }}</button></a>
<p>Released {{ site.data.joshua.release_date }}</p>
</center>
<hr>
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/025c8f35/language-packs/index.md
----------------------------------------------------------------------
diff --git a/language-packs/index.md b/language-packs/index.md
index c07b6f4..16c98f0 100644
--- a/language-packs/index.md
+++ b/language-packs/index.md
@@ -23,7 +23,7 @@ The default release version of Joshua includes hundreds of megabytes of dependen
for building models from your own language pairs. If you only wish to run language packs
(effectively using Joshua as a black-box translation engine), you can install a "light" version
of Joshua that includes only dependencies needed to run the translation piece. See the notes
-on [the installation page](../6/install.html).
+on [the installation page](../6.0/install.html).
## Available language packs
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/025c8f35/releases/runtime/index.html
----------------------------------------------------------------------
diff --git a/releases/runtime/index.html b/releases/runtime/index.html
new file mode 100644
index 0000000..7e254fe
--- /dev/null
+++ b/releases/runtime/index.html
@@ -0,0 +1,4 @@
+---
+redirect_to:
+ - http://cs.jhu.edu/~post/files/joshua-runtime-{{ site.data.joshua.release_version }}.tgz
+---
[25/50] incubator-joshua-site git commit: updated LM specification
Posted by mj...@apache.org.
updated LM specification
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/601d9f8b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/601d9f8b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/601d9f8b
Branch: refs/heads/master
Commit: 601d9f8ba40af8d8c75a46823d3ded717f379739
Parents: f74fc8c
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Jun 19 22:22:39 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Jun 19 22:22:39 2015 -0400
----------------------------------------------------------------------
6.0/decoder.md | 45 +++++++++++++++++++++++++++------------------
1 file changed, 27 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/601d9f8b/6.0/decoder.md
----------------------------------------------------------------------
diff --git a/6.0/decoder.md b/6.0/decoder.md
index 149f5e6..e8dc8c9 100644
--- a/6.0/decoder.md
+++ b/6.0/decoder.md
@@ -212,24 +212,33 @@ For reference, the following two translation model lines are used by the [pipeli
### Language model options <a id="lm" />
-Joshua supports any number of language models. To add a language
-model, add a line of the following format to the configuration file:
-
- lm = TYPE ORDER LEFT_STATE RIGHT_STATE CEILING_COST FILE
-
-where the six fields correspond to the following values:
-
-* `TYPE`: one of "kenlm", "berkeleylm", or "none"
-* `ORDER`: the order of the language model
-* `LEFT_STATE`: whether to use left-state minimization; currently only supported by KenLM
-* `RIGHT_STATE`: whether to use right equivalent state (currently unsupported)
-* `CEILING_COST`: the LM-specific ceiling cost of all n-grams (currently ignored)
-* `FILE`: the path to the language model file. All language model types support the standard ARPA
- format. Additionally, if the LM type is "kenlm", this file can be compiled into KenLM's compiled
- format (using the program at `$JOSHUA/bin/build_binary`); if the the LM type is "berkeleylm", it
- can be compiled by following the directions in
- `$JOSHUA/src/joshua/decoder/ff/lm/berkeley_lm/README`. The [pipeline](pipeline.html) will
- automatically compile either type.
+Joshua supports any number of language models. With Joshua 6.0, these
+are just regular feature functions:
+
+ feature-function = LanguageModel -lm_file /path/to/lm/file -lm_order N -lm_type TYPE
+ feature-function = StateMinimizingLanguageModel -lm_file /path/to/lm/file -lm_order N -lm_type TYPE
+
+`LanguageModel` is a generic language model, supporting types 'kenlm'
+(the default) and 'berkeleylm'. `StateMinimizingLanguageModel`
+implements LM state minimization to reduce the size of context n-grams
+where appropriate
+([Li and Khudanpur, 2008](http://www.aclweb.org/anthology/W08-0402.pdf);
+[Heafield et al., 2013](https://aclweb.org/anthology/N/N13/N13-1116.pdf)). This
+is currently only supported by KenLM, so the `-lm_type` option is not
+available here.
+
+The other key/value pairs are defined as follows:
+
+* `lm_type`: one of "kenlm" "berkeleylm"
+* `lm_order`: the order of the language model
+* `lm_file`: the path to the language model file. All language model
+ types support the standard ARPA format. Additionally, if the LM
+ type is "kenlm", this file can be compiled into KenLM's compiled
+ format (using the program at `$JOSHUA/bin/build_binary`); if the
+ the LM type is "berkeleylm", it can be compiled by following the
+ directions in
+ `$JOSHUA/src/joshua/decoder/ff/lm/berkeley_lm/README`. The
+ [pipeline](pipeline.html) will automatically compile either type.
For each language model, you need to specify a feature weight in the following format:
[20/50] incubator-joshua-site git commit: updated contributors
Posted by mj...@apache.org.
updated contributors
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/1f7b84fa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/1f7b84fa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/1f7b84fa
Branch: refs/heads/master
Commit: 1f7b84faa0f019c037bffab31f1342ef10c99cad
Parents: 5a64b32
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Jun 19 10:57:32 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Jun 19 10:57:32 2015 -0400
----------------------------------------------------------------------
contributors.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/1f7b84fa/contributors.md
----------------------------------------------------------------------
diff --git a/contributors.md b/contributors.md
index 629f318..5b716b5 100644
--- a/contributors.md
+++ b/contributors.md
@@ -12,12 +12,14 @@ alphabetical order).
- [Adam Pauls](http://www.eecs.berkeley.edu/~adpauls/) (Berkeley)
- [Chris Callison-Burch](http://cs.jhu.edu/~ccb/) (Penn)
- [Chris Dyer](http://www.cs.cmu.edu/~cdyer/) (Carnegie Mellon)
+- [Chris Mattmann](http://sunset.usc.edu/~mattmann/) (APL, USC)
- [Colin Bannard](http://comp.ling.utexas.edu/~bannard/) (UT Austin)
- [Jonathan Weese](http://cs.jhu.edu/~jonny/) (Johns Hopkins)
- <li>Josh Schroeder (Edinburgh)</li>
+- Josh Schroeder (Edinburgh)
- [Juri Ganitkevitch](http://cs.jhu.edu/~juri/) (Johns Hopkins)
- [Kenneth Heafield](http://kheafield.com/) (Carnegie Mellon)
- [Lane Schwartz](http://dowobeha.github.com) (Minnesota)
+- [Lewis John McGibbney](https://github.com/lewismc) (NASA JPL)
- [Luke](https://plus.google.com/107547504293849335564) [Orland](https://github.com/lukeorland) (Johns Hopkins)
- [Matt Post](http://cs.jhu.edu/~post/) (Johns Hopkins)
- [Omar F. Zaidan](http://cs.jhu.edu/~ozaidan/) (Johns Hopkins)
[04/50] incubator-joshua-site git commit: Updated the quick start
guide
Posted by mj...@apache.org.
Updated the quick start guide
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/9c64de09
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/9c64de09
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/9c64de09
Branch: refs/heads/master
Commit: 9c64de0919c6ee7ddeefc4b2590ee4cd23d7e016
Parents: afcbef1
Author: Matt Post <po...@cs.jhu.edu>
Authored: Tue May 19 18:39:15 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Tue May 19 18:39:15 2015 -0400
----------------------------------------------------------------------
6.0/quick-start.md | 56 +++++++++++++++++++++++++++++++++++--------------
1 file changed, 40 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/9c64de09/6.0/quick-start.md
----------------------------------------------------------------------
diff --git a/6.0/quick-start.md b/6.0/quick-start.md
index 1531521..55d025a 100644
--- a/6.0/quick-start.md
+++ b/6.0/quick-start.md
@@ -3,21 +3,30 @@ layout: default6
title: Quick Start
---
-The quickest way to use Joshua is to download a
-[pre-built model](/language-packs/) and use them to start translating data.
+If you just want to use Joshua to translate data, the quickest way is
+to download a [pre-built model](/language-packs/).
-Building your own models takes a bit more work, and requires you to
-supply parallel data that the models can be trained from. Information
-about how to do this can be found in [the pipeline documentation](/6.0/pipeline.html).
+If not language pack is available, or if you have your own parallel
+data that you want to train the translation engine on, then you have
+to build your own model. This takes a bit more knowledge and effort,
+but is made easier with Joshua's [pipeline script](pipeline.html),
+which runs all the steps of preparing data, aligning it, and
+extracting and tuning component models.
-Our <a href="pipeline.html">pipeline script</a> is the quickest way to get started. For example, to
-train and test a complete model translating from Bengali to English:
+Detailed information about running the pipeline can be found in
+[the pipeline documentation](/6.0/pipeline.html), but as a quick
+start, you can build a simple Bengali--English model by following
+these instructions.
-First, download the Indian languages data:
+*NOTE: We suggest you build models outside the `$JOSHUA` directory*.
+
+First, download the dataset:
- curl -#L https://github.com/joshua-decoder/indian-parallel-corpora/tarball/master > indian-languages.tgz
- tar xf indian-languages.tgz
- ln -s joshua-decoder-indian-parallel-corpora-* input
+ mkdir -p ~/models/bn-en/
+ cd ~/models/bn-en
+ curl -L https://github.com/joshua-decoder/indian-parallel-corpora/tarball/master > indian-languages.tgz
+ tar xf indian-languages.tgz
+ ln -s joshua-decoder-indian-parallel-corpora-* input
Then, train and test a model
@@ -27,8 +36,23 @@ Then, train and test a model
--tune input/bn-en/tok/dev.bn-en \
--test input/bn-en/tok/devtest.bn-en
-This will align the data with the Berkeley aligner, build a Hiero model, tune with MERT, decode the
-test sets, and reports results that should correspond with what you find on <a
-href="/indian-parallel-corpora/">the Indian Parallel Corpora page</a>. For
-more details, including information on the many options available with the pipeline script, please
-see <a href="pipeline.html">its documentation page</a>.
+This will align the data with the Berkeley aligner, build a Hiero
+model, tune with MERT, decode the test sets, and reports results that
+should correspond with what you find on
+[the Indian Parallel Corpora page](/indian-parallel-corpora/). For
+more details, including information on the many options available with
+the pipeline script, please see [its documentation page](pipeline.html).
+
+Finally, you can export the full model as a language pack:
+
+ ./run-bundler.py \
+ tune/joshua.config.final \
+ language-pack-bn-en \
+ --pack-tm grammar.gz
+
+(or possibly `tune/1/joshua.config.final` if you're using an older version of
+the pipeline).
+
+This will create a [runnable model](bundle.html) in
+`language-pack-bn-en`. See the `README` file in that directory for
+information on how to run the decoder.
[47/50] incubator-joshua-site git commit: link to paraphrase page
Posted by mj...@apache.org.
link to paraphrase page
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/4c1c9a5f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/4c1c9a5f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/4c1c9a5f
Branch: refs/heads/master
Commit: 4c1c9a5fcf71a891bbe627f52543428f1adad7d1
Parents: 723da06
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Apr 1 15:15:30 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Apr 1 15:15:30 2016 -0400
----------------------------------------------------------------------
language-packs/index.md | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/4c1c9a5f/language-packs/index.md
----------------------------------------------------------------------
diff --git a/language-packs/index.md b/language-packs/index.md
index e43efe8..24b2f74 100644
--- a/language-packs/index.md
+++ b/language-packs/index.md
@@ -27,6 +27,8 @@ on [the installation page](../6.0/install.html).
## Available language packs
+For information about English Paraphrase Language packs, please see [this page](paraphrase/).
+
<table class="table table-condensed">
<tr>
<th>Language pair</th>
[13/50] incubator-joshua-site git commit: updated pipeline doc for
6.0.3 release
Posted by mj...@apache.org.
updated pipeline doc for 6.0.3 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/f956df75
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/f956df75
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/f956df75
Branch: refs/heads/master
Commit: f956df750815d8170e368fa5b8cce359d792a800
Parents: 81748f5
Author: Matt Post <po...@cs.jhu.edu>
Authored: Mon Jun 1 11:42:51 2015 -0600
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Mon Jun 1 11:42:51 2015 -0600
----------------------------------------------------------------------
6.0/pipeline.md | 42 +++++++++++++++++-------------------------
_data/joshua.yaml | 4 ++--
2 files changed, 19 insertions(+), 27 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/f956df75/6.0/pipeline.md
----------------------------------------------------------------------
diff --git a/6.0/pipeline.md b/6.0/pipeline.md
index 5c34237..f35f618 100644
--- a/6.0/pipeline.md
+++ b/6.0/pipeline.md
@@ -4,6 +4,9 @@ category: links
title: The Joshua Pipeline
---
+*Please note that the Joshua 6.0.3 included some big changes to directory organization of the
+ pipeline's files.*
+
This page describes the Joshua pipeline script, which manages the complexity of training and
evaluating machine translation systems. The pipeline eases the pain of two related tasks in
statistical machine translation (SMT) research:
@@ -164,13 +167,13 @@ generated by the individual sub-steps).
corpus.en
thrax-input-file
tune/
- tune.tok.lc.ur
- tune.tok.lc.en
+ corpus.ur -> tune.tok.lc.ur
+ corpus.en -> tune.tok.lc.en
grammar.filtered.gz
grammar.glue
test/
- test.tok.lc.ur
- test.tok.lc.en
+ corpus.ur -> test.tok.lc.ur
+ corpus.en -> test.tok.lc.en
grammar.filtered.gz
grammar.glue
alignments/
@@ -182,14 +185,14 @@ generated by the individual sub-steps).
grammar.gz
lm.gz
tune/
- 1/
- decoder_command
- joshua.config
- params.txt
- joshua.log
- mert.log
- joshua.config.ZMERT.final
- final-bleu
+ decoder_command
+ model/
+ [model files]
+ params.txt
+ joshua.log
+ mert.log
+ joshua.config.final
+ final-bleu
These files will be described in more detail in subsequent sections of this tutorial.
@@ -554,17 +557,11 @@ memory specification (passed to its `-Xmx` flag).
Two optimizers are provided with Joshua: MERT and PRO (`--tuner {mert,pro}`). If Moses is
installed, you can also use Cherry & Foster's k-best batch MIRA (`--tuner mira`, recommended).
-Tuning is run till convergence in the `$RUNDIR/tune/N` directory, where N is the tuning instance.
-By default, tuning is run just once, but the pipeline supports running the optimizer an arbitrary
-number of times due to [recent work](http://www.youtube.com/watch?v=BOa3XDkgf0Y) pointing out the
-variance of tuning procedures in machine translation, in particular MERT. This can be activated
-with `--optimizer-runs N`. Each run can be found in a directory `$RUNDIR/tune/N`.
+Tuning is run till convergence in the `$RUNDIR/tune` directory.
When tuning is finished, each final configuration file can be found at either
- $RUNDIR/tune/N/joshua.config.final
-
-where N varies from 1..`--optimizer-runs`.
+ $RUNDIR/tune/joshua.config.final
## <a id="testing" /> 7. Testing
@@ -583,11 +580,6 @@ number of arguments:
This tells the decoder to start at the test step.
-- `--name NAME`
-
- A name is needed to distinguish this test set from the previous ones. Output for this test run
- will be stored at `$RUNDIR/test/NAME`.
-
- `--joshua-config CONFIG`
A tuned parameter file is required. This file will be the output of some prior tuning run.
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/f956df75/_data/joshua.yaml
----------------------------------------------------------------------
diff --git a/_data/joshua.yaml b/_data/joshua.yaml
index 23c55d6..595f48d 100644
--- a/_data/joshua.yaml
+++ b/_data/joshua.yaml
@@ -1,2 +1,2 @@
-release_version: 6.0.2
-release_date: April 10, 2015
+release_version: 6.0.3
+release_date: June 1, 2015
[16/50] incubator-joshua-site git commit: updated quick start
Posted by mj...@apache.org.
updated quick start
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/3dd4d775
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/3dd4d775
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/3dd4d775
Branch: refs/heads/master
Commit: 3dd4d7759e0cd36e7f261d9040a8f25deff0d31e
Parents: 5b80a14
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed Jun 10 10:03:49 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed Jun 10 10:03:49 2015 -0400
----------------------------------------------------------------------
6.0/install.md | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/3dd4d775/6.0/install.md
----------------------------------------------------------------------
diff --git a/6.0/install.md b/6.0/install.md
index df187e5..d242f86 100644
--- a/6.0/install.md
+++ b/6.0/install.md
@@ -9,26 +9,33 @@ To use Joshua as a standalone decoder (with
[language packs](/language-packs/)), you only need to download and
install the decoder. There are no external dependencies.
+1. Set up some basic environment variables. You need to define `$JAVA_HOME` and
+ also `$HADOOP`, if you have a Hadoop installation (Joshua's pipeline looks
+ for `$HADOOP/bin/hadoop`.
+
+ export JAVA_HOME=/path/to/java
+
+ # JAVA_HOME is not very standardized. Here are some places to look:
+ # OS X: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
+ # Linux: export JAVA_HOME=/usr/java/default
+
+ export HADOOP=/usr
+
1. Download Joshua
wget -q http://cs.jhu.edu/~post/joshua-v{{ site.data.joshua.release_version }}.tgz
-1. Next, unpack it, set environment variables, and compile everything. You need to define the
-`$JOSHUA` environment variable (which in turn requires `$JAVA_HOME`).
+1. Next, unpack it and compile everything.
tar xzf joshua-v{{ site.data.joshua.release_version }}.tgz
cd joshua-v{{ site.data.joshua.release_version }}
- # for bash
- export JAVA_HOME=/path/to/java
+ # Add this to your init files
export JOSHUA=$(pwd)
- echo "export JOSHUA=$JOSHUA" >> ~/.bashrc
# build everything
ant
- (If you don't know what to set `$JAVA_HOME` to, try `/usr/java/default`).
-
This compiles Joshua and also a number of support tools, such as KenLM and GIZA++.
1. [Download a model](/language-packes/) and start translating!
[38/50] incubator-joshua-site git commit: updated version link
Posted by mj...@apache.org.
updated version link
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/77ad61fa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/77ad61fa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/77ad61fa
Branch: refs/heads/master
Commit: 77ad61faa6cd0185b6462e5144de1140ef97f86a
Parents: 549fb14
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Nov 5 07:06:52 2015 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Nov 5 07:06:52 2015 -0500
----------------------------------------------------------------------
6.0/install.md | 6 +++---
releases/current/index.html | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/77ad61fa/6.0/install.md
----------------------------------------------------------------------
diff --git a/6.0/install.md b/6.0/install.md
index a6c9233..68c4720 100644
--- a/6.0/install.md
+++ b/6.0/install.md
@@ -23,12 +23,12 @@ install the decoder. There are no external dependencies.
1. Download Joshua
- wget -q http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz
+ wget -q http://cs.jhu.edu/~post/files/joshua-{{ site.data.joshua.release_version }}.tgz
1. Next, unpack it and compile everything.
- tar xzf joshua-v{{ site.data.joshua.release_version }}.tgz
- cd joshua-v{{ site.data.joshua.release_version }}
+ tar xzf joshua-{{ site.data.joshua.release_version }}.tgz
+ cd joshua-{{ site.data.joshua.release_version }}
# Add this to your init files
export JOSHUA=$(pwd)
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/77ad61fa/releases/current/index.html
----------------------------------------------------------------------
diff --git a/releases/current/index.html b/releases/current/index.html
index 17ec987..e845624 100644
--- a/releases/current/index.html
+++ b/releases/current/index.html
@@ -1,4 +1,4 @@
---
redirect_to:
- - http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz
+ - http://cs.jhu.edu/~post/files/joshua-{{ site.data.joshua.release_version }}.tgz
---
[42/50] incubator-joshua-site git commit: changed text
Posted by mj...@apache.org.
changed text
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/a0cf2f5c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/a0cf2f5c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/a0cf2f5c
Branch: refs/heads/master
Commit: a0cf2f5c232ff6e9be9496f910369a980a7a2ad0
Parents: 025c8f3
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Nov 6 10:07:34 2015 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Nov 6 10:07:34 2015 -0500
----------------------------------------------------------------------
_layouts/default6.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/a0cf2f5c/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index 94cd42c..d647655 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -49,7 +49,7 @@
<center>
<a href="/releases/current/" target="_blank"><button class="button">Download Joshua {{ site.data.joshua.release_version }}</button></a>
<br />
- <a href="/releases/runtime/" target="_blank"><button class="button">Download Joshua (runtime only) {{ site.data.joshua.release_version }}</button></a>
+ <a href="/releases/runtime/" target="_blank"><button class="button">Runtime only version</button></a>
<p>Released {{ site.data.joshua.release_date }}</p>
</center>
<hr>
[02/50] incubator-joshua-site git commit: fixed bug
Posted by mj...@apache.org.
fixed bug
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/43424bff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/43424bff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/43424bff
Branch: refs/heads/master
Commit: 43424bff1326eeb45273e1d543da4d98135bded9
Parents: 31ba97f
Author: Matt Post <po...@cs.jhu.edu>
Authored: Mon May 18 15:42:44 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Mon May 18 15:42:44 2015 -0400
----------------------------------------------------------------------
6.0/bundle.md | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/43424bff/6.0/bundle.md
----------------------------------------------------------------------
diff --git a/6.0/bundle.md b/6.0/bundle.md
index 48d356b..cc3b8a9 100644
--- a/6.0/bundle.md
+++ b/6.0/bundle.md
@@ -55,12 +55,9 @@ Here is an example invocation:
/path/to/rundir/test/1/joshua.config \
language-pack-YYYY-MM-DD \
--root /path/to/rundir \
- --tm "pt /path/to/rundir/grammar.gz" \
- --pack-tm \
+ --pack-tm /path/to/rundir/grammar.gz \
--copy-config-options \
- '-top-n 1 \
- -output-format %S \
- -mark-oovs false' \
+ '-top-n 1 -output-format %S -mark-oovs false' \
--server-port 5674
The copy config options tell the decoder to present just the
@@ -71,7 +68,9 @@ to OOVs (`-mark-oovs false`), and to use the grammar
[this page](decoder.html) for a longer list of decoder options.
The `--pack-tm` option tells the run bundler to
-[pack the grammar](packing.html), which can take some time.
+[pack the grammar](packing.html), which can take some time. Since it
+is the first of any `--[pack-]tm` arguments, it applies to the first
+TM encountered in the config file.
A new directory `language-pack-YYYY-MM-DD` will be created along with
a README and a number of support files.
[09/50] incubator-joshua-site git commit: playing around
Posted by mj...@apache.org.
playing around
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/4c8fd245
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/4c8fd245
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/4c8fd245
Branch: refs/heads/master
Commit: 4c8fd245032b674f0fa0e21dbe90e862d91c526a
Parents: d521a14
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 11:37:12 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 11:37:12 2015 -0400
----------------------------------------------------------------------
releases/6.0/index.html | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/4c8fd245/releases/6.0/index.html
----------------------------------------------------------------------
diff --git a/releases/6.0/index.html b/releases/6.0/index.html
index 5402eca..51d8644 100644
--- a/releases/6.0/index.html
+++ b/releases/6.0/index.html
@@ -1,3 +1,4 @@
---
+title: This page should redirect you
redirect_from: "http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz"
---
[05/50] incubator-joshua-site git commit: Fixed redirect and download
message
Posted by mj...@apache.org.
Fixed redirect and download message
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/29556b18
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/29556b18
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/29556b18
Branch: refs/heads/master
Commit: 29556b18c016261e20041f42013c038d1cfcc1b9
Parents: 9c64de0
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 11:06:37 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 11:06:37 2015 -0400
----------------------------------------------------------------------
6.0/pipeline.md | 18 ++++++++++--------
releases/6.0/index.html | 11 +----------
releases/current | 1 +
releases/index.html | 11 -----------
releases/index.md | 11 +++++++++++
5 files changed, 23 insertions(+), 29 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/29556b18/6.0/pipeline.md
----------------------------------------------------------------------
diff --git a/6.0/pipeline.md b/6.0/pipeline.md
index 35ad661..5c34237 100644
--- a/6.0/pipeline.md
+++ b/6.0/pipeline.md
@@ -26,10 +26,11 @@ To facilitate these tasks, the pipeline script:
- Allows you to jump into and out of the pipeline at a set of predefined places (e.g., the alignment
stage), so long as you provide the missing dependencies.
-The Joshua pipeline script is designed in the spirit of Moses' `train-model.pl`, and shares many of
-its features. It is not as extensive, however, as Moses'
+The Joshua pipeline script is designed in the spirit of Moses' `train-model.pl`, and shares
+(and has borrowed) many of its features. It is not as extensive as Moses'
[Experiment Management System](http://www.statmt.org/moses/?n=FactoredTraining.EMS), which allows
-the user to define arbitrary execution dependency graphs.
+the user to define arbitrary execution dependency graphs. However, it is significantly simpler to
+use, allowing many systems to be built with a single command (that may run for days or weeks).
## Installation
@@ -65,10 +66,11 @@ external packages, some of which are included with Joshua.
- [SRILM](http://www.speech.sri.com/projects/srilm/) (not included)
- By default, the pipeline uses a Java program from the
- [Berkeley LM](http://code.google.com/p/berkeleylm/) package that constructs an
- Kneser-Ney-smoothed language model in ARPA format from the target side of your training data. If
- you wish to use SRILM instead, you need to do the following:
+ By default, the pipeline uses the included [KenLM](https://kheafield.com/code/kenlm/) for
+ building (and also querying) language models. Joshua also includes a Java program from the
+ [Berkeley LM](http://code.google.com/p/berkeleylm/) package that contains code for constructing a
+ Kneser-Ney-smoothed language model in ARPA format from the target side of your training data.
+ There is no need to use SRILM, but if you do wish to use it, you need to do the following:
1. Install SRILM and set the `$SRILM` environment variable to point to its installed location.
1. Add the `--lm-gen srilm` flag to your pipeline invocation.
@@ -89,7 +91,7 @@ intermediate files in the *run directory*. By default, the run directory is the
but it can be changed with the `--rundir` parameter.
For this quick start, we will be working with the example that can be found in
-`$JOSHUA/examples/pipeline`. This example contains 1,000 sentences of Urdu-English data (the full
+`$JOSHUA/examples/training`. This example contains 1,000 sentences of Urdu-English data (the full
dataset is available as part of the
[Indian languages parallel corpora](/indian-parallel-corpora/) with
100-sentence tuning and test sets with four references each.
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/29556b18/releases/6.0/index.html
----------------------------------------------------------------------
diff --git a/releases/6.0/index.html b/releases/6.0/index.html
index 8e9a1af..91afe49 100644
--- a/releases/6.0/index.html
+++ b/releases/6.0/index.html
@@ -2,15 +2,6 @@
<html lang="en-US">
<head>
<meta charset="UTF-8">
- <meta http-equiv="refresh" content="1;url=http://example.com">
- <script type="text/javascript">
- window.location.href = "http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz"
- </script>
- <title>Page Redirection</title>
+ <meta http-equiv="refresh" content="1;url=http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz" />
</head>
- <body>
- If you are not redirected automatically, follow
- <a href='http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz'>this link</a>
- to download Joshua directly.
- </body>
</html>
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/29556b18/releases/current
----------------------------------------------------------------------
diff --git a/releases/current b/releases/current
new file mode 120000
index 0000000..5049538
--- /dev/null
+++ b/releases/current
@@ -0,0 +1 @@
+6.0
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/29556b18/releases/index.html
----------------------------------------------------------------------
diff --git a/releases/index.html b/releases/index.html
deleted file mode 100644
index 8ba2bfa..0000000
--- a/releases/index.html
+++ /dev/null
@@ -1,11 +0,0 @@
----
-layout: default6
-title: Joshua releases
----
-
-The current release of Joshua is version 6.0.1, which can be [downloaded
-here](6.0/). It was released on March 6, 2015.
-
-Older versions are also available:
-
-- [Version 5.0](5.0/) was released on November 1, 2014.
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/29556b18/releases/index.md
----------------------------------------------------------------------
diff --git a/releases/index.md b/releases/index.md
new file mode 100644
index 0000000..c08aebd
--- /dev/null
+++ b/releases/index.md
@@ -0,0 +1,11 @@
+---
+layout: default6
+title: Joshua releases
+---
+
+The current release of Joshua is version 6.0.2, which can be [downloaded
+here](6.0/). It was released on April 10, 2015.
+
+Older versions are also available:
+
+- [Version 5.0](5.0/) was released on November 1, 2014.
[32/50] incubator-joshua-site git commit: Updated language pack list
Posted by mj...@apache.org.
Updated language pack list
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/1535fdba
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/1535fdba
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/1535fdba
Branch: refs/heads/master
Commit: 1535fdba001ed00ac76a352739605b10a961a5d5
Parents: 5d88109
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Oct 1 15:32:24 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Oct 1 15:32:24 2015 -0400
----------------------------------------------------------------------
language-packs/index.md | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/1535fdba/language-packs/index.md
----------------------------------------------------------------------
diff --git a/language-packs/index.md b/language-packs/index.md
index 25cdf6e..bddfe1f 100644
--- a/language-packs/index.md
+++ b/language-packs/index.md
@@ -19,13 +19,34 @@ It is important to note the assumptions underlying the translation engine:
## Available language packs
-- [Spanish--English phrase-based model](es-en-phrase/) [1.9 GB], built on Europarl and the Fisher
- and CALLHOME parallel dataset.
-
-- [Arabic--English phrase-based model](ar-en-phrase/) [2.4 GB], built from
+<table>
+ <tr>
+ <th>Language pair</th>
+ <th>Type</th>
+ <th>Size</th>
+ <th>Details</th>
+ <th>Link</th>
+ </tr>
+ <tr>
+ <td>Spanish--English</td>
+ <td>phrase</td>
+ <td>1.9 GB</td>
+ <td>Built on Europarl and the Fisher and CALLHOME parallel dataset.</td>
+ <td><a href="es-en-phrase/">Download</a></td>
+ </tr>
+ <tr>
+ <td>Arabic--English</td>
+ <td>phrase</td>
+ <td>2.4 GB</td>
+ <td>Built from
[the LDC Arabic-Dialect/English parallel text](https://catalog.ldc.upenn.edu/LDC2012T09),
- [the ISI Arabic--English automatically extracted parallel text](https://catalog.ldc.upenn.edu/LDC2007T08), and
+ [the ISI Arabic--English automatically extracted parallel text](https://catalog.ldc.upenn.edu/LDC20
+07T08), and
[translations of the Arabic CALLHOME transcripts](https://catalog.ldc.upenn.edu/LDC97T19),
and with an English Gigaword language model.
+ </td>
+ <td><a href="ar-en-phrase/">Download</a></td>
+ </tr>
+</table>
Have a request? Please email [Matt Post](http://cs.jhu.edu/~post/).
[21/50] incubator-joshua-site git commit: fix mattmann affil.
Posted by mj...@apache.org.
fix mattmann affil.
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/ee0be097
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/ee0be097
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/ee0be097
Branch: refs/heads/master
Commit: ee0be097676bf78f35d0ff7a6de5ad25019c76c9
Parents: 1f7b84f
Author: Chris Mattmann <ma...@apache.org>
Authored: Fri Jun 19 14:27:18 2015 -0700
Committer: Chris Mattmann <ma...@apache.org>
Committed: Fri Jun 19 14:27:18 2015 -0700
----------------------------------------------------------------------
contributors.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/ee0be097/contributors.md
----------------------------------------------------------------------
diff --git a/contributors.md b/contributors.md
index 5b716b5..68c4584 100644
--- a/contributors.md
+++ b/contributors.md
@@ -12,7 +12,7 @@ alphabetical order).
- [Adam Pauls](http://www.eecs.berkeley.edu/~adpauls/) (Berkeley)
- [Chris Callison-Burch](http://cs.jhu.edu/~ccb/) (Penn)
- [Chris Dyer](http://www.cs.cmu.edu/~cdyer/) (Carnegie Mellon)
-- [Chris Mattmann](http://sunset.usc.edu/~mattmann/) (APL, USC)
+- [Chris Mattmann](http://sunset.usc.edu/~mattmann/) (NASA JPL, USC)
- [Colin Bannard](http://comp.ling.utexas.edu/~bannard/) (UT Austin)
- [Jonathan Weese](http://cs.jhu.edu/~jonny/) (Johns Hopkins)
- Josh Schroeder (Edinburgh)
[10/50] incubator-joshua-site git commit: redirects
Posted by mj...@apache.org.
redirects
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/7277eaf1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/7277eaf1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/7277eaf1
Branch: refs/heads/master
Commit: 7277eaf14a952d9c36edab232c55cb388da091cb
Parents: 4c8fd24
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 11:38:37 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 11:38:37 2015 -0400
----------------------------------------------------------------------
releases/6.0/index.html | 7 +++++++
1 file changed, 7 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/7277eaf1/releases/6.0/index.html
----------------------------------------------------------------------
diff --git a/releases/6.0/index.html b/releases/6.0/index.html
index 51d8644..826a0a9 100644
--- a/releases/6.0/index.html
+++ b/releases/6.0/index.html
@@ -2,3 +2,10 @@
title: This page should redirect you
redirect_from: "http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz"
---
+<!DOCTYPE HTML>
+<html lang="en-US">
+ <head>
+ <meta charset="UTF-8">
+ <meta http-equiv="refresh" content="1;url=http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz" />
+ </head>
+</html>
[11/50] incubator-joshua-site git commit: playing around wtih
redirects
Posted by mj...@apache.org.
playing around wtih redirects
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/509371a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/509371a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/509371a9
Branch: refs/heads/master
Commit: 509371a920288ed6968e1bb0aa2632a310f28501
Parents: 7277eaf
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 21:47:42 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 21:47:42 2015 -0400
----------------------------------------------------------------------
releases/6.0/index.html | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/509371a9/releases/6.0/index.html
----------------------------------------------------------------------
diff --git a/releases/6.0/index.html b/releases/6.0/index.html
index 826a0a9..17ec987 100644
--- a/releases/6.0/index.html
+++ b/releases/6.0/index.html
@@ -1,11 +1,4 @@
---
-title: This page should redirect you
-redirect_from: "http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz"
+redirect_to:
+ - http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz
---
-<!DOCTYPE HTML>
-<html lang="en-US">
- <head>
- <meta charset="UTF-8">
- <meta http-equiv="refresh" content="1;url=http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz" />
- </head>
-</html>
[50/50] incubator-joshua-site git commit: ack! forgot the header
Posted by mj...@apache.org.
ack! forgot the header
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/9a4315b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/9a4315b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/9a4315b5
Branch: refs/heads/master
Commit: 9a4315b566881b1a959570425696920d4c14e31b
Parents: f287587
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Apr 1 15:19:28 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Apr 1 15:19:28 2016 -0400
----------------------------------------------------------------------
language-packs/paraphrase/index.md | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/9a4315b5/language-packs/paraphrase/index.md
----------------------------------------------------------------------
diff --git a/language-packs/paraphrase/index.md b/language-packs/paraphrase/index.md
index 6127296..b1d92ae 100644
--- a/language-packs/paraphrase/index.md
+++ b/language-packs/paraphrase/index.md
@@ -1,3 +1,8 @@
+---
+layout: default6
+title: Paraphrase Packs
+---
+
# Joshua English Paraphrase Language Packs
Coming soon...
[33/50] incubator-joshua-site git commit: added table classes
Posted by mj...@apache.org.
added table classes
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/343353c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/343353c0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/343353c0
Branch: refs/heads/master
Commit: 343353c0d45d5b552f86a8324b1058e19839f30a
Parents: 1535fdb
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Oct 1 15:42:29 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Oct 1 15:42:29 2015 -0400
----------------------------------------------------------------------
language-packs/index.md | 12 +++++-------
1 file changed, 5 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/343353c0/language-packs/index.md
----------------------------------------------------------------------
diff --git a/language-packs/index.md b/language-packs/index.md
index bddfe1f..c016de1 100644
--- a/language-packs/index.md
+++ b/language-packs/index.md
@@ -19,7 +19,7 @@ It is important to note the assumptions underlying the translation engine:
## Available language packs
-<table>
+<table class="table table-condensed">
<tr>
<th>Language pair</th>
<th>Type</th>
@@ -31,7 +31,7 @@ It is important to note the assumptions underlying the translation engine:
<td>Spanish--English</td>
<td>phrase</td>
<td>1.9 GB</td>
- <td>Built on Europarl and the Fisher and CALLHOME parallel dataset.</td>
+ <td>Built on <a href="http://www.statmt.org/europarl/">Europarl</a> and the <a href="https://catalog.ldc.upenn.edu/LDC2014T23">Fisher and CALLHOME parallel dataset</a>.</td>
<td><a href="es-en-phrase/">Download</a></td>
</tr>
<tr>
@@ -39,11 +39,9 @@ It is important to note the assumptions underlying the translation engine:
<td>phrase</td>
<td>2.4 GB</td>
<td>Built from
- [the LDC Arabic-Dialect/English parallel text](https://catalog.ldc.upenn.edu/LDC2012T09),
- [the ISI Arabic--English automatically extracted parallel text](https://catalog.ldc.upenn.edu/LDC20
-07T08), and
- [translations of the Arabic CALLHOME transcripts](https://catalog.ldc.upenn.edu/LDC97T19),
- and with an English Gigaword language model.
+ <a href="https://catalog.ldc.upenn.edu/LDC2012T09">the LDC Arabic-Dialect/English parallel text</a>,
+ <a href="https://catalog.ldc.upenn.edu/LDC2007T08">the ISI Arabic--English automatically extracted parallel text</a>,
+ and <a href="https://catalog.ldc.upenn.edu/LDC97T19">translations of the Arabic CALLHOME transcripts</a>, and with an English Gigaword language model.
</td>
<td><a href="ar-en-phrase/">Download</a></td>
</tr>
[34/50] incubator-joshua-site git commit: Updated Indian data to
official 1.0 release
Posted by mj...@apache.org.
Updated Indian data to official 1.0 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/b4032167
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/b4032167
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/b4032167
Branch: refs/heads/master
Commit: b4032167ceab9b08af7528046407ee7db4bef1fb
Parents: 343353c
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Oct 2 13:32:06 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Oct 2 13:32:06 2015 -0400
----------------------------------------------------------------------
6.0/quick-start.md | 6 +++---
data/indian-parallel-corpora/index.html | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/b4032167/6.0/quick-start.md
----------------------------------------------------------------------
diff --git a/6.0/quick-start.md b/6.0/quick-start.md
index 094b615..53814ae 100644
--- a/6.0/quick-start.md
+++ b/6.0/quick-start.md
@@ -24,9 +24,9 @@ First, download the dataset:
mkdir -p ~/models/bn-en/
cd ~/models/bn-en
- curl -L https://github.com/joshua-decoder/indian-parallel-corpora/tarball/master > indian-languages.tgz
- tar xf indian-languages.tgz
- ln -s joshua-decoder-indian-parallel-corpora-* input
+ wget -q https://github.com/joshua-decoder/indian-parallel-corpora/archive/1.0.tar.gz
+ tar xzf indian-parallel-corpora-1.0.tar.gz
+ ln -s indian-parallel-corpora-1.0 input
Then, train and test a model
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/b4032167/data/indian-parallel-corpora/index.html
----------------------------------------------------------------------
diff --git a/data/indian-parallel-corpora/index.html b/data/indian-parallel-corpora/index.html
index 29168b1..777b18e 100644
--- a/data/indian-parallel-corpora/index.html
+++ b/data/indian-parallel-corpora/index.html
@@ -10,7 +10,7 @@ title: Indian Languages Parallel Corpora
<h1>Datasets</h1>
<h2>Indian Parallel Languages</h2>
<span id="download">
- <a href="https://github.com/joshua-decoder/indian-parallel-corpora/zipball/master">Download</a>
+ <a href="https://github.com/joshua-decoder/indian-parallel-corpora/archive/1.0.tar.gz">Download</a>
</span>
</div>
</div>
[26/50] incubator-joshua-site git commit: minor updates
Posted by mj...@apache.org.
minor updates
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/4b3bdd31
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/4b3bdd31
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/4b3bdd31
Branch: refs/heads/master
Commit: 4b3bdd31023372b769d9f3bd60f01365503a9e0b
Parents: 601d9f8
Author: Matt Post <po...@cs.jhu.edu>
Authored: Mon Jun 22 22:17:24 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Mon Jun 22 22:17:24 2015 -0400
----------------------------------------------------------------------
6.0/pipeline.md | 58 ++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 45 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/4b3bdd31/6.0/pipeline.md
----------------------------------------------------------------------
diff --git a/6.0/pipeline.md b/6.0/pipeline.md
index f35f618..35d408d 100644
--- a/6.0/pipeline.md
+++ b/6.0/pipeline.md
@@ -35,7 +35,7 @@ The Joshua pipeline script is designed in the spirit of Moses' `train-model.pl`,
the user to define arbitrary execution dependency graphs. However, it is significantly simpler to
use, allowing many systems to be built with a single command (that may run for days or weeks).
-## Installation
+## Dependencies
The pipeline has no *required* external dependencies. However, it has support for a number of
external packages, some of which are included with Joshua.
@@ -67,7 +67,11 @@ external packages, some of which are included with Joshua.
standalone installation and use it to extract your grammar. This behavior will be triggered if
`$HADOOP` is undefined.
-- [SRILM](http://www.speech.sri.com/projects/srilm/) (not included)
+- [Moses](http://statmt.org/moses/) (not included). Moses is needed
+ if you wish to use its 'kbmira' tuner (--tuner kbmira), or if you
+ wish to build phrase-based models.
+
+- [SRILM](http://www.speech.sri.com/projects/srilm/) (not included; not needed; not recommended)
By default, the pipeline uses the included [KenLM](https://kheafield.com/code/kenlm/) for
building (and also querying) language models. Joshua also includes a Java program from the
@@ -83,9 +87,9 @@ external packages, some of which are included with Joshua.
having been supplanted by [KenLM](http://kheafield.com/code/kenlm/) (the default) and
BerkeleyLM).
-- [Moses](http://statmt.org/moses/) (not included)
-
-Make sure that the environment variable `$JOSHUA` is defined, and you should be all set.
+After installing any dependencies, follow the brief instructions on
+the [installation page](install.html), and then you are ready to build
+models.
## A basic pipeline run
@@ -124,6 +128,8 @@ Running the pipeline requires two main steps: data preparation and invocation.
1. Run the pipeline. The following is the minimal invocation to run the complete pipeline:
$JOSHUA/bin/pipeline.pl \
+ --rundir . \
+ --type hiero \
--corpus input/train \
--tune input/tune \
--test input/devtest \
@@ -158,7 +164,8 @@ producing BLEU scores at the end. As it runs, you will see output that looks li
took 0 seconds (0s)
...
-And in the current directory, you will see the following files (among other intermediate files
+And in the current directory, you will see the following files (among
+other files, including intermediate files
generated by the individual sub-steps).
data/
@@ -179,6 +186,8 @@ generated by the individual sub-steps).
alignments/
0/
[giza/berkeley aligner output files]
+ 1/
+ ...
training.align
thrax-hiero.conf
thrax.log
@@ -193,12 +202,21 @@ generated by the individual sub-steps).
mert.log
joshua.config.final
final-bleu
+ test/
+ model/
+ [model files]
+ output
+ final-bleu
These files will be described in more detail in subsequent sections of this tutorial.
Another useful flag is the `--rundir DIR` flag, which chdir()s to the specified directory before
running the pipeline. By default the rundir is the current directory. Changing it can be useful
-for organizing related pipeline runs. Relative paths specified to other flags (e.g., to `--corpus`
+for organizing related pipeline runs. In fact, we highly recommend
+that you organize your runs using consecutive integers, also taking a
+minute to pass a short note with the `--readme` flag, which allows you
+to quickly generate reports on [groups of related experiments](#managing).
+Relative paths specified to other flags (e.g., to `--corpus`
or `--lmfile`) are relative to the directory the pipeline was called *from*, not the rundir itself
(unless they happen to be the same, of course).
@@ -217,7 +235,7 @@ of traditional pipeline tasks:
These steps are discussed below, after a few intervening sections about high-level details of the
pipeline.
-## Managing groups of experiments
+## <a id="managing" /> Managing groups of experiments
The real utility of the pipeline comes when you use it to manage groups of experiments. Typically,
there is a held-out test set, and we want to vary a number of training parameters to determine what
@@ -225,7 +243,7 @@ effect this has on BLEU scores or some other metric. Joshua comes with a script
`$JOSHUA/scripts/training/summarize.pl` that collects information from a group of runs and reports
them to you. This script works so long as you organize your runs as follows:
-1. Your runs should be grouped together in a root directory, which I'll call `$RUNDIR`.
+1. Your runs should be grouped together in a root directory, which I'll call `$EXPDIR`.
2. For comparison purposes, the runs should all be evaluated on the same test set.
@@ -241,6 +259,10 @@ the summarize script:
[other files]
2/
README.txt
+ test/
+ final-bleu
+ final-times
+ [other files]
...
You can get such directories using the `--rundir N` flag to the pipeline.
@@ -259,9 +281,11 @@ More details are below.
## Grammar options
-Joshua can extract three types of grammars: Hiero grammars, GHKM, and SAMT grammars. As described
-on the [file formats page](file-formats.html), all of them are encoded into the same file format,
-but they differ in terms of the richness of their nonterminal sets.
+Hierarchical Joshua can extract three types of grammars: Hiero
+grammars, GHKM, and SAMT grammars. As described on the
+[file formats page](file-formats.html), all of them are encoded into
+the same file format, but they differ in terms of the richness of
+their nonterminal sets.
Hiero grammars make use of a single nonterminals, and are extracted by computing phrases from
word-based alignments and then subtracting out phrase differences. More detail can be found in
@@ -280,6 +304,12 @@ By default, the Joshua pipeline extract a Hiero grammar, but this can be altered
but you can also use Moses' extractor with `--ghkm-extractor moses`. Galley's extractor only outputs
two features, so the scores tend to be significantly lower than that of Moses'.
+Joshua (new in version 6) also includes an unlexicalized phrase-based
+decoder. Building a phrase-based model requires you to have Moses
+installed, since its `train-model.perl` script is used to extract the
+phrase table. You can enable this by defining the `$MOSES` environment
+variable and then specifying `--type phrase`.
+
## Other high-level options
The following command-line arguments control run-time behavior of multiple steps:
@@ -294,7 +324,9 @@ The following command-line arguments control run-time behavior of multiple steps
This enables parallel operation over a cluster using the qsub command. This feature is not
well-documented at this point, but you will likely want to edit the file
`$JOSHUA/scripts/training/parallelize/LocalConfig.pm` to setup your qsub environment, and may also
- want to pass specific qsub commands via the `--qsub-args "ARGS"` command.
+ want to pass specific qsub commands via the `--qsub-args "ARGS"`
+ command. We suggest you stick to the standard Joshua model that
+ tries to use as many cores as are available with the `--threads N` option.
## Restarting failed runs
[17/50] incubator-joshua-site git commit: indentation
Posted by mj...@apache.org.
indentation
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/899bd5fb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/899bd5fb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/899bd5fb
Branch: refs/heads/master
Commit: 899bd5fb93b43d6d6aa8dd3d039f17f13b367dde
Parents: 3dd4d77
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed Jun 10 10:05:12 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed Jun 10 10:05:12 2015 -0400
----------------------------------------------------------------------
6.0/install.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/899bd5fb/6.0/install.md
----------------------------------------------------------------------
diff --git a/6.0/install.md b/6.0/install.md
index d242f86..7b544f4 100644
--- a/6.0/install.md
+++ b/6.0/install.md
@@ -13,14 +13,14 @@ install the decoder. There are no external dependencies.
also `$HADOOP`, if you have a Hadoop installation (Joshua's pipeline looks
for `$HADOOP/bin/hadoop`.
- export JAVA_HOME=/path/to/java
+ export JAVA_HOME=/path/to/java
- # JAVA_HOME is not very standardized. Here are some places to look:
- # OS X: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
- # Linux: export JAVA_HOME=/usr/java/default
+ # JAVA_HOME is not very standardized. Here are some places to look:
+ # OS X: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
+ # Linux: export JAVA_HOME=/usr/java/default
- export HADOOP=/usr
-
+ export HADOOP=/usr
+
1. Download Joshua
wget -q http://cs.jhu.edu/~post/joshua-v{{ site.data.joshua.release_version }}.tgz
[35/50] incubator-joshua-site git commit: updated packedGrammar
documentation for 6.0.5 release
Posted by mj...@apache.org.
updated packedGrammar documentation for 6.0.5 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/0652f03d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/0652f03d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/0652f03d
Branch: refs/heads/master
Commit: 0652f03da9134c8ebedf5e4062764a2f5b275fdb
Parents: b403216
Author: Felix Hieber <fh...@amazon.com>
Authored: Thu Oct 22 14:13:57 2015 +0200
Committer: Felix Hieber <fh...@amazon.com>
Committed: Thu Oct 22 14:13:57 2015 +0200
----------------------------------------------------------------------
6.0/packing.md | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/0652f03d/6.0/packing.md
----------------------------------------------------------------------
diff --git a/6.0/packing.md b/6.0/packing.md
index 8189c66..8d84004 100644
--- a/6.0/packing.md
+++ b/6.0/packing.md
@@ -18,12 +18,12 @@ The script can be found at
example usage. You can then add it to a Joshua config file, simply
replacing a `tm` path to the compressed text-file format with a path
to the packed grammar directory (Joshua will automatically detect that
-it is packed.
+it is packed, since a packed grammar is a directory).
-Packing the grammar requires first sorting it, which can take quite a
-bit of temporary space.
+Packing the grammar requires first sorting it by the rules source side,
+which can take quite a bit of temporary space.
-*CAVEAT*: You may run into problems packing very large hiero
+*CAVEAT*: You may run into problems packing very very large Hiero
grammars. Email the support list if you do.
### Examples
@@ -31,7 +31,7 @@ bit of temporary space.
A Hiero grammar, using the compressed text file version:
tm = hiero -owner pt -maxspan 20 -path grammar.filtered.gz
-
+
Pack it:
$JOSHUA/scripts/support/grammar-packer.pl grammar.filtered.gz grammar.packed
@@ -51,3 +51,24 @@ You have a different temp file location:
Update the config file line:
tm = hiero -owner pt -maxspan 20 -path grammar.packed
+
+### Using multiple packed grammars (Joshua 6.0.5)
+
+Packed grammars serialize their vocabularies which prevented the use of multiple
+packed grammars during decoding. With Joshua 6.0.5, it is possible to use multiple packed grammars during decoding if they have the same serialized vocabulary.
+This is achieved by packing these grammars jointly using a revised packing CLI.
+
+To pack multiple grammars:
+
+ $JOSHUA/scripts/support/grammar-packer.pl grammar1.filtered.gz grammar2.filtered.gz [...] grammar1.packed grammar2.packed [...]
+
+This will produce two packed grammars with the same vocabulary. To use them in the decoder, put this in your ```joshua.config```:
+
+ tm = hiero -owner pt -maxspan 20 -path grammar1.packed
+ tm = hiero -owner pt2 -maxspan 20 -path grammar2.packed
+
+Note the different owners.
+If you are trying to load multiple packed grammars that do not have the same
+vocabulary, the decoder will throw a RuntimeException at loading time:
+
+ Exception in thread "main" java.lang.RuntimeException: Trying to load multiple packed grammars with different vocabularies! Have you packed them jointly?
[12/50] incubator-joshua-site git commit: fixed download command
Posted by mj...@apache.org.
fixed download command
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/81748f5b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/81748f5b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/81748f5b
Branch: refs/heads/master
Commit: 81748f5b7abd7b76a372c212135610ca9a5a1b2a
Parents: 509371a
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 21:53:35 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 21:53:35 2015 -0400
----------------------------------------------------------------------
6.0/install.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/81748f5b/6.0/install.md
----------------------------------------------------------------------
diff --git a/6.0/install.md b/6.0/install.md
index f6165e5..df187e5 100644
--- a/6.0/install.md
+++ b/6.0/install.md
@@ -11,13 +11,13 @@ install the decoder. There are no external dependencies.
1. Download Joshua
- curl -#L http://joshua-decoder.org/releases/6.0 > joshua-v6.0.tgz
+ wget -q http://cs.jhu.edu/~post/joshua-v{{ site.data.joshua.release_version }}.tgz
1. Next, unpack it, set environment variables, and compile everything. You need to define the
`$JOSHUA` environment variable (which in turn requires `$JAVA_HOME`).
- tar xzf joshua-v6.0.tgz
- cd joshua-v6.0
+ tar xzf joshua-v{{ site.data.joshua.release_version }}.tgz
+ cd joshua-v{{ site.data.joshua.release_version }}
# for bash
export JAVA_HOME=/path/to/java
@@ -39,19 +39,19 @@ If you wish to build models for new language pairs from existing data
(such as the [WMT data](http://statmt.org/wmt14/)), you need to
install some additional dependencies.
-1. For learning hierarchical models, Joshua includes a tool called [Thrax](/6.0/thrax.html), which
+1. For learning hierarchical models, Joshua includes a tool called [Thrax](thrax.html), which
is built on Hadoop. If you have a Hadoop installation, make sure that the environment variable
`$HADOOP` is set and points to it. If you don't, Joshua will roll one out for you in standalone
mode. Hadoop is only needed if you plan to build new models with Joshua.
1. You will need to install Moses if either of the following applies to you:
- - You wish to build [phrase-based models](/6.0/phrase.html) (Joshua 6.0 includes a phrase-based
+ - You wish to build [phrase-based models](phrase.html) (Joshua 6 includes a phrase-based
decoder, but not the tools for building such a model)
- You are building your own models (phrase- or syntax-based) and wish to use Cherry & Foster's
[batch MIRA tuner](http://aclweb.org/anthology-new/N/N12/N12-1047v2.pdf) instead of the included
-MERT implementation, [Z-MERT](/6.0/zmert.html).
+MERT implementation, [Z-MERT](zmert.html).
Follow [the instructions for installing Moses
here](http://www.statmt.org/moses/?n=Development.GetStarted), and then define the `$MOSES`
[27/50] incubator-joshua-site git commit: updated packing info
Posted by mj...@apache.org.
updated packing info
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/700e8581
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/700e8581
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/700e8581
Branch: refs/heads/master
Commit: 700e85817db9f6dce152131b196f8b34195b2255
Parents: 4b3bdd3
Author: Matt Post <po...@cs.jhu.edu>
Authored: Mon Jun 22 22:17:43 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Mon Jun 22 22:17:43 2015 -0400
----------------------------------------------------------------------
6.0/packing.md | 91 +++++++++++++++++----------------------------
_layouts/default6.html | 1 +
2 files changed, 35 insertions(+), 57 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/700e8581/6.0/packing.md
----------------------------------------------------------------------
diff --git a/6.0/packing.md b/6.0/packing.md
index e911095..8189c66 100644
--- a/6.0/packing.md
+++ b/6.0/packing.md
@@ -4,73 +4,50 @@ category: advanced
title: Grammar Packing
---
-Grammar packing refers to the process of taking a textual grammar output by [Thrax](thrax.html) and
-efficiently encoding it for use by Joshua. Packing the grammar results in significantly faster load
-times for very large grammars.
+Grammar packing refers to the process of taking a textual grammar
+output by [Thrax](thrax.html) (or Moses, for phrase-based models) and
+efficiently encoding it so that it can be loaded
+[very quickly](https://aclweb.org/anthology/W/W12/W12-3134.pdf) ---
+packing the grammar results in significantly faster load times for
+very large grammars. Packing is done automatically by the
+[Joshua pipeline](pipeline.html), but you can also run the packer
+manually.
-Soon, the [Joshua pipeline script](pipeline.html) will add support for grammar packing
-automatically, and we will provide a script that automates these steps for you.
+The script can be found at
+`$JOSHUA/scripts/support/grammar-packer.pl`. See that script for
+example usage. You can then add it to a Joshua config file, simply
+replacing a `tm` path to the compressed text-file format with a path
+to the packed grammar directory (Joshua will automatically detect that
+it is packed.
-1. Make sure the grammar is labeled. A labeled grammar is one that has feature names attached to
-each of the feature values in each row of the grammar file. Here is a line from an unlabeled
-grammar:
+Packing the grammar requires first sorting it, which can take quite a
+bit of temporary space.
- [X] ||| [X,1] অন্যান্য [X,2] ||| [X,1] other [X,2] ||| 0 0 1 0 0 1.02184
+*CAVEAT*: You may run into problems packing very large hiero
+ grammars. Email the support list if you do.
- and here is one from an labeled grammar (note that the labels are not very useful):
+### Examples
- [X] ||| [X,1] অন্যান্য [X,2] ||| [X,1] other [X,2] ||| f1=0 f2=0 f3=1 f4=0 f5=0 f6=1.02184
+A Hiero grammar, using the compressed text file version:
- If your grammar is not labeled, you can use the script `$JOSHUA/scripts/label_grammar.py`:
-
- zcat grammar.gz | $JOSHUA/scripts/label_grammar.py > grammar-labeled.gz
+ tm = hiero -owner pt -maxspan 20 -path grammar.filtered.gz
+
+Pack it:
- As a side-effect of this step is to produce a file 'dense_map' in the current directory,
- containing the mapping between feature names and feature columns. This file is needed in later
- steps.
+ $JOSHUA/scripts/support/grammar-packer.pl grammar.filtered.gz grammar.packed
-1. The packer needs a sorted grammar. It is sufficient to sort by the first word:
+Pack a really big grammar:
- zcat grammar-labeled.gz | sort -k3,3 | gzip > grammar-sorted.gz
-
- (The reason we need a sorted grammar is because the packer stores the grammar in a trie. The
- pieces can't be more than 2 GB due to Java limitations, so we need to ensure that rules are
- grouped by the first arc in the trie to avoid redundancy across tries and to simplify the
- lookup).
-
-1. In order to pack the grammar, we need two pieces of information: (1) a packer configuration file,
- and (2) a dense map file.
-
- 1. Write a packer config file. This file specifies items such as the chunk size (for the packed
- pieces) and the quantization classes and types for each feature name. Examples can be found
- at
-
- $JOSHUA/test/packed/packer.config
- $JOSHUA/test/bn-en/packed/packer.quantized
- $JOSHUA/test/bn-en/packed/packer.uncompressed
-
- The quantizer lines in the packer config file have the following format:
-
- quantizer TYPE FEATURES
-
- where `TYPE` is one of `boolean`, `float`, `byte`, or `8bit`, and `FEATURES` is a
- space-delimited list of feature names that have that quantization type.
-
- 1. Write a dense_map file. If you labeled an unlabeled grammar, this was produced for you as a
- side product of the `label_grammar.py` script you called in Step 1. Otherwise, you need to
- create a file that lists the mapping between feature names and (0-indexed) columns in the
- grammar, one per line, in the following format:
-
- feature-index feature-name
-
-1. To pack the grammar, type the following command:
+ $JOSHUA/scripts/support/grammar-packer.pl -m 30g grammar.filtered.gz grammar.packed
+
+Be a little more verbose:
+
+ $JOSHUA/scripts/support/grammar-packer.pl -m 30g grammar.filtered.gz grammar.packed
- java -cp $JOSHUA/bin joshua.tools.GrammarPacker -c PACKER_CONFIG_FILE -p OUTPUT_DIR -g GRAMMAR_FILE
+You have a different temp file location:
- This will read in your packer configuration file and your grammar, and produced a packed grammar
- in the output directory.
+ $JOSHUA/scripts/support/grammar-packer.pl -T /local grammar.filtered.gz grammar.packed
-1. To use the packed grammar, just point to the packed directory in your Joshua configuration file.
+Update the config file line:
- tm-file = packed-grammar/
- tm-format = packed
+ tm = hiero -owner pt -maxspan 20 -path grammar.packed
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/700e8581/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index 3d19a7b..3737c63 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -86,6 +86,7 @@
<li><a href="/6.0/bundle.html">Building language packs</a></li>
<li><a href="/6.0/decoder.html">Decoder options</a></li>
<li><a href="/6.0/file-formats.html">File formats</a></li>
+ <li><a href="/6.0/packing.html">Packing TMs</a></li>
</ol>
</div>
[30/50] incubator-joshua-site git commit: added support links and
embed
Posted by mj...@apache.org.
added support links and embed
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/1a7aa71a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/1a7aa71a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/1a7aa71a
Branch: refs/heads/master
Commit: 1a7aa71a98d2332d5ef04e4211ac98a06da8942e
Parents: 676736c
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Jul 23 22:04:44 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Jul 23 22:04:44 2015 -0400
----------------------------------------------------------------------
_layouts/default6.html | 1 +
support/index.md | 25 +++++++++++++++++++++++++
2 files changed, 26 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/1a7aa71a/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index 2aaed53..0ca8adb 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -27,6 +27,7 @@
<!-- <a class="blog-nav-item" href="/6.0/whats-new.html">New features</a> -->
<a class="blog-nav-item" href="/language-packs/">Language packs</a>
<a class="blog-nav-item" href="/data/">Datasets</a>
+ <a class="blog-nav-item" href="/support/">Support</a>
<a class="blog-nav-item" href="/contributors.html">Contributors</a>
</nav>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/1a7aa71a/support/index.md
----------------------------------------------------------------------
diff --git a/support/index.md b/support/index.md
new file mode 100644
index 0000000..4f05939
--- /dev/null
+++ b/support/index.md
@@ -0,0 +1,25 @@
+---
+layout: default6
+title: Support
+---
+
+You can
+[join the Joshua users group](https://groups.google.com/forum/?fromgroups#!forum/joshua_support)
+for user-oriented techical support, or the
+[developer's list](https://groups.google.com/forum/?fromgroups#!forum/joshua_developers)
+for developer support. You can also use the form below.
+
+<iframe id="forum_embed"
+ src="javascript:void(0)"
+ scrolling="no"
+ frameborder="0"
+ width="900"
+ height="700">
+</iframe>
+
+<script type="text/javascript">
+ document.getElementById("forum_embed").src =
+ "https://groups.google.com/forum/embed/?place=forum/joshua_support" +
+ "&showsearch=true&showpopout=true&parenturl=" +
+ encodeURIComponent(window.location.href);
+</script>
[40/50] incubator-joshua-site git commit: Updated with instructions
about installing joshua-runtime
Posted by mj...@apache.org.
Updated with instructions about installing joshua-runtime
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/33eedd8d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/33eedd8d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/33eedd8d
Branch: refs/heads/master
Commit: 33eedd8db5753379d4fa90027c12341734bcc1f6
Parents: eb822f3
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Nov 6 09:50:09 2015 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Nov 6 09:50:09 2015 -0500
----------------------------------------------------------------------
6.0/install.md | 47 +++++++++++++++++++++++++++-----------------
language-packs/index.md | 8 ++++++++
2 files changed, 37 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/33eedd8d/6.0/install.md
----------------------------------------------------------------------
diff --git a/6.0/install.md b/6.0/install.md
index 68c4720..6d13e4d 100644
--- a/6.0/install.md
+++ b/6.0/install.md
@@ -5,30 +5,34 @@ title: Installation
### Download and install
-To use Joshua as a standalone decoder (with
-[language packs](/language-packs/)), you only need to download and
-install the decoder. There are no external dependencies.
+To use Joshua as a standalone decoder (with [language packs](/language-packs/)), you only need to download and install the runtime version of the decoder.
+If you also wish to build translation models from your own data, you will want to install the full version.
+See the instructions below.
-1. Set up some basic environment variables. You need to define `$JAVA_HOME` and
- also `$HADOOP`, if you have a Hadoop installation (Joshua's pipeline looks
- for `$HADOOP/bin/hadoop`.
+1. Set up some basic environment variables.
+ You need to define `$JAVA_HOME`
export JAVA_HOME=/path/to/java
# JAVA_HOME is not very standardized. Here are some places to look:
# OS X: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
# Linux: export JAVA_HOME=/usr/java/default
-
+
+1. If you are installing the full version of Joshua, you also need to define `$HADOOP` to point to your Hadoop installation.
+ (Joshua looks for the Hadoop executuble in `$HADOOP/bin/hadoop`)
+
export HADOOP=/usr
-
-1. Download Joshua
- wget -q http://cs.jhu.edu/~post/files/joshua-{{ site.data.joshua.release_version }}.tgz
+ If you don't have a Hadoop installation, [Joshua's pipeline](pipeline.html) can install a standalone version for you.
+
+1. To install just the runtime version of Joshua, type
-1. Next, unpack it and compile everything.
+ wget -q http://cs.jhu.edu/~post/files/joshua-runtime-{{ site.data.joshua.release_version }}.tgz
- tar xzf joshua-{{ site.data.joshua.release_version }}.tgz
- cd joshua-{{ site.data.joshua.release_version }}
+ Then build everything
+
+ tar xzf joshua-runtime-{{ site.data.joshua.release_version }}.tgz
+ cd joshua-runtime-{{ site.data.joshua.release_version }}
# Add this to your init files
export JOSHUA=$(pwd)
@@ -36,15 +40,22 @@ install the decoder. There are no external dependencies.
# build everything
ant
- This compiles Joshua and also a number of support tools, such as KenLM and GIZA++.
+1. To instead install the full version, type
+
+ wget -q http://cs.jhu.edu/~post/files/joshua-{{ site.data.joshua.release_version }}.tgz
+
+ tar xzf joshua-{{ site.data.joshua.release_version }}.tgz
+ cd joshua-{{ site.data.joshua.release_version }}
-1. [Download a model](/language-packes/) and start translating!
+ # Add this to your init files
+ export JOSHUA=$(pwd)
+
+ # build everything
+ ant
### Building new models
-If you wish to build models for new language pairs from existing data
-(such as the [WMT data](http://statmt.org/wmt14/)), you need to
-install some additional dependencies.
+If you wish to build models for new language pairs from existing data (such as the [WMT data](http://statmt.org/wmt14/)), you need to install some additional dependencies.
1. For learning hierarchical models, Joshua includes a tool called [Thrax](thrax.html), which
is built on Hadoop. If you have a Hadoop installation, make sure that the environment variable
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/33eedd8d/language-packs/index.md
----------------------------------------------------------------------
diff --git a/language-packs/index.md b/language-packs/index.md
index c016de1..c07b6f4 100644
--- a/language-packs/index.md
+++ b/language-packs/index.md
@@ -17,6 +17,14 @@ It is important to note the assumptions underlying the translation engine:
- Additionally, the input must be tokenized. To tokenize your data, you can use the script provided
in each language pack.
+## Runtime decoder
+
+The default release version of Joshua includes hundreds of megabytes of dependencies used
+for building models from your own language pairs. If you only wish to run language packs
+(effectively using Joshua as a black-box translation engine), you can install a "light" version
+of Joshua that includes only dependencies needed to run the translation piece. See the notes
+on [the installation page](../6/install.html).
+
## Available language packs
<table class="table table-condensed">
[39/50] incubator-joshua-site git commit: updated version link
Posted by mj...@apache.org.
updated version link
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/eb822f35
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/eb822f35
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/eb822f35
Branch: refs/heads/master
Commit: eb822f35507d8b6be91e78da093af287707005ce
Parents: 77ad61f
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Nov 5 07:11:22 2015 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Nov 5 07:11:22 2015 -0500
----------------------------------------------------------------------
releases/6.0/index.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/eb822f35/releases/6.0/index.html
----------------------------------------------------------------------
diff --git a/releases/6.0/index.html b/releases/6.0/index.html
index 17ec987..e845624 100644
--- a/releases/6.0/index.html
+++ b/releases/6.0/index.html
@@ -1,4 +1,4 @@
---
redirect_to:
- - http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz
+ - http://cs.jhu.edu/~post/files/joshua-{{ site.data.joshua.release_version }}.tgz
---
[46/50] incubator-joshua-site git commit: added dummy title to get
website to update
Posted by mj...@apache.org.
added dummy title to get website to update
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/723da06b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/723da06b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/723da06b
Branch: refs/heads/master
Commit: 723da06b78a9ad677ee68c102aaa3f48fc58a825
Parents: 846b566
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Apr 1 14:50:58 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Apr 1 14:50:58 2016 -0400
----------------------------------------------------------------------
language-packs/paraphrase/index.md | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/723da06b/language-packs/paraphrase/index.md
----------------------------------------------------------------------
diff --git a/language-packs/paraphrase/index.md b/language-packs/paraphrase/index.md
index 684e27f..6127296 100644
--- a/language-packs/paraphrase/index.md
+++ b/language-packs/paraphrase/index.md
@@ -1 +1,3 @@
+# Joshua English Paraphrase Language Packs
+
Coming soon...
[24/50] incubator-joshua-site git commit: updated documentation on
-output-format
Posted by mj...@apache.org.
updated documentation on -output-format
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/f74fc8cd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/f74fc8cd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/f74fc8cd
Branch: refs/heads/master
Commit: f74fc8cd24e04a9035bf0fc89cbd91772c6a5f96
Parents: 62a3273
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Jun 19 22:09:03 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Jun 19 22:09:03 2015 -0400
----------------------------------------------------------------------
6.0/decoder.md | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/f74fc8cd/6.0/decoder.md
----------------------------------------------------------------------
diff --git a/6.0/decoder.md b/6.0/decoder.md
index 295cf87..149f5e6 100644
--- a/6.0/decoder.md
+++ b/6.0/decoder.md
@@ -257,7 +257,9 @@ where the indices correspond to the order of the language model declaration line
$ echo "¿ who you lookin' at , mr. ?" | $JOSHUA/bin/decoder -output-format "%S" -mark-oovs false 2> /dev/null
¿Who you lookin' at, Mr.?
- - `%t`: the synchronous derivation
+ - `%t`: the target-side tree projection, all printed on one line (PTB style)
+
+ - `%d`: the synchronous derivation, with each rules printed indented on their own lines
- `%f`: the list of feature values (as name=value pairs)
@@ -265,7 +267,7 @@ where the indices correspond to the order of the language model declaration line
- `%w`: the weight vector (unimplemented)
- - `%a`: the alignments between source and target words (unimplemented)
+ - `%a`: the alignments between source and target words (currently broken for hierarchical mode)
The default value is
[23/50] incubator-joshua-site git commit: fixed run bundler
documentation for 6.0.4+
Posted by mj...@apache.org.
fixed run bundler documentation for 6.0.4+
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/62a32736
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/62a32736
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/62a32736
Branch: refs/heads/master
Commit: 62a32736db6d564fe4ed45ec9ee0471d8e3ff01c
Parents: 3fd1606
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Jun 19 20:42:30 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Jun 19 20:42:30 2015 -0400
----------------------------------------------------------------------
6.0/bundle.md | 92 +++++++++++++++++++++++++++++++++---------------------
1 file changed, 57 insertions(+), 35 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/62a32736/6.0/bundle.md
----------------------------------------------------------------------
diff --git a/6.0/bundle.md b/6.0/bundle.md
index eb2ff5a..f433172 100644
--- a/6.0/bundle.md
+++ b/6.0/bundle.md
@@ -14,50 +14,60 @@ tuned a model using the provided
from a pipeline training directory and bundles them together for easy
distribution and release.
-The script takes a number of mandatory arguments in the following
-order:
+The script takes just two mandatory arguments in the following order:
1. The path to the Joshua configuration file to base the bundle
on. This file should contain the tuned weights from the tuning run, so
you can use either the final tuned file from the tuning run
- (`tune/1/joshua.config.final`) or from the test run
- (`test/1/joshua.config`).
-1. The base directory of the pipeline run directory. This is needed
- because all of the paths in the config file are relative to this
- directory.
+ (`tune/joshua.config.final`) or from the test run
+ (`test/model/joshua.config`).
1. The directory to place the language pack in. If this directory
already exists, the script will die, unless you also pass `--force`.
-The config file options that are used in the pipeline are likely not
-the ones you want if you release a model. For example, the tuning
-configuration file contains options that tell Joshua to output 300
-translation candidates for each sentence (`-top-n 300`) and to include lots of
-detail about each translation (`-output-format '%i ||| %s ||| %f ||| %c'`).
-Because of this, you will want to tell the run bundler to change many
-of the config file options to be more geared towards human-readable
-output.
+In addition, there are a number of other arguments that may be important.
-Another important issue has to do with the translation model (the
-"TM", also sometimes called the grammar). The translation model can be
-very large, so that it takes a long time to load and to
-[pack](packing.html). To reduce this time, the translation model is
-filtered against the tuning and testing data in the pipeline, and
-these filtered models will be what is listed in the source config
-files. However, when exporting a model for use as a language pack, you
-need to export the full model instead of the filtered one. The
-`--tm` parameter is used to accomplish this; it takes an argument
-specifying the path to the full model. Multiple `--tm` parameters can
-be passed; they will replace TMs found in the config file in order. If
-you would additionally like the large model to be
-[packed](packing.html) (which reformats it so that it can be quickly
-loaded at run time), you can use `--pack-tm` instead. You can only
-pack one TM, but typically there is only one anyway.
+- `--root /path/to/root`. If file paths in the Joshua config file are
+ not absolute, you need to provide relative root. If you specify a
+ tuned pipeline file (such as `tune/joshua.config.final` above), the
+ paths should all be absolute. If you instead provide a config file
+ from a previous run bundle (e.g., `test/model/joshua.config`), the
+ bundle directory above is the relative root.
-Here is an example invocation:
+- The config file options that are used in the pipeline are likely not
+ the ones you want if you release a model. For example, the tuning
+ configuration file contains options that tell Joshua to output 300
+ translation candidates for each sentence (`-top-n 300`) and to
+ include lots of detail about each translation (`-output-format '%i
+ ||| %s ||| %f ||| %c'`). Because of this, you will want to tell the
+ run bundler to change many of the config file options to be more
+ geared towards human-readable output. The default copy-config
+ options are options are `-top-n 0 -output-format %S -mark-oovs
+ false`, which accomplishes exactly this (human readability).
+
+- A very important issue has to do with the translation model (the
+ "TM", also sometimes called the grammar or phrase table). The
+ translation model can be very large, so that it takes a long time to
+ load and to [pack](packing.html). To reduce this time during model
+ training, the translation model is filtered against the tuning and
+ testing data in the pipeline, and these filtered models will be what
+ is listed in the source config files. However, when exporting a
+ model for use as a language pack, you need to export the full model
+ instead of the filtered one so as to maximize your coverage on new
+ test data. The `--tm` parameter is used to accomplish this; it takes
+ an argument specifying the path to the full model. If you would
+ additionally like the large model to be [packed](packing.html) (this
+ is recommended; it reformats the TM so that it can be quickly loaded
+ at run time), you can use `--pack-tm` instead. You can only pack one
+ TM (but typically there is only TM anyway). Multiple `--tm`
+ parameters can be passed; they will replace TMs found in the config
+ file in the order they are found.
+
+Here is an example invocation for packing a hierarchical model using
+the final tuned Joshua config file:
./run-bundler.py \
- --force \
- /path/to/rundir/test/1/joshua.config \
+ --force --verbose \
+ /path/to/rundir/tune/joshua.config.final \
language-pack-YYYY-MM-DD \
--root /path/to/rundir \
--pack-tm /path/to/rundir/grammar.gz \
@@ -74,5 +84,17 @@ before placing it in the bundle. Note that these arguments to
`--copy-config` are the default, so you could leave this off entirely.
See [this page](decoder.html) for a longer list of decoder options.
-A new directory `language-pack-YYYY-MM-DD` will be created along with
-a README and a number of support files.
+This command is a slight variation used for phrase-based models, which
+instead takes the test-set Joshua config (the result is the same):
+
+ ./run-bundler.py \
+ --force --verbose \
+ /path/to/rundir/test/model/joshua.config \
+ --root /path/to/rundir/test/model \
+ language-pack-YYYY-MM-DD \
+ --pack-tm /path/to/rundir/model/phrase-table.gz \
+ --server-port 5674
+
+In both cases, a new directory `language-pack-YYYY-MM-DD` will be
+created along with a README and a number of support files.
+
[43/50] incubator-joshua-site git commit: clarified some text (should
fix #5)
Posted by mj...@apache.org.
clarified some text (should fix #5)
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/3677ad28
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/3677ad28
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/3677ad28
Branch: refs/heads/master
Commit: 3677ad286129308b61d55611d4dd1b604a22931a
Parents: a0cf2f5
Author: Matt Post <po...@cs.jhu.edu>
Authored: Mon Jan 11 16:04:36 2016 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Mon Jan 11 16:04:36 2016 -0500
----------------------------------------------------------------------
6.0/pipeline.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/3677ad28/6.0/pipeline.md
----------------------------------------------------------------------
diff --git a/6.0/pipeline.md b/6.0/pipeline.md
index 35d408d..4389435 100644
--- a/6.0/pipeline.md
+++ b/6.0/pipeline.md
@@ -144,10 +144,10 @@ Running the pipeline requires two main steps: data preparation and invocation.
extension to a language identified by the "en" extension, that the training data can be found at
`input/train.en` and `input/train.ur`, and so on.
-*Don't* run the pipeline directly from `$JOSHUA`. We recommend creating a run directory somewhere
- else to contain all of your experiments in some other location. The advantage to this (apart from
- not clobbering part of the Joshua install) is that Joshua provides support scripts for visualizing
- the results of a series of experiments that only work if you
+*Don't* run the pipeline directly from `$JOSHUA`, or, for that matter, in any directory with lots of other files.
+This can cause problems because the pipeline creates lots of files under `--rundir` that can clobber existing files.
+You should run experiments in a clean directory.
+For example, if you have Joshua installed in `$HOME/code/joshua`, manage your runs in a different location, such as `$HOME/expts/joshua`.
Assuming no problems arise, this command will run the complete pipeline in about 20 minutes,
producing BLEU scores at the end. As it runs, you will see output that looks like the following:
[18/50] incubator-joshua-site git commit: fixed corpus name
Posted by mj...@apache.org.
fixed corpus name
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/ac9e2c97
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/ac9e2c97
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/ac9e2c97
Branch: refs/heads/master
Commit: ac9e2c979011e20c37af1188cb3fc93a59f59463
Parents: 899bd5f
Author: Matt Post <po...@cs.jhu.edu>
Authored: Mon Jun 15 17:10:17 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Mon Jun 15 17:10:17 2015 -0400
----------------------------------------------------------------------
6 | 1 +
6.0/tutorial.md | 9 +++++----
2 files changed, 6 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/ac9e2c97/6
----------------------------------------------------------------------
diff --git a/6 b/6
new file mode 120000
index 0000000..5049538
--- /dev/null
+++ b/6
@@ -0,0 +1 @@
+6.0
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/ac9e2c97/6.0/tutorial.md
----------------------------------------------------------------------
diff --git a/6.0/tutorial.md b/6.0/tutorial.md
index d167cdc..482162f 100644
--- a/6.0/tutorial.md
+++ b/6.0/tutorial.md
@@ -34,12 +34,12 @@ data was collected by translating transcribed speech from previous LDC releases.
Download the data and install it somewhere:
cd ~/data
- wget --no-check -O fisher-spanish-corpus.zip https://github.com/joshua-decoder/fisher-callhome-corpus/archive/master.zip
- unzip fisher-spanish-corpus.zip
+ wget --no-check -O fisher-callhome-corpus.zip https://github.com/joshua-decoder/fisher-callhome-corpus/archive/master.zip
+ unzip fisher-callhome-corpus.zip
Then define the environment variable `$FISHER` to point to it:
- cd ~/data/fisher-spanish-corpus-master
+ cd ~/data/fisher-callhome-corpus-master
export FISHER=$(pwd)
### Preparing the data
@@ -76,7 +76,8 @@ this **not** be inside your `$JOSHUA` directory*.
We will now create the baseline run, using a particular directory structure for experiments that
will allow us to take advantage of scripts provided with Joshua for displaying the results of many
-related experiments. Because this can take quite some time to run, we are going to add a crippling
+related experiments. Because this can take quite some time to run, we are going to reduce the model
+by quite a bit by
restriction: Joshua will only use sentences in the training sets with ten or fewer words on either
side (Spanish or English):
[07/50] incubator-joshua-site git commit: Fixing redirect
Posted by mj...@apache.org.
Fixing redirect
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/2c8c9aab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/2c8c9aab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/2c8c9aab
Branch: refs/heads/master
Commit: 2c8c9aabe2ad226fddc34f8e528c8fb4843f3468
Parents: 66249a4
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 11:33:26 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 11:33:26 2015 -0400
----------------------------------------------------------------------
_config.yml | 3 +++
releases/6.0/index.html | 10 +++-------
2 files changed, 6 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/2c8c9aab/_config.yml
----------------------------------------------------------------------
diff --git a/_config.yml b/_config.yml
index 0238845..65aebd9 100644
--- a/_config.yml
+++ b/_config.yml
@@ -1,2 +1,5 @@
timezone: US/Eastern
markdown: kramdown
+
+gems:
+ - jekyll-redirect-from
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/2c8c9aab/releases/6.0/index.html
----------------------------------------------------------------------
diff --git a/releases/6.0/index.html b/releases/6.0/index.html
index 91afe49..136ece3 100644
--- a/releases/6.0/index.html
+++ b/releases/6.0/index.html
@@ -1,7 +1,3 @@
-<!DOCTYPE HTML>
-<html lang="en-US">
- <head>
- <meta charset="UTF-8">
- <meta http-equiv="refresh" content="1;url=http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz" />
- </head>
-</html>
+---
+redirect_from: "http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz"
+---
[03/50] incubator-joshua-site git commit: cleared up description for
language packs
Posted by mj...@apache.org.
cleared up description for language packs
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/afcbef14
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/afcbef14
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/afcbef14
Branch: refs/heads/master
Commit: afcbef14f1a8c517e8b53cf7207a976c0c6e3fcf
Parents: 43424bf
Author: Matt Post <po...@cs.jhu.edu>
Authored: Tue May 19 11:12:18 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Tue May 19 11:12:31 2015 -0400
----------------------------------------------------------------------
6.0/bundle.md | 24 +++++++++++++-----------
1 file changed, 13 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/afcbef14/6.0/bundle.md
----------------------------------------------------------------------
diff --git a/6.0/bundle.md b/6.0/bundle.md
index cc3b8a9..eb2ff5a 100644
--- a/6.0/bundle.md
+++ b/6.0/bundle.md
@@ -44,9 +44,14 @@ very large, so that it takes a long time to load and to
filtered against the tuning and testing data in the pipeline, and
these filtered models will be what is listed in the source config
files. However, when exporting a model for use as a language pack, you
-need to export the full model instead of the filtered one. The `--tm`
-line is used to accomplish this; it takes two arguments: the first
-identifies the TM's owner, and the second, the updated path.
+need to export the full model instead of the filtered one. The
+`--tm` parameter is used to accomplish this; it takes an argument
+specifying the path to the full model. Multiple `--tm` parameters can
+be passed; they will replace TMs found in the config file in order. If
+you would additionally like the large model to be
+[packed](packing.html) (which reformats it so that it can be quickly
+loaded at run time), you can use `--pack-tm` instead. You can only
+pack one TM, but typically there is only one anyway.
Here is an example invocation:
@@ -63,14 +68,11 @@ Here is an example invocation:
The copy config options tell the decoder to present just the
single-best (`-top-n 0`) translated output string that has been
heuristically capitalized (`-output-format %S`), to not append `_OOV`
-to OOVs (`-mark-oovs false`), and to use the grammar
-`/path/to/rundir/grammar.gz` as the main translation grammar. See
-[this page](decoder.html) for a longer list of decoder options.
-
-The `--pack-tm` option tells the run bundler to
-[pack the grammar](packing.html), which can take some time. Since it
-is the first of any `--[pack-]tm` arguments, it applies to the first
-TM encountered in the config file.
+to OOVs (`-mark-oovs false`), and to use the translation model
+`/path/to/rundir/grammar.gz` as the main translation model, packing it
+before placing it in the bundle. Note that these arguments to
+`--copy-config` are the default, so you could leave this off entirely.
+See [this page](decoder.html) for a longer list of decoder options.
A new directory `language-pack-YYYY-MM-DD` will be created along with
a README and a number of support files.
[44/50] incubator-joshua-site git commit: added zh-en-hiero
Posted by mj...@apache.org.
added zh-en-hiero
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/a062afdf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/a062afdf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/a062afdf
Branch: refs/heads/master
Commit: a062afdfa50b9b1bd0e94e477e2eb56dbe4c11db
Parents: 3677ad2
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed Jan 13 16:12:48 2016 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed Jan 13 16:12:48 2016 -0500
----------------------------------------------------------------------
language-packs/index.md | 11 ++++++++++-
language-packs/zh-en-hiero/index.html | 16 ++++++++++++++++
2 files changed, 26 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/a062afdf/language-packs/index.md
----------------------------------------------------------------------
diff --git a/language-packs/index.md b/language-packs/index.md
index 16c98f0..e43efe8 100644
--- a/language-packs/index.md
+++ b/language-packs/index.md
@@ -45,7 +45,7 @@ on [the installation page](../6.0/install.html).
<tr>
<td>Arabic--English</td>
<td>phrase</td>
- <td>2.4 GB</td>
+ <td>2.1 GB</td>
<td>Built from
<a href="https://catalog.ldc.upenn.edu/LDC2012T09">the LDC Arabic-Dialect/English parallel text</a>,
<a href="https://catalog.ldc.upenn.edu/LDC2007T08">the ISI Arabic--English automatically extracted parallel text</a>,
@@ -53,6 +53,15 @@ on [the installation page](../6.0/install.html).
</td>
<td><a href="ar-en-phrase/">Download</a></td>
</tr>
+ <tr>
+ <td>Chinese--English</td>
+ <td>hiero</td>
+ <td>2.4 GB</td>
+ <td>Built from about 2 million sentences of parallel news. Contains the Joshua runtime,
+ so there are no external dependencies (including Joshua).
+ </td>
+ <td><a href="zh-en-hiero/">Download</a></td>
+ </tr>
</table>
Have a request? Please email [Matt Post](http://cs.jhu.edu/~post/).
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/a062afdf/language-packs/zh-en-hiero/index.html
----------------------------------------------------------------------
diff --git a/language-packs/zh-en-hiero/index.html b/language-packs/zh-en-hiero/index.html
new file mode 100644
index 0000000..44ef883
--- /dev/null
+++ b/language-packs/zh-en-hiero/index.html
@@ -0,0 +1,16 @@
+<!DOCTYPE HTML>
+<html lang="en-US">
+ <head>
+ <meta charset="UTF-8">
+ <meta http-equiv="refresh" content="1;url=http://example.com">
+ <script type="text/javascript">
+ window.location.href = "http://cs.jhu.edu/~post/language-packs/zh-en-hiero-2016-01-13.tgz"
+ </script>
+ <title>Page Redirection</title>
+ </head>
+ <body>
+ If you are not redirected automatically, follow
+ <a href='http://cs.jhu.edu/~post/language-packs/zh-en-hiero-2016-01-13.tgz'>this link</a>
+ to download Joshua directly.
+ </body>
+</html>
[29/50] incubator-joshua-site git commit: fixed link
Posted by mj...@apache.org.
fixed link
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/676736c1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/676736c1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/676736c1
Branch: refs/heads/master
Commit: 676736c161f8215042ee3858867856ccc380412f
Parents: 7ecb805
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Jul 17 17:11:06 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Jul 17 17:11:06 2015 -0400
----------------------------------------------------------------------
6.0/install.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/676736c1/6.0/install.md
----------------------------------------------------------------------
diff --git a/6.0/install.md b/6.0/install.md
index 7b544f4..a6c9233 100644
--- a/6.0/install.md
+++ b/6.0/install.md
@@ -23,7 +23,7 @@ install the decoder. There are no external dependencies.
1. Download Joshua
- wget -q http://cs.jhu.edu/~post/joshua-v{{ site.data.joshua.release_version }}.tgz
+ wget -q http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz
1. Next, unpack it and compile everything.
[14/50] incubator-joshua-site git commit: fixed current link
Posted by mj...@apache.org.
fixed current link
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/dc45f503
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/dc45f503
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/dc45f503
Branch: refs/heads/master
Commit: dc45f5039c4f82abaa9270c459cda04eafffd9da
Parents: f956df7
Author: Matt Post <po...@cs.jhu.edu>
Authored: Tue Jun 2 09:48:50 2015 -0600
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Tue Jun 2 09:48:50 2015 -0600
----------------------------------------------------------------------
releases/current | 1 -
releases/current/index.html | 4 ++++
2 files changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/dc45f503/releases/current
----------------------------------------------------------------------
diff --git a/releases/current b/releases/current
deleted file mode 120000
index 5049538..0000000
--- a/releases/current
+++ /dev/null
@@ -1 +0,0 @@
-6.0
\ No newline at end of file
diff --git a/releases/current/index.html b/releases/current/index.html
new file mode 100644
index 0000000..17ec987
--- /dev/null
+++ b/releases/current/index.html
@@ -0,0 +1,4 @@
+---
+redirect_to:
+ - http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz
+---
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/dc45f503/releases/current/index.html
----------------------------------------------------------------------
diff --git a/releases/current/index.html b/releases/current/index.html
new file mode 100644
index 0000000..17ec987
--- /dev/null
+++ b/releases/current/index.html
@@ -0,0 +1,4 @@
+---
+redirect_to:
+ - http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz
+---
[19/50] incubator-joshua-site git commit: incremented release
Posted by mj...@apache.org.
incremented release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/5a64b327
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/5a64b327
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/5a64b327
Branch: refs/heads/master
Commit: 5a64b32794e35cdaa2047a7602af2b4ebcf6a6b7
Parents: ac9e2c9
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed Jun 17 10:10:40 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed Jun 17 10:10:40 2015 -0400
----------------------------------------------------------------------
_data/joshua.yaml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/5a64b327/_data/joshua.yaml
----------------------------------------------------------------------
diff --git a/_data/joshua.yaml b/_data/joshua.yaml
index 595f48d..63e0b75 100644
--- a/_data/joshua.yaml
+++ b/_data/joshua.yaml
@@ -1,2 +1,2 @@
-release_version: 6.0.3
-release_date: June 1, 2015
+release_version: 6.0.4
+release_date: June 15, 2015
[08/50] incubator-joshua-site git commit: playing around
Posted by mj...@apache.org.
playing around
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/d521a149
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/d521a149
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/d521a149
Branch: refs/heads/master
Commit: d521a149056a169dc24c979e8fc3feb8cce23b97
Parents: 2c8c9aa
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 11:36:19 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 11:36:19 2015 -0400
----------------------------------------------------------------------
releases/6.0/index.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/d521a149/releases/6.0/index.html
----------------------------------------------------------------------
diff --git a/releases/6.0/index.html b/releases/6.0/index.html
index 136ece3..5402eca 100644
--- a/releases/6.0/index.html
+++ b/releases/6.0/index.html
@@ -1,3 +1,3 @@
---
-redirect_from: "http://cs.jhu.edu/~post/files/joshua-v{{ site.data.joshua.release_version }}.tgz"
+redirect_from: "http://cs.jhu.edu/~post/files/joshua-v6.0.2.tgz"
---
[31/50] incubator-joshua-site git commit: added --type flag which is
now require
Posted by mj...@apache.org.
added --type flag which is now require
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/5d881092
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/5d881092
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/5d881092
Branch: refs/heads/master
Commit: 5d8810929ec335157cb0a6c9cd538b91511a8e07
Parents: 1a7aa71
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Oct 1 10:00:00 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Oct 1 10:00:00 2015 -0400
----------------------------------------------------------------------
6.0/quick-start.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/5d881092/6.0/quick-start.md
----------------------------------------------------------------------
diff --git a/6.0/quick-start.md b/6.0/quick-start.md
index 55d025a..094b615 100644
--- a/6.0/quick-start.md
+++ b/6.0/quick-start.md
@@ -31,6 +31,7 @@ First, download the dataset:
Then, train and test a model
$JOSHUA/bin/pipeline.pl --source bn --target en \
+ --type hiero \
--no-prepare --aligner berkeley \
--corpus input/bn-en/tok/training.bn-en \
--tune input/bn-en/tok/dev.bn-en \
[22/50] incubator-joshua-site git commit: Merge pull request #2 from
chrismattmann/add-mattmann-mcgibbney
Posted by mj...@apache.org.
Merge pull request #2 from chrismattmann/add-mattmann-mcgibbney
fix mattmann affil.
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/3fd16062
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/3fd16062
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/3fd16062
Branch: refs/heads/master
Commit: 3fd160628d837bce43204b02865e3ada40535c43
Parents: 1f7b84f ee0be09
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Jun 19 17:31:37 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Jun 19 17:31:37 2015 -0400
----------------------------------------------------------------------
contributors.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
[45/50] incubator-joshua-site git commit: Create index.md
Posted by mj...@apache.org.
Create index.md
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/846b5667
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/846b5667
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/846b5667
Branch: refs/heads/master
Commit: 846b5667c90acf16ededa9c1d58acfb8c4603b8f
Parents: a062afd
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Apr 1 14:40:11 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Apr 1 14:40:11 2016 -0400
----------------------------------------------------------------------
language-packs/paraphrase/index.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/846b5667/language-packs/paraphrase/index.md
----------------------------------------------------------------------
diff --git a/language-packs/paraphrase/index.md b/language-packs/paraphrase/index.md
new file mode 100644
index 0000000..684e27f
--- /dev/null
+++ b/language-packs/paraphrase/index.md
@@ -0,0 +1 @@
+Coming soon...
[06/50] incubator-joshua-site git commit: Added jekyll variables to
store version and other info
Posted by mj...@apache.org.
Added jekyll variables to store version and other info
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/66249a4f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/66249a4f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/66249a4f
Branch: refs/heads/master
Commit: 66249a4f1430bbd2251490d8cd20e5f1ee95e795
Parents: 29556b1
Author: Matt Post <po...@cs.jhu.edu>
Authored: Wed May 20 11:20:12 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Wed May 20 11:20:12 2015 -0400
----------------------------------------------------------------------
_config.yml | 38 ++------------------------------------
_data/joshua.yaml | 2 ++
_layouts/default6.html | 4 ++--
index.md | 4 ++--
releases/index.md | 4 ++--
5 files changed, 10 insertions(+), 42 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/66249a4f/_config.yml
----------------------------------------------------------------------
diff --git a/_config.yml b/_config.yml
index 2cd848e..0238845 100644
--- a/_config.yml
+++ b/_config.yml
@@ -1,36 +1,2 @@
-source: .
-plugins: ./_plugins
-url: http://joshua-decoder.org/
-
-future: true
-lsi: false
-pygments: true
-markdown: kramdown
-permalink: /:title.html
-
-#exclude: index.html,docs.html,images,indian-parallel-corpora,joshua.bib,publications,style.css,CNAME,joshua.css
-
-maruku:
- use_tex: false
- use_divs: false
- png_engine: blahtex
- png_dir: images/latex
- png_url: /images/latex
-
-rdiscount:
- extensions: []
-
-kramdown:
- auto_ids: true,
- footnote_nr: 1
- entity_output: as_char
- toc_levels: 1..6
- use_coderay: false
-
- coderay:
- coderay_wrap: div
- coderay_line_numbers: inline
- coderay_line_numbers_start: 1
- coderay_tab_width: 4
- coderay_bold_every: 10
- coderay_css: style
+timezone: US/Eastern
+markdown: kramdown
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/66249a4f/_data/joshua.yaml
----------------------------------------------------------------------
diff --git a/_data/joshua.yaml b/_data/joshua.yaml
new file mode 100644
index 0000000..23c55d6
--- /dev/null
+++ b/_data/joshua.yaml
@@ -0,0 +1,2 @@
+release_version: 6.0.2
+release_date: April 10, 2015
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/66249a4f/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index 5c981ea..63a8adf 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -51,8 +51,8 @@
</div>
<hr>
<center>
- <a href="/releases/6.0/" target="_blank"><button class="button">Download Joshua 6.0.2</button></a>
- <p>Released April 10, 2015</p>
+ <a href="/releases/current/" target="_blank"><button class="button">Download Joshua {{ site.data.joshua.release_version }}</button></a>
+ <p>Released {{ site.data.joshua.release_date }}</p>
</center>
<hr>
<!-- <div class="sidebar-module"> -->
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/66249a4f/index.md
----------------------------------------------------------------------
diff --git a/index.md b/index.md
index 6d9a9df..b9a9644 100644
--- a/index.md
+++ b/index.md
@@ -12,8 +12,8 @@ at [Johns Hopkins University](http://www.jhu.edu/).
### Latest Release
-The latest release of Joshua is version 6.0.2 (April 10,
-2015). Information about downloading and using Joshua can be found
+The latest release of Joshua is version {{ site.data.joshua.release_version }}
+({{ site.data.joshua.release_date }}). Information about downloading and using Joshua can be found
[in the Joshua documentation](/6.0/). You can [download it here](releases/6.0/) (tarball).
### Features
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/66249a4f/releases/index.md
----------------------------------------------------------------------
diff --git a/releases/index.md b/releases/index.md
index c08aebd..c44b676 100644
--- a/releases/index.md
+++ b/releases/index.md
@@ -3,8 +3,8 @@ layout: default6
title: Joshua releases
---
-The current release of Joshua is version 6.0.2, which can be [downloaded
-here](6.0/). It was released on April 10, 2015.
+The current release of Joshua is version {{ site.data.joshua.release_version }}, which can be [downloaded
+here](current/). It was released on {{ site.data.joshua.release_date }}.
Older versions are also available:
[36/50] incubator-joshua-site git commit: Merge pull request #3 from
fhieber/packing
Posted by mj...@apache.org.
Merge pull request #3 from fhieber/packing
updated packedGrammar documentation for 6.0.5 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/624ca35f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/624ca35f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/624ca35f
Branch: refs/heads/master
Commit: 624ca35ff155a7f2ef14e45e441d8d5eb4994f2e
Parents: b403216 0652f03
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Oct 22 11:58:01 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Oct 22 11:58:01 2015 -0400
----------------------------------------------------------------------
6.0/packing.md | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
[48/50] incubator-joshua-site git commit: removed dir since page not
building
Posted by mj...@apache.org.
removed dir since page not building
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/0344946b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/0344946b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/0344946b
Branch: refs/heads/master
Commit: 0344946b5853c0d0203585bbec3c267ba837db14
Parents: 4c1c9a5
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Apr 1 15:16:54 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Apr 1 15:16:54 2016 -0400
----------------------------------------------------------------------
language-packs/paraphrase/index.md | 3 ---
1 file changed, 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/0344946b/language-packs/paraphrase/index.md
----------------------------------------------------------------------
diff --git a/language-packs/paraphrase/index.md b/language-packs/paraphrase/index.md
deleted file mode 100644
index 6127296..0000000
--- a/language-packs/paraphrase/index.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# Joshua English Paraphrase Language Packs
-
-Coming soon...
[28/50] incubator-joshua-site git commit: added large lms link
Posted by mj...@apache.org.
added large lms link
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/7ecb805c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/7ecb805c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/7ecb805c
Branch: refs/heads/master
Commit: 7ecb805c1cfc8ed82a44c9d44b76747edfcfb415
Parents: 700e858
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Jun 25 23:37:38 2015 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Jun 25 23:37:38 2015 -0400
----------------------------------------------------------------------
_layouts/default6.html | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/7ecb805c/_layouts/default6.html
----------------------------------------------------------------------
diff --git a/_layouts/default6.html b/_layouts/default6.html
index 3737c63..2aaed53 100644
--- a/_layouts/default6.html
+++ b/_layouts/default6.html
@@ -87,6 +87,7 @@
<li><a href="/6.0/decoder.html">Decoder options</a></li>
<li><a href="/6.0/file-formats.html">File formats</a></li>
<li><a href="/6.0/packing.html">Packing TMs</a></li>
+ <li><a href="/6.0/large-lms.html">Building large LMs</a></li>
</ol>
</div>
[49/50] incubator-joshua-site git commit: added stub page
Posted by mj...@apache.org.
added stub page
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/f2875873
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/f2875873
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/f2875873
Branch: refs/heads/master
Commit: f287587358304c920f7b35fe96e036b4d6fcc5f9
Parents: 0344946
Author: Matt Post <po...@cs.jhu.edu>
Authored: Fri Apr 1 15:17:35 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Fri Apr 1 15:17:35 2016 -0400
----------------------------------------------------------------------
language-packs/paraphrase/index.md | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/f2875873/language-packs/paraphrase/index.md
----------------------------------------------------------------------
diff --git a/language-packs/paraphrase/index.md b/language-packs/paraphrase/index.md
new file mode 100644
index 0000000..6127296
--- /dev/null
+++ b/language-packs/paraphrase/index.md
@@ -0,0 +1,3 @@
+# Joshua English Paraphrase Language Packs
+
+Coming soon...
[37/50] incubator-joshua-site git commit: 6.0.5 release
Posted by mj...@apache.org.
6.0.5 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/549fb142
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/549fb142
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/549fb142
Branch: refs/heads/master
Commit: 549fb142da49aa740f022d17e9ee97e90a143bcb
Parents: 624ca35
Author: Matt Post <po...@cs.jhu.edu>
Authored: Thu Nov 5 07:02:32 2015 -0500
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Thu Nov 5 07:02:32 2015 -0500
----------------------------------------------------------------------
_data/joshua.yaml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/549fb142/_data/joshua.yaml
----------------------------------------------------------------------
diff --git a/_data/joshua.yaml b/_data/joshua.yaml
index 63e0b75..1358510 100644
--- a/_data/joshua.yaml
+++ b/_data/joshua.yaml
@@ -1,2 +1,2 @@
-release_version: 6.0.4
-release_date: June 15, 2015
+release_version: 6.0.5
+release_date: November 5, 2015