You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@joshua.apache.org by mj...@apache.org on 2016/08/30 13:03:54 UTC
[1/4] incubator-joshua git commit: Fix Travis-CI by adding a step to
install ant on OSX
Repository: incubator-joshua
Updated Branches:
refs/heads/7 0b543276e -> bb3c3004e
Fix Travis-CI by adding a step to install ant on OSX
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua/commit/0b447595
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua/tree/0b447595
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua/diff/0b447595
Branch: refs/heads/7
Commit: 0b447595ae51ab3d2956d60748962a092e192286
Parents: 2d106df
Author: Kellen Sunderland <ke...@amazon.com>
Authored: Tue Aug 30 10:38:35 2016 +0200
Committer: Kellen Sunderland <ke...@amazon.com>
Committed: Tue Aug 30 10:38:35 2016 +0200
----------------------------------------------------------------------
.travis.yml | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua/blob/0b447595/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index 429042a..8f78ca2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,6 +20,8 @@ matrix:
- os: osx
osx_image: xcode8
before_install:
+ - brew update
+ - brew install ant
- export JOSHUA=`pwd`
- export JAVA_HOME=`/usr/libexec/java_home -v 1.8`
- ./download-deps.sh
[3/4] incubator-joshua git commit: Merge branch 'kenlm_pool_fix' of
github.com:KellenSunderland/incubator-joshua
Posted by mj...@apache.org.
Merge branch 'kenlm_pool_fix' of github.com:KellenSunderland/incubator-joshua
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua/commit/9385cf7a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua/tree/9385cf7a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua/diff/9385cf7a
Branch: refs/heads/7
Commit: 9385cf7abccceab2bc86f0f5ddface1a3f5e0d8e
Parents: 0b44759 8e7e777
Author: Matt Post <po...@cs.jhu.edu>
Authored: Tue Aug 30 09:03:33 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Tue Aug 30 09:03:33 2016 -0400
----------------------------------------------------------------------
jni/kenlm_wrap.cc | 33 +++++++++++++++++++++++----------
1 file changed, 23 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
[2/4] incubator-joshua git commit: Fixed crashing when using Trie
based KenLM models with State Minimization
Posted by mj...@apache.org.
Fixed crashing when using Trie based KenLM models with State Minimization
There are two separate issues that are causing very rare crashes when using KenLM with Joshua. The only configuration under which we've seen crashing, is when using more than one Trie-based state-minimizing model. One of these issues is addressed in this patch, another will have to be contributed to the KenLM repo.
This first patch deals with hash collisions in the mapping between a given KenLM state, and the stored memory address of the state in KenLM\u2019s memory pool. Previously these collisions were ignored. The problem with ignoring collisions here is that the two models can have different Vocabulary sizes. This means if you collide and take the state of a model with a larger vocabulary you could potentially have a unigram word id that is out of range. When searched for this will cause a crash.
The second issue (to be patched in KenLM) is that the equality operator in state.hh isn't comparing enough values to properly differentiate between states during a collision.
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua/commit/8e7e7776
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua/tree/8e7e7776
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua/diff/8e7e7776
Branch: refs/heads/7
Commit: 8e7e7776b8b0fa862f08750b1dddd93f9e5f3487
Parents: 2d106df
Author: Kellen Sunderland <ke...@amazon.com>
Authored: Tue Aug 30 10:52:07 2016 +0200
Committer: Kellen Sunderland <ke...@amazon.com>
Committed: Tue Aug 30 11:04:06 2016 +0200
----------------------------------------------------------------------
jni/kenlm_wrap.cc | 33 +++++++++++++++++++++++----------
1 file changed, 23 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-joshua/blob/8e7e7776/jni/kenlm_wrap.cc
----------------------------------------------------------------------
diff --git a/jni/kenlm_wrap.cc b/jni/kenlm_wrap.cc
index 97c52f1..11d9c28 100644
--- a/jni/kenlm_wrap.cc
+++ b/jni/kenlm_wrap.cc
@@ -45,13 +45,14 @@ template<> struct StaticCheck<true> {
typedef StaticCheck<sizeof(jint) == sizeof(lm::WordIndex)>::StaticAssertionPassed FloatSize;
-typedef std::unordered_map<uint64_t, lm::ngram::ChartState*> PoolHash;
+typedef std::unordered_multimap<uint64_t, lm::ngram::ChartState*> PoolHash;
/**
- * A Chart bundles together a hash_map that maps ChartState signatures to a single object
- * instantiated using a pool. This allows duplicate states to avoid allocating separate
- * state objects at multiple places throughout a sentence, and also allows state to be
- * shared across KenLMs for the same sentence.
+ * A Chart bundles together a unordered_multimap that maps ChartState signatures to a single
+ * object instantiated using a pool. This allows duplicate states to avoid allocating separate
+ * state objects at multiple places throughout a sentence, and also allows state to be shared
+ * across KenLMs for the same sentence. Multimap is used to avoid hash collisions which can
+ * return incorrect results, and cause out-of-bounds lookups when multiple KenLMs are in use.
*/
struct Chart {
// A cache for allocated chart objects
@@ -71,15 +72,27 @@ struct Chart {
}
lm::ngram::ChartState* put(const lm::ngram::ChartState& state) {
+ lm::ngram::ChartState* state_ptr = nullptr;
uint64_t hashValue = lm::ngram::hash_value(state);
+ auto state_it = poolHash->find(hashValue);
- if (poolHash->find(hashValue) == poolHash->end()) {
- lm::ngram::ChartState* pointer = (lm::ngram::ChartState *)pool->Allocate(sizeof(lm::ngram::ChartState));
- *pointer = state;
- (*poolHash)[hashValue] = pointer;
+ // Try to retrieve a matching ChartState pointer from our Pool
+ while(state_it != poolHash->end()) {
+ if (state == *(state_it->second)) {
+ state_ptr = state_it->second;
+ break;
+ }
+ state_it++;
+ }
+
+ // Unable to find this ChartState in our pool, allocate new space for it
+ if (!state_ptr) {
+ state_ptr = (lm::ngram::ChartState *) pool->Allocate(sizeof(lm::ngram::ChartState));
+ *state_ptr = state;
+ (*poolHash).insert({hashValue, state_ptr});
}
- return (*poolHash)[hashValue];
+ return state_ptr;
}
};
[4/4] incubator-joshua git commit: Merge branch 'master' into 7
Posted by mj...@apache.org.
Merge branch 'master' into 7
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua/commit/bb3c3004
Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua/tree/bb3c3004
Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua/diff/bb3c3004
Branch: refs/heads/7
Commit: bb3c3004e30a86628c3725bcd90ae4e93d9c424d
Parents: 0b54327 9385cf7
Author: Matt Post <po...@cs.jhu.edu>
Authored: Tue Aug 30 09:03:44 2016 -0400
Committer: Matt Post <po...@cs.jhu.edu>
Committed: Tue Aug 30 09:03:44 2016 -0400
----------------------------------------------------------------------
.travis.yml | 2 ++
jni/kenlm_wrap.cc | 33 +++++++++++++++++++++++----------
2 files changed, 25 insertions(+), 10 deletions(-)
----------------------------------------------------------------------