Re: Joshua Model Input Format(s) and LM Loading

[kellen sunderland <ke...@gmail.com>]

I've done fair amount of measurement and profiling in this area over the
last year, so I can offer a little bit of advice as well.

First of all make sure you're not just using a Java profiler if you're
looking at a model with KenLM.  The perf impact of language model calls
will be under-reported.  If you want to optimized LM calls using a Java
profiler I'd recommend using a Berkley model and measuring the minimization
of  lm calls there.  If you can reduce the amount of work (lm calls) to
Berkley, the speed improvements should carry over to KenLM as well.  (Of
course providing the optimizations are in the Joshua code).

If you want to optimize KenLM models you're better off using some
combination of a native profiler and a JVM profiler.  Even then I'd
consider the impact of the calls to the LM as under-reported.  This is
because making frequent JNI calls amplifies the amount of work required in
GC (I think we've discussed that before).

Another idea would be to call KenLM with an RPC framework (like gRPC).
It'll likely be slower, but then you could measure the Java process with a
Java Profiler (without ill-effects on GC).  You could also then
independently measure the KenLM process with a native profiler.  This might
give you a fairly accurate view of what to optimize.

-Kellen


On Wed, Oct 26, 2016 at 9:34 AM, lewis john mcgibbney <le...@apache.org>
wrote:

> I hear ye loud and clear Matt :) Thank you for the response.
>
> On Wed, Oct 26, 2016 at 12:30 AM, <
> dev-digest-help@joshua.incubator.apache.org> wrote:
>
> >
> > From: Matt Post <po...@cs.jhu.edu>
> > To: dev@joshua.incubator.apache.org
> > Cc:
> > Date: Tue, 25 Oct 2016 08:49:19 -0400
> > Subject: Re: Joshua Model Input Format(s) and LM Loading
> > Hi Lewis,
> >
> > Joshua supports two language model representation packages: KenLM [0] and
> > BerkeleyLM [1]. These were both developed at about the same time, and
> > represented huge gains in doing this task efficiently, over what had
> > previously been the standard approach (SRILM). Ken Heafield (who has
> > contributed a lot to Joshua) went on to contribute a lot of other
> > improvements to language model representation, decoder integration, and
> > also the actual construction of language models and their efficient
> > interpolation. His goal for a while was to make SRILM completely
> > unnecessary, and I think he succeeded.
> >
> > BerkeleyLM was more of a one-off project. It is slower than KenLM and
> > hasn't been touched in years. If you want to understand, your efforts are
> > probably best spent looking into KenLM papers. But it's also worth noting
> > that Ken is a crack C++ programmer who has spent years hacking away on
> > these problems, and your chances of finding any further efficiencies
> there
> > are probably quite limited unless you have a lot of background in the
> area.
> > But even if you did, I would recommend you not spend your time that way
> — I
> > basically consider the LM representation problem to have been solved by
> > KenLM. That's not to say that there are some improvements to be had on
> the
> > Joshua / JNI bridge, but even there, there are probably better things to
> do.
> >
> > matt
> >
> > [0] KenLM: Faster and Smaller Language Model Queries
> > http://www.kheafield.com/professional/avenue/kenlm.pdf
> >
> > [1] Faster and Smaller N-Gram Language Models
> > http://nlp.cs.berkeley.edu/pubs/Pauls-Klein_2011_LM_paper.pdf
> >
> >
>