Optimization of AbstractLogisticRegression.regularize?

[Frank Scholten <fr...@frankscholten.nl>]

Hi all,

I profiled the AbstractLogisticRegression.regularize() method with
JVisualVM and I could reduce the CPU time by 10% by a doing the following:

* Using setQuick() instead of set()
* Switching the isSealed() and updateSteps == null check
* Computing lambda * learningRate outside of the loop
* Iterating over the categories first and over elements second

I like to experiment a bit more on performance tuning and looking at
HotSpot and such because I am not sure why certain changes made the
improvements. Does anyone know how to go from here, in terms of which tools
to use and what kind of approach to take?

Also, is this optimization even worth it or perhaps there are other areas
more important in Mahout to optimize? I just ran VisualVM again and now it
isn't at the top of profiler anymore, now it is
FeatureVectorEncoder.bytesForString()

Cheers,

Frank

Re: Optimization of AbstractLogisticRegression.regularize?

[Suneel Marthi <su...@yahoo.com>]

On Monday, February 3, 2014 7:08 PM, Ted Dunning <te...@gmail.com> wrote:
 
Optimization of this kind is definitely worthwhile, but 10% improvements
probably are a bit small.  Your difficulty in reproducing the results
indicates how ephemeral small improvements can be.  If you find a 2x
difference, however, we should definitely talk.

It isn't too surprising that inverting loops might make a difference since
it makes a big difference in matrix multiplication as well.  I would worry
a bit about iterating over categories first because that changes the heart
of the algorithm.  If setQuick by itself made a big difference, I might
consider doing it, but I would also expect set to get inlined before long
and optimized into a simple setQuick call.

One of the biggest places to make a serious dent would be in the text
processing that is done in a few places.  I have seen gains of 10x over
naive code that is simply blasting through text or CSV.  If you come up
with a clean and easy to use DFA implementation of that, then it would make
a significant difference.

>>> +1 to using DFA as dictionary types, that should speed up seq2sparse significantly.

Beyond that, math library speedups are always welcome.  For instance,
having an extension that offers JBLAS matrices and vectors would be cool
for some applications.




On Mon, Feb 3, 2014 at 12:53 PM, Frank Scholten <fr...@frankscholten.nl>wrote:

> Hi all,
>
> I profiled the AbstractLogisticRegression.regularize() method with
> JVisualVM and I could reduce the CPU time by 10% by a doing the following:
>
> * Using setQuick() instead of set()
> * Switching the isSealed() and updateSteps == null check
> * Computing lambda * learningRate outside of the loop
> * Iterating over the categories first and over elements second
>
> I like to experiment a bit more on performance tuning and looking at
> HotSpot and such because I am not sure why certain changes made the
> improvements. Does anyone know how to go from here, in terms of which tools
> to use and what kind of approach to take?
>
> Also, is this optimization even worth it or perhaps there are other areas
> more important in Mahout to optimize? I just ran VisualVM again and now it
> isn't at the top of profiler anymore, now it is
> FeatureVectorEncoder.bytesForString()
>
> Cheers,
>
> Frank
>

Re: Optimization of AbstractLogisticRegression.regularize?

[Ted Dunning <te...@gmail.com>]

Optimization of this kind is definitely worthwhile, but 10% improvements
probably are a bit small.  Your difficulty in reproducing the results
indicates how ephemeral small improvements can be.  If you find a 2x
difference, however, we should definitely talk.

It isn't too surprising that inverting loops might make a difference since
it makes a big difference in matrix multiplication as well.  I would worry
a bit about iterating over categories first because that changes the heart
of the algorithm.  If setQuick by itself made a big difference, I might
consider doing it, but I would also expect set to get inlined before long
and optimized into a simple setQuick call.

One of the biggest places to make a serious dent would be in the text
processing that is done in a few places.  I have seen gains of 10x over
naive code that is simply blasting through text or CSV.  If you come up
with a clean and easy to use DFA implementation of that, then it would make
a significant difference.

Beyond that, math library speedups are always welcome.  For instance,
having an extension that offers JBLAS matrices and vectors would be cool
for some applications.



On Mon, Feb 3, 2014 at 12:53 PM, Frank Scholten <fr...@frankscholten.nl>wrote:

> Hi all,
>
> I profiled the AbstractLogisticRegression.regularize() method with
> JVisualVM and I could reduce the CPU time by 10% by a doing the following:
>
> * Using setQuick() instead of set()
> * Switching the isSealed() and updateSteps == null check
> * Computing lambda * learningRate outside of the loop
> * Iterating over the categories first and over elements second
>
> I like to experiment a bit more on performance tuning and looking at
> HotSpot and such because I am not sure why certain changes made the
> improvements. Does anyone know how to go from here, in terms of which tools
> to use and what kind of approach to take?
>
> Also, is this optimization even worth it or perhaps there are other areas
> more important in Mahout to optimize? I just ran VisualVM again and now it
> isn't at the top of profiler anymore, now it is
> FeatureVectorEncoder.bytesForString()
>
> Cheers,
>
> Frank
>