[Bug 6963] New: Anybody cares for a saved millisecond or two in computing bayes probabilities for tokens?

[bu...@bugzilla.spamassassin.org]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6963

            Bug ID: 6963
           Summary: Anybody cares for a saved millisecond or two in
                    computing bayes probabilities for tokens?
           Product: Spamassassin
           Version: SVN Trunk (Latest Devel Version)
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: Libraries
          Assignee: dev@spamassassin.apache.org
          Reporter: Mark.Martinec@ijs.si

Wondering where 'b_comp_prob' reported timing entry spends its time
(computing bayes probabilities for tokens), I played a bit with the
beautiful NYTProf perl profiler and shuffled some Bayes code while
keeping its functionality unchanged.

The basic idea is to compute a probability for all tokens in one
go, instead of calling the _compute_prob_for_token() for each token.
This allows for factoring out unchanging sections from the loop.
So instead of:
  Plugin::Bayes::_compute_prob_for_token
we now call:
  Plugin::Bayes::_compute_prob_for_all_tokens
(and the _compute_prob_for_token() is now just a wrapper).

Savings are less than I hoped, about 1.2 ms for a typical larger
message with one or two hundred tokens, and a barely noticeable
speedup for messages with only a few tokens. When dumping tokens
(sa-learn --dump) the saving is about 6 seconds (out of one minute)
with my current redis database.

Still, the work is done now, I wonder whether we like it folded in,
or not.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 6963] Anybody cares for a saved millisecond or two in computing bayes probabilities for tokens?

[bu...@bugzilla.spamassassin.org]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6963

Mark Martinec <Ma...@ijs.si> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|Undefined                   |3.4.0

--- Comment #3 from Mark Martinec <Ma...@ijs.si> ---
trunk:
  Bug 6963: Anybody cares for a saved millisecond or two
  in computing bayes probabilities for tokens?
Sending lib/Mail/SpamAssassin/Plugin/Bayes.pm
Committed revision 1511229.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 6963] Anybody cares for a saved millisecond or two in computing bayes probabilities for tokens?

[bu...@bugzilla.spamassassin.org]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6963

Karsten Bräckelmann <gu...@rudersport.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #4 from Karsten Bräckelmann <gu...@rudersport.de> ---
(In reply to Mark Martinec from comment #3)
> Committed revision 1511229.

RESOLVED FIXED

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 6963] Anybody cares for a saved millisecond or two in computing bayes probabilities for tokens?

[bu...@bugzilla.spamassassin.org]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6963

--- Comment #1 from Mark Martinec <Ma...@ijs.si> ---
Created attachment 5163
  --> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5163&action=edit
The optimization patch

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 6963] Anybody cares for a saved millisecond or two in computing bayes probabilities for tokens?

[bu...@bugzilla.spamassassin.org]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6963

--- Comment #2 from Mark Martinec <Ma...@ijs.si> ---
Created attachment 5164
  --> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5164&action=edit
The scattergram showing the drop in timing

The attached diagram shows elapsed times for computing bayes probabilities
for all messages passing through our server today during a day.
The x-axis is a time of the day in hours (morning to evening),
and the y-axis is elapsed time in milliseconds for each message,
as reported in the 'b_comp_prob' timing log entry.

At around 14:20 the original code was replaced by the optimized code,
which explains the drop in timing. The scattergram is bi-modal,
messages with just few tokens found in a bayes database are almost
unaffected by the change, just a tiny drop is perceived.
The cluster of typical larger messages show about 1.2 ms of
elapsed time reduction.

-- 
You are receiving this mail because:
You are the assignee for the bug.