Re: Individual scoring at SMTP time

[Justin Mason <jm...@jmason.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


BTW, if SpamAssassin patches were in the picture ;), the ideal would
be as follows:

- - if the mail has only one recipient:

  - scan it as the user/recipient as we currently do.

- - else:

  - scan the mail once, upfront, using system-wide settings

  - skip all rules that use user configuration or databases (bayes, AWL
    etc.)

  - skip all rules that are score == 0 in the system-wide settings

  - store the scan results in a database somewhere

- - then, for each recipient "user"/address:

  - load those scan results from the database

  - if the user has changed certain settings that affect rule behaviour
    (like trusted_networks, internal_networks), then:
    
    - throw out the cached scan results and perform a total rescan for
      that user.  (or, future optimisation: this could be optimised if we
      knew what rules are modified by trusted_networks; that could be
      indicated through a tflag.)

  - else:

    - foreach rule that fired in that scan:

      - add its score as per the user's configuration to the "hits"
        accumulator (by using the user's score settings, that allows users
        to disable rules by setting their score == 0.)

    - run the "user configuration" rules like Bayes/AWL

  - finally, tot up the score against the user's required_hits, and report
    appropriately


So in other words split the scan into a two-stage process, with a single
"heavy" upfront run and several "light" per-user runs.

BTW I understand Amavisd does some of this, but to really complete the
picture, SpamAssassin patches would be required in my opinion.

And -- finally -- it's VERY important to note that that could all be
entirely pointless, at the end of it.  We have found in the past that
adding that much additional "smarts" to the core scanning loops sometimes
turns out to be slower than just doing the donkey work of re-scanning for
each user!

- --j.


Matthew.van.Eerde@hbinc.com writes:
> Matthew.van.Eerde wrote:
> > At RCPT TO: time, check the pref of the user.
> > 
> > The pref of the FIRST ACCEPTED USER should be used to scan the mail.
> > 
> > The SECOND AND FURTHER legitimate users should be accepted or
> > tempfailed according to this recipe: 
> > If this user has the same pref as the first accepted user, accept.
> > If this user has a different pref, tempfail.  (Optionally build a
> > greylist entry.) 
> 
> After some further thought... what if an email is addressed to 100 Doormat users and a single Draconian user?  A sane email sender will batch the RCPT's 50 at a time... but if the Draconian user is second in each batch (including retries) it might be an awfully long time before he's accepted...
> 
> So lets make the greylist entry mandatory.
>  
> > After the DATA is received, all users should be in agreement as they
> > share the same spam preferences. 
> 
> Also, let's scan the email five times, under all five different prefs, and remember the results in the greylist as well.
>  
> > If the message is legitimate, the sender will retry all the
> > tempfailed RCPT's. 
> > The fact that they retry may be good enough to accept without
> > spam-scanning at all.  Or it might just be worth a few negative spam
> > points.  
> 
> On the retry, we already know the spam-score results under all five different prefs.  So we can accept and reject at RCPT TO: time for all the retries, plus all the new email addresses from the next batch of RCPTs.
> 
> Consider:
> 
> Email from a MLM vendor sender@example.com
> A particular organization example.org has 100 subscribees to this MLM email:
> user1@example.org
> ...
> user100@example.org
> 
> The MLM vendor batches their RCPTs in 20 per SMTP transaction.
> 
> The first attempt to deliver the mail goes like this:
> -> HELO mail.example.com
> <- OK
> -> MAIL FROM: sender@example.com
> <- OK
> -> RCPT TO: user1@example.org
> <- OK
> -> RCPT TO: user2@example.org
> <- 4xx retry later, different spam pref
> -> RCPT TO: user3@example.org
> <- OK
> ...
> -> RCPT TO: user20@example.org
> <- 4xx retry later, different spam pref
> -> DATA ...
> 
> At this point the data is scanned.  let's suppose it is acceptable to all but Draconian users.  user1, user3 and others are not Draconian.
> <- OK
> 
> Later, the MLM vendor connects again.
> -> HELO mail.example.com
> <- OK
> -> MAIL FROM: sender@example.com
> <- OK
> -> RCPT TO: user2@example.org
> <- 5xx user2 has Draconian spam policies and doesn't want that
> -> RCPT TO: user20@example.org
> <- OK user20 has Doormat spam policies and will be happy to accept that
> -> RCPT TO: user21@example.org
> <- OK user21 has a medium-level spam pref but I know this email is good under that
> -> RCPT TO: user22@example.org
> <- 5xx user22 has Draconian spam policies and doesn't want that...
> ...
> -> DATA ...
> 
> No need to re-spam scan here, just use the known results
> 
> <- OK
> 
> This assumes that the triplet of (connecting IP, HELO, MAIL FROM) uniquely identifies spam-classes of email for all users, at least within a greylist window.
> 
> -- 
> Matthew.van.Eerde (at) hbinc.com               805.964.4554 x902
> Hispanic Business Inc./HireDiversity.com       Software Engineer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFDUAS4MJF5cimLx9ARAshLAKCoEJ6FwWhDP/gHIFlJvAkwFFq3nACggCq2
e8mynz8lRoLg2SaXNPxdpL0=
=Z2G4
-----END PGP SIGNATURE-----