plugins and parrallelization

["Eric A. Hall" <eh...@ehsco.com>]

It seems that the plugin architecture only allows a single pass/fail
result, so if you want to have multiple tests with different shades of
results, you have to call the plugin multiple times. Is that right?

Over the weekend I banged together a preliminary ldapBlacklist.pm plugin
which lets the master process query an ldap server for whitelist or
blacklist flags associated with the connecting SMTP client's reverse DNS,
the HELO identifer, the mail-from address, the From address, and so
forth... The problem is that each of these tests have to do a fair amount
of processing with some significant serialization (ie, DNS lookup for SRV
RRs, DNS lookup for ldap server, connect->bind->query the server, as well
as the rest of the background code. Using the pass/fail model as a
front-end to this system, each test basically has to be its own rule, and
each rule has to call its own eval() in order for each rule to use its
defined weighting (eg, -50 for whitelisted, +50 for blacklisted, on a
per-test basisc. But in that model, the core LDAP stuff has to be run ~six
times to process ~six tests, and that's a significant serialization
penalty in sum, just to find out if one of the sending domains is listed
as blacklisted or whitelisted in a local LDAP server. It's so bad that I'm
not sure it's feasible to do this.

What are the thoughts?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Re: plugins and (more)

[Michael Parker <pa...@pobox.com>]

On Tue, Mar 15, 2005 at 02:20:48AM -0500, Eric A. Hall wrote:
>  - is there a way to force a plugin to load last? like, if I want SPF
>    and all of the other validation stuff to get called first, but not
>    to rely on it (it may not be installed), is there a way to force
>    the plugin to get called last (presumably this is done by numbering
>    the ldapBlacklist.cf to something like 99_ldap_blacklist.cf, but
>    maybe there's a better way)?

Load or run?

If it's run, then check out the priority tflag.

Michael

Re: plugins and (more)

["Eric A. Hall" <eh...@ehsco.com>]

Eric A. Hall wrote:

> Over the weekend I banged together a preliminary ldapBlacklist.pm plugin
> which lets the master process query an ldap server for whitelist or
> blacklist flags associated with the connecting SMTP client's reverse DNS,
> the HELO identifer, the mail-from address, the From address, and so
> forth... The problem is that each of these tests have to do a fair amount

I got this working more fully, including with the persistency stuff 
(thanks again). Couple of other things I'm looking for help on:

  - is there an internal means to determine the local domain name? I'm
    having trouble with Sys::Hostname::Long on a couple of systems and
    would rather use something internal anyway since it's sure to work
    everywhere that SA itself works.

  - is there a way to force a plugin to load last? like, if I want SPF
    and all of the other validation stuff to get called first, but not
    to rely on it (it may not be installed), is there a way to force
    the plugin to get called last (presumably this is done by numbering
    the ldapBlacklist.cf to something like 99_ldap_blacklist.cf, but
    maybe there's a better way)?

Thanks

-- 
Eric A. Hall                                      http://www.ehsco.com/
Internet Core Protocols        http://www.oreilly.com/catalog/coreprot/

Re: plugins and parrallelization

["Eric A. Hall" <eh...@ehsco.com>]

On 3/14/2005 12:32 PM, Daryl C. W. O'Shea wrote:

> There's no need to do your lookups more than once.  Save the results the 
> first time you do the lookups/processing.
> 
> See the SPF.pm plugin for a good example.

Oh right, it's persistent. Duh.

Thanks


-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Re: plugins and parrallelization

["Daryl C. W. O'Shea" <sp...@dostech.ca>]

Eric A. Hall wrote:
> It seems that the plugin architecture only allows a single pass/fail
> result, so if you want to have multiple tests with different shades of
> results, you have to call the plugin multiple times. Is that right?
> 
> Over the weekend I banged together a preliminary ldapBlacklist.pm plugin
> which lets the master process query an ldap server for whitelist or
> blacklist flags associated with the connecting SMTP client's reverse DNS,
> the HELO identifer, the mail-from address, the From address, and so
> forth... The problem is that each of these tests have to do a fair amount
> of processing with some significant serialization (ie, DNS lookup for SRV
> RRs, DNS lookup for ldap server, connect->bind->query the server, as well
> as the rest of the background code. Using the pass/fail model as a
> front-end to this system, each test basically has to be its own rule, and
> each rule has to call its own eval() in order for each rule to use its
> defined weighting (eg, -50 for whitelisted, +50 for blacklisted, on a
> per-test basisc. But in that model, the core LDAP stuff has to be run ~six
> times to process ~six tests, and that's a significant serialization
> penalty in sum, just to find out if one of the sending domains is listed
> as blacklisted or whitelisted in a local LDAP server. It's so bad that I'm
> not sure it's feasible to do this.
> 
> What are the thoughts?

There's no need to do your lookups more than once.  Save the results the 
first time you do the lookups/processing.

See the SPF.pm plugin for a good example.


Daryl