SYMBOLS with test description

[Pedro Padron <pp...@w3p.com.br>]

Hi,

I was messing around with the spamd/spamc protocol, and I saw that the
SYMBOLS method only return test names.
However, it would be nice to be able to get also the individual test score
and it's description, what do you think?
I understand that modifying the output of SYMBOLS would break the clients,
so another method would have to be created, something like
SYMBOLS_DESCRIPTION.
It could return the same response headers as SYMBOLS, but the response
content could be CSV list like:

-0.001;NO_RELAYS;Informational: message was not relayed via SMTP
1000;GTUBE;Generic Test for Unsolicited Bulk Email

If there is interest in this, I can provide a patch.

Thanks!

-- 
Pedro Padron
Tecnologia & Desenvolvimento | W3P Projetos Web
+55 11 7518 6664
+55 11 2609 1989
http://www.w3p.com.br/

Re: SYMBOLS with test description

["Kevin A. McGrail" <KM...@PCCC.com>]

>> I was going to say the same thing.
> Including the caffeine disclaimer? ;)
OK, not *exactly* the same thing but as I use spamd/c with the REPORT 
option quite a bit and have found/fixed bugs in it before, my reaction 
was technically the same as yours including your more detailed 
follow-up.  Right down to writing a new tag instead of modifying Spamd/c.

Regards,
KAM

Re: PHP API to SpamAssassin spamd Protocol

[Karsten Bräckelmann <gu...@rudersport.de>]

On Tue, 2010-10-19 at 18:23 -0200, Pedro Padron wrote:

> I will leave the client library as is, and extend it in my application
> to parse the report the way I need. However, I will still need
> different users (for my app), because I will also work with rule
> descriptions in different languages.

Right, moving such parsing (if needed) up in the stack does the trick,
too. My main point was the client's usability with a low barrier and as
little requirements as absolutely needed. If it does not (necessarily)
require any modifications to the server (which includes a need for a
dedicated user), it will be immediately useful to a broader user base.

Regarding language support -- frankly, I haven't checked at all in a
long time and don't care much personally, but I believe the existing
translations to be lagging behind. Putting it mildly.


> Good idea, I'll work on this today =) I'll also add support for
> multiple hosts and zlib compression.
> 
> Thanks again for the feedback!

You're welcome.

In particular the size limit is important, since it prevents huge
messages (both ham and spam) from consuming too many resources. SA
itself, as well as spamd, does not have any size constraint and happily
will try no matter what to grab the resources necessary to complete the
scan.


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: PHP API to SpamAssassin spamd Protocol (was: Re: SYMBOLS with test description)

[Pedro Padron <pp...@w3p.com.br>]

2010/10/19 Karsten Bräckelmann <gu...@rudersport.de>

> I'm, not a PHP guy myself (let's just leave it at that ;), but that's
> quite nice -- a SA spamd client for PHP. :)
>

Don't worry, if you want to tell us how you feel about PHP, I will not be
offended =)

If the client can parse the default _SUMMARY_ template, it will be
> usable out-of-the-box without any SA code modification or the need for a
> custom plugin.



Moreover, if you make your client smart enough to ignore the leading
> part prior to the _SUMMARY_, you'd also get rid of the requirement of a
> dedicated user and specific preferences.
>

I will leave the client library as is, and extend it in my application to
parse the report the way I need. However, I will still need different users
(for my app), because I will also work with rule descriptions in different
languages.

Since spamc was your inspiration, you might want to consider inheriting
> some of its behavior and options. In particular, the hostname and port
> as set in your $params should be default anyway.
>
> Another useful and likely easy to implement option would be the size
> limit. A message exceeding the threshold would not be passed to spamd.
>

Good idea, I'll work on this today =) I'll also add support for multiple
hosts and zlib compression.

Thanks again for the feedback!

-- 
Pedro Padron
Tecnologia & Desenvolvimento | W3P Projetos Web
+55 11 7518 6664
+55 11 2609 1989
http://www.w3p.com.br/

PHP API to SpamAssassin spamd Protocol (was: Re: SYMBOLS with test description)

[Karsten Bräckelmann <gu...@rudersport.de>]

On Tue, 2010-10-19 at 16:12 -0200, Pedro Padron wrote:
> Yep, based on it I'm writing a PHP client [1] for the spamd protocol,
> which is why I'm interested in this =)

I'm, not a PHP guy myself (let's just leave it at that ;), but that's
quite nice -- a SA spamd client for PHP. :)


Some feedback, after reading your blog post [1].

If the client can parse the default _SUMMARY_ template, it will be
usable out-of-the-box without any SA code modification or the need for a
custom plugin.

Moreover, if you make your client smart enough to ignore the leading
part prior to the _SUMMARY_, you'd also get rid of the requirement of a
dedicated user and specific preferences.


Since spamc was your inspiration, you might want to consider inheriting
some of its behavior and options. In particular, the hostname and port
as set in your $params should be default anyway.

Another useful and likely easy to implement option would be the size
limit. A message exceeding the threshold would not be passed to spamd.


> [1] http://ppadron.blog.br/2010/05/04/php-api-to-spamassassin-spamd-protocol/

-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: SYMBOLS with test description

[Karsten Bräckelmann <gu...@rudersport.de>]

On Tue, 2010-10-19 at 16:12 -0200, Pedro Padron wrote:
> 2010/10/19 Karsten Bräckelmann

> Yep, based on it I'm writing a PHP client [1] for the spamd protocol,
> which is why I'm interested in this =)
>  
> > Oh, and if you really want a CSV separated output, it should be possibly
> > to write a plugin that provides a custom Template Tag to be used instead
> > of the default _SUMMARY_, without hacking the spamd source itself.
> 
> I have already tried to use only _SUMMARY_, but I was thinking that I
> should not mess with the report template because it should provide a
> human readable explanation, since it is used in PROCESS to return a
> modified version of the message.

Unless report_safe is set to 0. Frankly, IMHO both the rewrite_header
and report_safe 1 (default) really are only useful with broken clients
who cannot filter based on arbitrary headers. I suggest to not use
either of them if using IMAP and server side filtering, or an MUA with
half a brain.

Encapsulating the original message makes user review of identified but
possibly FP a real pain in the ass.


> However... I just realized that I can set different report templates
> per user... This should do the trick, along with a custom template
> tag.

Yep, the report option is a user preference.

However, rather than diving into SA plugin internals and writing Perl
code, it might be easier for you to just have your PHP client parse the
default _SUMMARY_ template output.


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: SYMBOLS with test description

[Pedro Padron <pp...@w3p.com.br>]

2010/10/19 Karsten Bräckelmann <gu...@rudersport.de>

>
> Since these are actually described right next to each other -- Pedro,
> you're looking into the spamd protocol, so I assume you did find the
> spamd/PROTOCOLS file in the source?


Yep, based on it I'm writing a PHP client [1] for the spamd protocol, which
is why I'm interested in this =)


> Oh, and if you really want a CSV separated output, it should be possibly
> to write a plugin that provides a custom Template Tag to be used instead
> of the default _SUMMARY_, without hacking the spamd source itself.


I have already tried to use only _SUMMARY_, but I was thinking that I should
not mess with the report template because it should provide a human readable
explanation, since it is used in PROCESS to return a modified version of the
message. However... I just realized that I can set different report
templates per user... This should do the trick, along with a custom template
tag.

Thanks Karsten and Kevin.

[1]
http://ppadron.blog.br/2010/05/04/php-api-to-spamassassin-spamd-protocol/

-- 
Pedro Padron
Tecnologia & Desenvolvimento | W3P Projetos Web
+55 11 7518 6664
+55 11 2609 1989
http://www.w3p.com.br/

Re: SYMBOLS with test description

[Karsten Bräckelmann <gu...@rudersport.de>]

On Tue, 2010-10-19 at 12:07 -0400, Kevin A. McGrail wrote:
> > Without digging deep into this right now, and without a sufficient dose
> > of caffeine...
> >
> > Isn't that what the spamd protocol REPORT command does?
> >
> >   SYMBOLS  -- Check if message is spam or not, and return score plus list
> >               of symbols hit
> >
> >   REPORT   -- Check if message is spam or not, and return score plus report

Since these are actually described right next to each other -- Pedro,
you're looking into the spamd protocol, so I assume you did find the
spamd/PROTOCOLS file in the source?


> I was going to say the same thing.

Including the caffeine disclaimer? ;)

Pedro, just checked some details and it appears indeed to be what you
asked for -- almost. The main difference being the report template's
default content, which includes a "detection" disclaimer and content
preview.

With the following local.cf options, the output should look like what
you wanted.

  clear_report_template
  report _SUMMARY_

Also see 10_default_prefs.cf, which defines the default being overridden
here. Basically, that's a minimal report header, getting rid of
everything but the summary.


When using this with spamc, it's the -r or -R option. :)

Oh, and if you really want a CSV separated output, it should be possibly
to write a plugin that provides a custom Template Tag to be used instead
of the default _SUMMARY_, without hacking the spamd source itself.


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: SYMBOLS with test description

["Kevin A. McGrail" <KM...@PCCC.com>]

> Without digging deep into this right now, and without a sufficient dose
> of caffeine...
>
> Isn't that what the spamd protocol REPORT command does?
>
>   SYMBOLS  -- Check if message is spam or not, and return score plus list
>               of symbols hit
>
>   REPORT   -- Check if message is spam or not, and return score plus report
>
>

I was going to say the same thing.

Regards,
KAM

Re: SYMBOLS with test description

[Karsten Bräckelmann <gu...@rudersport.de>]

On Tue, 2010-10-19 at 13:09 -0200, Pedro Padron wrote:
> I was messing around with the spamd/spamc protocol, and I saw that the
> SYMBOLS method only return test names.
> However, it would be nice to be able to get also the individual test
> score and it's description, what do you think?

Without digging deep into this right now, and without a sufficient dose
of caffeine...

Isn't that what the spamd protocol REPORT command does?

 SYMBOLS  -- Check if message is spam or not, and return score plus list
             of symbols hit

 REPORT   -- Check if message is spam or not, and return score plus report


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}